I’ve just finished watching (for the first time!) the Part 1 videos, which feels like a lot to take in and an incredible resource! I’m a bit unsure though on how to progress with a structured data problem, and in particular the revisit of it in Lesson 6 felt a bit rushed compared to the NLP/computer vision revisits. I’ve played with my own data set a bit, but I just feel like I’m experimenting with hyperparameters without the same sort of understanding as we developed for image classification/NLP etc. I know Jeremy said the approach of using DL for structured data is quite unusual, but is there any further reading anyone’s used to deepen their understanding of it?
This deals with one of the more fascinating topics for me in ML, distributed representations or in other words embeddings 
If that is the bit you would like to gain a greater intuition on, I recommend the original paper on this by Geoffrey Hinton. I found it very approachable and a great read.
2 Likes
Thanks for the suggestion @radek and congrats on the recent Kaggle win. The Hinton paper was very helpful. I’ve spent a lot of time searching for more information on DL for structured data as well and would love to have the topic revisited in a (hopefully) future video.
My interest in DL for structured data is due to the fact that tree ensemble models can’t project into the future effectively on data outside of their input range, particularly time-series data.
Outside of utilizing embeddings, I’m trying to learning more about:
- architecture options (MLP’s versus RNN’s vs ?)
- Data transformation ideas
- Test and Validation Set Creation for multi-index datasets
Has anyone found researchers or folks on twitter that have been helpful on structured DL ? Just from searching the forums on here a lot there seems to be a consistent core of people looking for this info. It would definitely be helpful to get a master thread of resources on this topic together.
I completely agree with what you mention.Use of Deep neural nets on structured data is one of the most underrated problem as per me.I have in the past tried to reach out to people to identify if there has been any experimental proof across large scale problems where we may say Deep nets beats RF/GBDT etc on tabular data but I have not got any concrete answer.I am even game to create a work group where we pick up different structured problems on kaggle and identify what is the benefit if any we are observing of using Deep neural nets against other algorithms.
1 Like
Maybe we can use this thread to share resources on using neural networks for structured data?
On Kaggle, I have noticed that GBM often outperform NN. However, sometimes you get some pearls like this one which placed 3rd in the instacast competition:
At least check the readme, we have first level models on each data table, mainly RNNs, but also one CNN/RNN combination. Then second level models, one using LGBM, and one NN.
Any other good examples out there?
2 Likes