Lesson 7 - Official topic

[FraPochetti (Francesco Pochetti)]

It does not help at all for Random Forest.
Tree Ensembles are scale invariant.

It does help in case you are training a NN, in terms of learning stability.

[jcatanza (Joseph Catanzarite)]

You are correct. Standardizing the data (i.e. centering to zero the mean and re-scaling to get unit variance) is not necessary for tree-based models.

The reason is that decisions (splits) are based on inequalities, which compare relative feature values and are agnostic to scale and mean.

[muellerzr (Zachary Mueller)]

@muellerzr

I don’t quite recall the sizes, let me try to find it

Found it: Deep learning for tabular data: an exploratory study by Jan-Andre Marais. Published in 2019. He went into great detail trying out various hyperparameters and techniques with the fastai model

Fair warning it’s a hefty read (144 pages!). The bits we care about are from pages 62 to 85. Everything from attention to self normalization to unsupervised pretraining were attempted or discussed

[harish3110 (Harish Vadlamani)]

What are the other fancy methods one can use to ensemble a rf model and a nn tabular model other than averaging the results?

[victor.vargas]

Can you talk a little bit about the COVID-19 efforts at the end if possible? Would love to know what is the latest.

[ilovescience (Tanishq)]

Here is the questionnaire solutions if you want to check your answers or are struggling (work in progress ):

Fastbook Chapter 9 questionnaire solutions (wiki)

Here are the questions: What is a continuous variable? What is a categorical variable? Provide 2 of the words that are used for the possible values of a categorical variable. What is a “dense layer”? Equivalent to what we call linear layers. How do entity embeddings reduce memory usage and speed up neural networks? What kind of datasets are entity embeddings especially useful for? What are the two main families of machine learning algorithms? Ensemble of decision trees are best for str…

[jona (Jona R)]

Jeremy’s mic is dropping very infrequently.
Specifically about what to expect next week?

[tonibagur]

Question: Could we use the boosting technique with a neural net?

[mlabs (Aaron)]

jeremy’s audio is now cutting out … choppy…

[steef (Steef van Winkel)]

Thanks Rachel & Jeremy!

Next week NLP?

[MaheshKhatri]

Since Entity Embeddings was covered extensively in today’s class, sharing my Medium article with a link to my talk on the same at NY ML community last year.

An ‘Entity Embeddings’ sharing with New York’s AI Community

[giacomov (Giacomo Vianello)]

You’ve been waiting for 7 whole weeks now!

[steef (Steef van Winkel)]

Jeremy’s final words during today’s course: Next week NLP & Computer Vision

[victor.vargas]

Thanks Jeremy and Rachel

[ilovescience (Tanishq)]

Thanks for another great lecture! Interesting to learn more about some “traditional” ML techniques like random forests!

[steef (Steef van Winkel)]

Hahaha - yes I have

Excited it’s coming next week

[MaheshKhatri]

Thanks Jeremy, Rachel & Sylvain.

[ganesh.bhat]

Can someone please point to what entity embedding mean from the last bit of the lesson? Is it the one hot encoding that we discussed or something else?

[username_not_found (unknown)]

Thanks, see you all next week!

[jcatanza (Joseph Catanzarite)]

The enhancements that make Random Forest such a powerful Decision Tree model are:

1. Bootstrap sampling, which is selecting a random subset of the data (i.e. rows in the data table) to construct each decision tree, and
2. Selecting a random subset of the features (i.e columns in the data table) to make a ‘split’ at each ‘node’ in a decision tree.
3. Ensembling, which is contructing a group of models (in this case, a ‘forest’ of trees) and averaging their votes to make each final classification.

The first two of these enhancements are analogous in their application and their effect to the Dropout technique in Neural Networks.