Lesson 4 In-Class Discussion

[vikbehal (Vikrant Behal)]

Guess both are embeddings. Instead of using Word2Vec embeddings, we create our own from scratch.
Per @yinterian question, we could use both…like merging or something?

[zaoyang]

When you use multiple languages, does this mean you have to use multiple models and select between them? Is there a way to use one big model?

[vikbehal (Vikrant Behal)]

Interesting. As Jeremy said, the model learnt English. Maybe, when we’ve multiple languages, it’ll learn to distinguish? Thought - Start with a model and throw data to teach multiple languages and use that as a pre-trained model?

[anamariapopescug]

Word2vec learns an embedding matrix as well. There are multiple ways of learning embeddings.

[Even (Even Oldridge)]

@Jeremy, If you don’t mind going into it, how does this method of creating the language embeddings differ from those that would be created by running GloVe or word2Vec on the custom dataset?

Is this method superior because it’s leveraging some of the new techniques you’ve covered like gradient descent with restart and (I’m assuming) bi-lstms with dropout, or is there something else going on under the hood?

[rikiya (Rikiya Yamashita)]

@jeremy Is there any rule of thumb for choosing “categorical” or “continuous” when it comes to discrete variables? What are pros and cons of each method? For example, we could say “year” as both categorical and continuous. Thanks in advance.

[jeremy (Jeremy Howard)]

What does dropping a unit mean? Set it’s activation to 0? What happens in the back propagation step? Does the unit get it’s weights updated?

Yes set to zero. So no gradients are propagated back from that unit for that mini batch.

[jeremy (Jeremy Howard)]

Can you double-count a column by using it categorically and continuously?

Not with the fastai lib, unless you make a copy of it.

[jeremy (Jeremy Howard)]

Why are school holiday and promo in continous var not catagorical ?

Since the are binary, it doesn’t really matter where they go.

[jeremy (Jeremy Howard)]

Sunday was a Rank1 matrix of length 4 (4x1). How could it fit (get appended) into the original input, which was nx1. I did not understand the dimensions.

n*1 concat 4*1 --> (n+4)*1

Make sense?

[jeremy (Jeremy Howard)]

Why is 0.04 used as dropout for early layer. I thought 0.25 or 0.5 was standard? Why is [0.001, 0.01] dropout used for later layers?

With different types of datasets you can need very different amounts of dropout. The normal amounts for image CNNs are very different to structured data, for instance.

[jeremy (Jeremy Howard)]

Is there a reason why [1000, 500] is used for number of activations? Is there a heuristic to know what number to use?

Not really - you have a try a few amounts. But somewhere around this number should be OK. If you have less data, you’ll need less activations, or you’ll overfit.

[jeremy (Jeremy Howard)]

I was wondering can the data from embeding matrix after training may be used as some kind of clustering.

Yes it works great. Remind me to show you next week if I forget!

[jeremy (Jeremy Howard)]

I thought embeddings were more like One Hot Encoding, which is a permanent label that doesn’t change. If SGD changes the embedding, what’s the point? Isn’t the purpose of embedding just an encoding why constantly change this encoding of “Sunday”?

The point is to learn the embedding of ‘Sunday’. We don’t know up front what an appropriate encoding is.

[jeremy (Jeremy Howard)]

Is there a way we can get the individual relative importances of the variables by using the neural network approach for regression similar to how we can get the relative feature importances in http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html # feature_importances_

Conceptually, yes, you can use the same approach, but it’s not in the library yet (or any other software). Would make for a good project!

[jeremy (Jeremy Howard)]

This is a very particular question around zipcodes. They are categorical by construct even though they are numbers. So @jeremy suggested to use embeddings, but at the same time, aren’t these zipcodes a function of (locality,area,city,county,state)? What I’m trying to say is, its already a 1 dimensional embedding pre-designed by whosoever came up with how to design zipcodes.

Case in point, do we still do embeddings on zipcodes?

Great point. I’d suggest adding columns for locality, area, city, county, and state, and then make them all (including zip code) separate categorical variables!

[jeremy (Jeremy Howard)]

I think he meant he hasn’t seen the usage of augmentation/ bootstrapping in the exact similar fashion as embeddings. But still, I think he will be better able to answer your question.

Yeah I guess SMOTE is, strictly speaking, a data augmentation technique for structured data. But I haven’t seen it used with these types of models or know if it’s helpful. It seems tricky to implement with embeddings - not quite sure how it would work…

[jeremy (Jeremy Howard)]

Why are embeddings typically between 50 and 600?

It’s just what empirically has been found to work by NLP researchers. >600 hasn’t been found helpful so far. It seems the implicit dimensionality of words in English just isn’t higher than that.

[binga (Phani Srikanth)]

@yinterian just a note to ask jeremy to tell us the trick (at 53:00 minute) towards the end of the class.

bump! @jeremy

[rsrivastava (rachana)]

I am using Random Forest model for a project there we are analyzing URLs and content. But features looks like this:
http://kase.twemail.com.de/contact/dmB4L3686825_frIV688.iTBV516_92qwH.html 0.0,0.0,2570.0,783.0,0.0,0.0,0.0,1.0,0.0,0.0,20.0,6.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,2.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,3.0,1.0,1.0,1.0,3.0,0.0,1.0,103.0,1.0,2

Here 2 at the end id the label and rest are feature. Valid values for most of the feature is -1, 0, 1. Seems like features are already categorical with three categories. Not sure how to add more dimensions using embedding to leverage deep learning.