Lesson 5 In-Class Discussion

Part 1 (2018)
175

Thanks!

I also have a problem further on when creating the examples by scratch:

fit(model, data, 3, opt, F.mse_loss)


Epoch
0% 0/3 [00:00<?, ?it/s]


  0%|          | 0/1251 [00:00<?, ?it/s]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-25-09a5bc469dee> in <module>()
----> 1 fit(model, data, 3, opt, F.mse_loss)

~/fastai/courses/dl1/fastai/model.py in fit(model, data, epochs, opt, crit, metrics, callbacks, **kwargs)
     83             batch_num += 1
     84             for cb in callbacks: cb.on_batch_begin()
---> 85             loss = stepper.step(V(x),V(y))
     86             avg_loss = avg_loss * avg_mom + loss * (1-avg_mom)
     87             debias_loss = avg_loss / (1 - avg_mom**batch_num)

~/fastai/courses/dl1/fastai/model.py in step(self, xs, y)
     41         if isinstance(output,(tuple,list)): output,*xtra = output
     42         self.opt.zero_grad()
---> 43         loss = raw_loss = self.crit(output, y)
     44         if self.reg_fn: loss = self.reg_fn(output, xtra, raw_loss)
     45         loss.backward()

~/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/functional.py in mse_loss(input, target, size_average)
    817 
    818 def mse_loss(input, target, size_average=True):
--> 819     return _functions.thnn.MSELoss.apply(input, target, size_average)
    820 
    821 

~/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/_functions/thnn/auto.py in forward(ctx, input, target, *args)
     45         output = input.new(1)
     46         getattr(ctx._backend, update_output.name)(ctx._backend.library_state, input, target,
---> 47                                                   output, *ctx.additional_args)
     48         return output
     49 

TypeError: CudaMSECriterion_updateOutput received an invalid combination of arguments - got (int, torch.cuda.FloatTensor, torch.cuda.DoubleTensor, torch.cuda.FloatTensor, bool), but expected (int state, torch.cuda.FloatTensor input, torch.cuda.FloatTensor target, torch.cuda.FloatTensor output, bool sizeAverage)
176

OK these should both be fixed now - thanks for reporting the problem!

1 Like
177

Thanks @jeremy - all good for the first part, but again with the “from scratch” area of my notebook I got the cuda error - but then saw that you had replaced y = ratings[‘rating’] with y = ratings[‘rating’].astype(np.float32) - implementing this fixed things.

I guess anyone running old code based on the previous version of the noteboook will still get those cuda errors…

178

Hi guys!

Is there a way to calculate the accuracy of the CollabFilter? Just by the validation loss is difficult to understand how good is the algorithm, especially when you are using your own dataset and don’t have a benchmark.

I tried to use the accuracy_thresh(0) that uses accuracy_multi behind the scenes, but the result is always the same 0.3414.

179

I used:

  • AUC for 2 class CF
  • % of correctly classified in each bucket for multiclass
180

Video timelines for Lesson 5

  • 00:00:01 Review of students articles and works

    • “Structured Deep Learning” for structured data using Entity Embeddings,
    • “Fun with small image data-sets (part 2)” with unfreezing layers and downloading images from Google,
    • “How do we train neural networks” technical writing with detailled walk-through,
    • “Plant Seedlings Kaggle competition”

  • 00:07:45 Starting the 2nd half of the course: what’s next ?
    MovieLens dataset: build an effective collaborative filtering model from scratch

  • 00:12:15 Why a matrix factorization and not a neural net ?
    Using Excel solver for Gradient Descent ‘GRG Nonlinear’

  • 00:23:15 What are the negative values for ‘movieid’ & ‘userid’, and more student questions

  • 00:26:00 Collaborative filtering notebook, ‘n_factors=’, ‘CollabFilterDataset.from_csv’

  • 00:34:05 Dot Product example in PyTorch, module ‘DotProduct()’

  • 00:41:45 Class ‘EmbeddingDot()’

  • 00:47:05 Kaiming He Initialization (via DeepGrid),
    sticking an underscore ‘_’ in PyTorch, ‘ColumnarModelData.from_data_frame()’, ‘optim.SGD()’

  • Pause

  • 00:58:30 ‘fit()’ in ‘model.py’ walk-through

  • 01:00:30 Improving the MovieLens model in Excel again,
    adding a constant for movies and users called “a bias”

  • 01:02:30 Function ‘get_emb(ni, nf)’ and Class ‘EmbeddingDotBias(nn.Module)’, ‘.squeeze()’ for broadcasting in PyTorch

  • 01:06:45 Squeashing the ratings between 1 and 5, with Sigmoid function

  • 01:12:30 What happened in the Netflix prize, looking at ‘column_data.py’ module and ‘get_learner()’

  • 01:17:15 Creating a Neural Net version “of all this”, using the ‘movielens_emb’ tab in our Excel file, the “Mini net” section in ‘lesson5-movielens.ipynb’

  • 01:33:15 What is happening inside the “Training Loop”, what the optimizer ‘optim.SGD()’ and ‘momentum=’ do, spreadsheet ‘graddesc.xlsm’ basic tab

  • 01:41:15 “You don’t need to learn how to calculate derivates & integrals, but you need to learn how to think about the spatially”, the ‘chain rule’, ‘jacobian’ & ‘hessian’

  • 01:53:45 Spreadsheet ‘Momentum’ tab

  • 01:59:05 Spreasheet ‘Adam’ tab

  • 02:12:01 Beyond Dropout: ‘Weight-decay’ or L2 regularization

6 Likes
181

Thanks @sermakarevich!

182

Embedding Dimensions:
In the Excel spread sheet the Embedding for movies is a 5x15 matrix (before the bias is applied) and the user Embedding matrix is 15x5.

However the Embedding matrices used in the Lesson5 Notebook seem to be off where users is 671x50 and movies is 9066x50. I thought to multiply matrices the number of columns must match the number of rows vice versa…

embedded
embedded.png1268×245 20.2 KB

It seemed like Jeremy first multiplied the movie embedding and user embeddings to (replacing empty ratings with zero) to determine the first initial predictions and to update the factors in the embedding matrices.

Do I have this right?

1 Like
183

I had the same issue with the Rossman Notebook, I fixed by replacing:
md = ColumnarModelData.from_data_frame(PATH, val_idx, df, yl, cat_flds=cat_vars, bs=128, test_df=df_test)

with

md = ColumnarModelData.from_data_frame(PATH, val_idx, df, yl.astype(np.float32), cat_flds=cat_vars, bs=128, test_df=df_test)

until (I think) it will modified directly inside the method :wink:

184

Thanks !

I agree, would be better if the method coped with the data type and ensured it is correct - seems awkward to have to explicity cast it this way… Hopefully @jeremy will tidy it up after the “dust has settled” - though I figure that he is incredibly busy all of the time!

185

Variation in result of movielens

While nb has validation loss of 0.767
image
my validation loss is .894 which doesn’t get a better solution than previous benchmarks. My RMSE is .945 vs .91 of best. Fastai’s nb has RMSE of .881

image
image.png721×318 13 KB

Q1. What could be the reason behind this variation?
Link to my nb

Q2. There training loss seems to be drastically changing from .84 to .3 in just 3 epochs vs .81 to .6 in fastai nb.

1 Like
186

I’d like some clarity as well, but this is what might be happening:

The results that you see in the notebook are from a previous run when we were using SGD with momentum as the optimizer (the previous default in fastai).

The notebooks were updated recently with AdamW implementation (as did the fastai code) to reflect the new optimizer usage, but doesn’t seem to have been rerun. So you see the old results with new code changes.

Owing to this, I’d expect to see some variation. But it seems to kinda start overfitting after only 1 epoch. :slight_smile:

[Edit]: If I do learn.lr_find(), it seems to indicate that an lr of 1e-2 might be high.

image

Somewhere around 3e-3 or 4e-3 seemed to work better.

1 Like
187

I’m blind to overfitting. :frowning:

But, I’m curious to know what would be optimal lr in this case?

188

xe-3 would have been a better choice because we want the weights(ultimately the loss I mean) to converge slowly and steadily or otherwise we might end up jumping around the two cliffs as Jeremy said…(if we choose the lr to be too close around the sunken part in the graph)

Our motive is to select the best possiblelr which will help us to converge and I always do a lr/10 of the value shown in the graph(min value) and it seems to work but sometimes need to play around with it also as loss didn’t converge when it goes to high…

Please correct me if I understood it incorrectly…

2 Likes
189

At the moment there isn’t separate datasets of this type for regression vs classification, so it’s up to you to ensure your dependent is of the right type. If we cast it to float always (as I used to do) it doesn’t work for classification.

190

I actually think 1e-2 looks about right here. It’s dropping quickly at that point.

That’s not what we want if we use SGDR (which I think we always should) - we want a max learning rate which does cause a bit of a jump!

2 Likes
191

That isn’t used unless you add a param to explicitly enable it.

1 Like
192

Thanks for the explanation Jeremy :smile:

193

Ahh my bad. I saw the movielens notebook using optim.Adam call and fit using weight decay and weight decay scheduler. Doesn’t this mean it’s using AdamW? Or do we need another param to enable this? (I might have a totally incorrect understanding of AdamW at this point).

194

@A_TF57 You need to set use_wd_sched=True to use AdamW, check it here: New AdamW optimizer now available

3 Likes