Part 2 Lesson 10 wiki

[wgpubs (WG)]

@blakewest

In the video (right here: https://youtu.be/_ercfViGrgY?t=51m29s), Jeremy says, “as per usual, we do a single epoch on the last layer”, which he explains are the embedding weights, and he does this because they’re the thing that will be the most wrong.

But the code, right before he runs the epoch, but he calls learner.unfreeze() just a few cells above his epoch run, which unfreezes all layers. I’m left to assume there’s some missing code that should be like learner.freeze(-1) or something right before he starts training the language model, no? Any insights here?

Is there a definitive answer to this question anywhere?

Based on the ULMFiT paper, the recommendation is to fine-tune “only the last layer” (section 3.2) before unfreezing and applying discriminative learning rates to the other layers. As such, why is there the line learner.unfreeze() immediately before fitting the model begins? It seems that it should be learner.freeze(-1) unless I’m missing something (which is typically the case )