When should I train on a freezed model and when on unfreezed one? Are there particular indicators that tell us that?
Right now, I’m just randomly freezing and unfreezing to see what happens, but I don’t quite understand it.
Thanks for the help.
very short answer: keep watching the lessons, it all gets explained, but basically you train with it frozen for a few epochs (normally 3-4) and then unfreeze and train some more.
1 Like
I agree with @joedockrill. It will become clearer as the course proceeds. But still, i’ll try to add my own 2 cents.
There’s always a fixed routine. First train with a ‘frozen’ model, then unfreeze it and train it some more. Thats where we stop.
The idea behind is as follows:
We use a pretrained model for efficient training, right? Fastai doesnt exactly use the architecture that was designed originally for ImageNet (which most models are built upon), but removes the last layer, and appends 2 fully connected layers. The second one has as many output neurons as your number of categories.
Now, the last two layers are particularly untrained, where as the rest of the model is (pre)trained. So we first tweak the model by training only the last 2 layers(though you can change how many layers at the end you want to train). Then, as you know, every problem leads to different model weights. A model pretrained on Imagenet is not the best solution for your custom dataset. Even though the whole model need not be trained from scratch, because we imagine the earlier layers of the CNN to learn the basic features of images in general, be it ImageNet images or any other images, it still needs some tweak. Thats why we unfreeze the model and train for a little longer!
Hope that makes it slightly more clearer!
By the way, Jeremy and Sylvain found that only one epoch was really needed to bring up the weights. The new course will explore this further but we have a fine_tune function in v2, and by default it will do one epoch frozen 
5 Likes