I’m trying to resolve a puzzler. I’m following v1 of the course; I’m on lesson 2. I am trying to train all the dense layers after finetuning the model. Surprisingly, the loss becomes catastrophically worse after I compile the new model (goes from 1.0 to 15.0 on a State Farm 10 class classification problem). Here’s the detailed breakdown of what I do:
- I load the VGG model.
- I use vgg.finetune() to finetune the last layer and switch it to 10 classes.
- I train on a small sample set.
- The loss starts high (~6) and over 30 epochs goes down to ~1.
- I run predictions on a small test set.
- I modify the model to make all dense layers trainable.
- I run predictions on the same small test set, for later comparison.
- I compile the model.
- I run predictions on the same small test set, for later comparison. These three runs should have the same results.
- I train the model on the sample set.
- The loss goes to ~15
- The loss never improves, over 30 epochs.
- I run predictions on the same small test set again.
- The first three sets of predictions match (i.e. predict the same classes). This is expected, since the model hasn’t change.d
- The fourth, computed after the model is retrained, only ever predicts one category, always the same (‘c3’).
If anyone is interested, the Jupyter notebook is at https://github.com/nomothetis/fastai/blob/master/lesson-2/lesson-2-control.ipynb
The first cell can be ignored (I believe…); it’s mostly setup. It’s the following cells that actually do the work.
One thing I tried is not calling
compile(), which seems to work, but puts out a warning about the trainable parameter count not matching. I’m pretty sure that means I’m only train the old model. Predictably, the training continues correctly.
Any help is appreciated.