Hello,
I’ve used lr_find() to find a suitable learning rate but I got the graph below.
In this case what learning rate should I choose ?
Your loss is already quite low to begin with. Is this a model that has been trained already?
Yes, the model has already been trained but the accuracy is low (around 49%).
It’s a multilabel image classier.
I’ve followed the steps from lesson 1 and 2
Review: easy steps to train a world-class image classifier
Enable data augmentation, and precompute=True
Use lr_find() to find highest learning rate where loss is still clearly improving
Train last layer from precomputed activations for 1-2 epochs
Train last layer with data augmentation (i.e. precompute=False) for 2-3 epochs with cycle_len=1
Unfreeze all layers
Set earlier layers to 3x-10x lower learning rate than next higher layer
Use lr_find() again
Train full network with cycle_mult=2 until over-fitting
I’m now on step 7 (the second lr_find() ).
The first lr_find was pretty obvious (see attached).
Was it said that the trained model shouldn’t be used to calculate LR again?
May be I missed something.
Could you explain your thought ?
I began using resnet50 as a base model.
Did lr_find()
Trained for 4 epochs
Trained for another 4 epochs with precompute=False
Unfreezed all layers
Used lr_find() again and here I got the plot
In both cases you didn’t go far enough to find the good LR: on both your first graph we don’t get the part where the curve goes up again. In the second one, it’s logical to have something flat at the beginning (the variations we see have a 0.002 amplitude) because your network has been pretrained.
I’d run the LR Finder with end_lr=100 or 1000 to get a clearer curve.
2 Likes
Thank you. I’ll give it a try.