LR finder for fine-tuning


#1

Hello, I was wondering if a LR finder for fine-tuning is in the works? To the best of my understand, the way to use a pretrained network is:

  1. Freeze pretrained layers, and train added layers (LR finder works fine here)
  2. Unfreeze pretrained layers and train whole network with differential training rates, as you can only have one LR for the LR finder, we wouldn’t be able to find the optimal learning rates for both the pretrained layers (which usually require a smaller LR) and the layers that you have added.

Would be happy to make a PR, as the project I’m currently working on needs this :slight_smile:


#2

In old fastai as in new, you can pass an array of lrs in the parameter start_lr/end_lr of lr_finder, and have a LR Finder for discriminative learning rates.


#3

Hi @sgugger,
Thanks for your contribution to this amazing package.

I still don’t understand how to get multiple LRs for fine-tuning.
Is there any notebook that I could check to see how to pass the arguments?

Is the following code snippet suppose to generate three plots/curves?
I’m getting a single plot with a single curve.

learn = ConvLearner(data, arch=tvm.resnet34)
learn.unfreeze()
learn.lr_find([1e-7, 1e-7, 1e-7], [1,1,1],100)
learn.recorder.plot()

Thanks


#4

You will only get one curve, which are the losses against the maximum lr. What happens behind the scenes is that we go exponentially from end_lr to start_lr, whether it’s one value or an array of values.


#5

Thanks for the prompt reply.
But I was wondering how to use this to find differential learning rates
If I get one curve how can I find the best learning rate for each group?
I am looking for a method that returns a curve for each layer group.

Thanks


#6

This method doesn’t exist. We haven’t tried modifying separately the learning rates yet.