About learning rate

[lizhuofeng (李卓风)]

Please look at this picture, why the best learning rate is not taken at the lowest point of loss
please help me,thanks very much

[mike.moloch (github:aeamaea)]

I have similar question. I found this article:

Using the Learning Rate Finder (Beginner) | walkwithfastai

[ilovescience (Tanishq)]

In a nutshell, you want an area where the loss is decreasing the fastest. If you select the learning rate at the minimum, the loss isn’t really changing there, and that’s not the behavior you want for your training. This is why we mostly choose the steepest part of the loss curve.

[wyquek (Qbiwan)]

The fastai book has a section on learning rate finder in chapter 5, section Improving Our Model, The Learning Rate Finder.

[lizhuofeng (李卓风)]

@ilovescience

In a nutshell, you want an area where the loss is decreasing the fastest. If you select the learning rate at the minimum, the loss isn’t really changing there, and that’s not the behavior you want for your training. This is why we mostly choose the steepest part of the loss curve.

I know that this graph is not a curve of learning rate and loss，So it is meaningful to examine the derivative of learning rate and loss

[ilovescience (Tanishq)]

@lizhuofeng

So it is meaningful to examine the derivative of learning rate and loss

Yeah we are looking for the steepest slope so you would think this would be the best approach. However, sometimes there are certain areas that may have a good derivative value but not the best LR to select. For example, here is an loss vs LR plot and red dot on the right has the steepest value but it’s clearly not ideal as that is when the loss is increasing and plateauing: