How is Learning rate chosen?

[cereal_bird (William)]

In the videos and in thedocs, Jeremy says to pick the learning rate between a range where the loss graph has the steepest slope. Why is this? Shouldn’t the learning rate be fixed at the point where loss is the lowest? That would find the local minimum that fastest and most efficiently, no? Could anyone clarify this for me.
Thanks

[tokyolights (Mario Meissner)]

Hello!

The plot is made by training the model while increasing the learning rate at each step. The horizontal axis can thus be regarded not only as the learning rate, but also time, as the model trains and improves. You are looking for a learning rate that can decrease the loss quickly, thus you look for where the slope is the steepest.

If you look at the lowest loss value, and look left and right, you see that the loss wasn’t improving that much at that point in the training. That means that those learning rates aren’t helpful anymore for reducing the loss.

I hope this was helpful! I was confused at the beginning as well.

[mrfabulous1]

Hi cereal_bird hope you are having a marvelous day!

I always find looking at two or three explanations helps me understand things easier.
Below are some links that cover learning rate!

Picking optimal learning rates?

Hi all, How do you all approach picking the optimal learning rates to fine-tune your models? I am under the impression I should be picking a range that has the deepest declining slope. In my last few cells, I thought a range of 1e-4, 1e-2 or 1e-3, 1e-1 were of the steepest slopes but my results aren’t as good as 1e-6, 1e-4 (which looks flat in the curve to me). V2_Question_on_LR_Lesson1.ipynb Appreciate any thoughts

Doubt - how to use lr_find plot

I am trying to understand the logic behind choosing learning rates for the given lr_find plot.Apparently choosing the slicing range around 1e-01 (corresponding to the lowest loss) blows the validation loss out of proportions, while slice(1e-06,1e-04) has high losses but works just fine. I was under the notion that we choose the range where the loss is the lowest [fastai2] Now, I do understand that slicing range around 1e-01 is a very high range and the exploding validation losses is something …

Cheers mrfabulous1

[cereal_bird (William)]

Thanks for the answer. To clarify, the learning rate is always changing?

[cereal_bird (William)]

Thanks

[tokyolights (Mario Meissner)]

The specifics of how the finder function is implemented are unknown to me, but I would say yes, at every step (weight update), they change the learning rate.

[cereal_bird (William)]

That’s where my confusion was. I though that the learning rate was a fixed number. Thanks.

[tokyolights (Mario Meissner)]

Glad that I could help!

[newvick]

So Jeremy sometimes passes in a single number to the LR and that may be where the confusion is. That single number is the maximum learning rate for the learner object, but it still means that the learning rate is always changing.

It starts off small, increases to the max LR, then decreases again. So it’s like this shape: /\

[cereal_bird (William)]

Is increasing then decreasing LR proven to have an advantage? How much does it help?

[shreyas606 (Shreyas Verma)]

Increasing the learning rate value to a specified max value and then decreasing it helps in generalizing better , in the sense,that this prevents the model to get stuck in some local minima present in the loss function and the model is able to generalize better by reaching the global minima of the loss.

The initially increasing slope of the learning rate cycle does its best to jump out of any local minima on the loss surface…post that, a decreasing slope of the learning rate helps you update the weights slowly when the model is in the global minima region