What is a good learning rate range?

Part 1 (2019) Advanced (Part 1 v3)
1

Hi All,

Could you tell me what would be good learning rate parameters (upper and lower bounds) considering the plot below?

Thanks in advance!

39
Screen Shot 2019-03-15 at 2.05.39.png757×421 18.4 KB

2

You’re looking for the steepest downward slope before that large upward spike, which is this nearly linear part between 1e-4 and 1e-3. I think going with 5e-4 or so is a good start. To be honest, you might want to rewatch the according lesson tho if you cannot answer this yourself. Jeremy explains it pretty nicely.

1 Like
3

The upper bound should be around 3e-04 and for the lower bound it completely depends on your data. Check lesson 3 and lesson 4

4

Thanks for your feedback. I had rewatched that part several times and it was in an attempt to deepen my understanding that I sought help.

5

Thanks for your feedback. I’m still on lesson 1 but will be sure to be on the look out for what you mentioned. Thanks again @swagman

6

I like to use between 1e-6 and 1e-1

7

Thank you

8

According to fast ai docs, you should choose a learning rate that is in the middle of the steepest downward slope. For range, you can choose a range from where the steepest downslope begins and just a little before it starts the upward trend. As mentioned above lesson 3 and 4 provides more info on choosing the best learning rate for different layers of the model.

One thing you can do is pass suggestion = True in learn.recorder.plot and it will mark the spot where it thinks the learning rate is best for your model. Also, you can get the lr by using the below code.
min_grad_lr = learn.recorder.min_grad_lr min_grad_lr

https://docs.fast.ai/callbacks.lr_finder.html#Suggested-LR