Thanks @Danielvs!
Yes, this is the fruits of work I had been doing until about March or so.
It originated in this thread from a few years back, but I will give a TL;DR to catch folks up to speed
Essentially before we had the two suggesters from the one-cycle paper, a global min and an estimated steepest slope (now called steep
and minimum
). We found that these tend to not give the best estimations for a good learning rate, and two other methods were created: the valley
algorithm and the slide
algorithm. They’re two other methods for getting a much better suggested learning rate to use OOTB without needing any graphical interpretation.
Valley and Slide can be used intermittently (or together as I’ll show in a moment!), but in general we found that: Valley > Slide > Steep > Minimum. Hence why the new default is valley
.
Now let’s bring it back to the LR Finder. You can pass in any or all of those above methods to lr_find
, and have all of them plotted on the same graph! So then you can interpret for yourself which LR you want to go with.
To do so all you need to do is:
learn = Learner(...) # any cnn_, tabular_, etc will work
lrs = learn.lr_find(suggest_funcs=(minimum, steep, valley, slide))
And this will then give you a pretty graph with those all plotted:
And the resulting return gives you a namespace object with all of those values stored. So ex you can do lrs.valley
to get the valley result, lrs.minimum
to get the minimum, or just do lr_min, lr_steep, lr_valley, lr_slide = learn.lr_find(....)
to get them returned too!
Along with this, it’s now super easy to write your own suggestion algorithm that you may want to experiment with. All you need to do is write your own function that accepts three params, as detailed in the documentation here: Hyperparam schedule | fastai
I hope that answers your questions Very happy to finally see this in the library, I’ve been baking it since March
Also, here is the table from our experiments.
A key:
- Leslie Smith = Steep
- ESRI = Valley
- Novetta = Slide