Lr finder , any guaranty not to be over_shooted

Part 1 (2019)
1

Hi all

by lr_finder

when we are at the bottom of loss curve of val_set , we find biggest Lr that times by slope of the point (some where at the bottom)does not over_shoot us

like this link {https://www.google.com/imgres?imgurl=https%3A%2F%2Fwingshore.files.wordpress.com%2F2014%2F11%2Falpha3.png%3Fw%3D662&imgrefurl=https%3A%2F%2Fwingshore.wordpress.com%2F2014%2F11%2F19%2Flinear-regression-in-one-variable-gradient-descent-contd%2F&docid=xAjpT4vjhawiCM&tbnid=ehi30eMGO0YUiM%3A&vet=10ahUKEwiyqNvkpOTkAhV15eAKHYxQBvsQMwhTKAcwBw..i&w=577&h=415&bih=754&biw=1536&q=overshooting%20by%20high%20learning%20rate&ved=0ahUKEwiyqNvkpOTkAhV15eAKHYxQBvsQMwhTKAcwBw&iact=mrc&uact=8}

but , after finding lr , when we are gonna use fit function
we can be over_shooted , because we have bigger slope , this time , because we are not at the bottom

and another question is
in using that Lr for SGD with restart
for each restart , how is it possible to be shooted in another non_spiky valley ?

in picking Lr in Lr_finder, we were supposed to choose the biggest lr which can not over_shoot us

2

You pick the learning rate for which the slope decreases for the maximum length, which is before the minimum of the lr_find curve.

3

thanks for your reply @kushaj

as you said we pick lr some where near the bottom , where slope is decreased
so later when we are gonna use that for the fit , we are not necessarily near the the bottom
so we have bigger slope and timing that slope by our Lr can lead to over shooting

1 Like