The 1cycle policy - an experiment that investigate super-convergence phenomenon described in Leslie Smith's research

[cedric (Cedric Chee)]

Hi guys. As I can’t edit my original post. Just to put this out there. My original post at this point is outdated as fastai library changed quite a bit since. Please refer to Sylvain Gugger’s posts from here on. Sylvain implemented the 1cycle policy and cyclical momentum (use_clr_beta) method in fastai library. I suggest you start looking here:

Using use_clr_beta and new plotting tools

I’ve written a few things for the fastai library while working on my project around Leslie Smith’s last article and I wanted to share them with you. use_clr_beta : This is the implementation of Leslie Smith’s 1cycle policy as detailed here. He recommends to use a cycle like this one: [art5_lr_schedule] for the learning rates during the training: equal time growing and descending, and a bit of space left to get the learning rate really slow. When used in fit, the value at the top (here 0.01)…