About the one Cycle policy in fastai


#1

Hi everyone,
In fastai, when calling Learner.fit_one_cycle(n), and seeing the learning rate evolution at the end of training, I am under the impression, that we only go through one cycle, and n is the number of epoch that it takes to perform that cycle.

My understanding of Leslie N. Smith paper and one cycle policy is the following. Instead of having one constant or decreasing learning rate, you modify the learning rate during training in a cyclic manner, going from low learning rates to higher learning rates. From what I read in the paper, the idea is to go through many cycle.

For instance, if I want to train my model fors 50 epochs, my understanding of the Leslie N. Smith paper makes me think that I would get better result by doing let’s say 5 cycles of 10 epochs that 1 cycle of 50 epochs. If I wanted to do those 5 cycles in fastai, I would have to call Learner.fit_one_cycle(10) 5 times which seems very weird to me.

Is my understanding of fit_one_cycle correct? And if so, what’s the best way to train a model for like 50 epochs?


(Zahar) #2

Good question (I don’t have the answer)

In this project I needed 4000 epochs to converge. Each epoch is small, only about 50 images. I ended up running 40 cycles 100 epochs each. I tried running 10 and 200 epochs per cycle, but didn’t notice any improvement over 100. However it was a very limited experiment.


#3

I think you may be confusing two papers. Leslie N. Smith first introduces cyclical learning rates, where there were multiple cycles, but then changed it to 1cycle, which, as the name indicates, consists of only one cycle.


#4

Thanks for the clarification.


#5

Hi, think this warrants a bit more discussion. If I am doing a medical segmentation problem where the images are large and it takes about 1.5 hours to get through an epoch on GPU with batch size 2, would a one cycle also be good for such a large size data? I only have around 230 training samples but they are 3D images. I was thinking the same as OP to apply multiple cycles. What do you think I should do?