Lower accuracy when running fit_one_cycle again

It seems that when I have run fit_one_cycle for some epochs
and to it again, the accuracy is often lower than it was at the end of the previous cycle.
Is this common and do others experience this?

can it be that some state is not saved?

1 Like

I am also facing the exact same issue.

I believe the idea of fit_one_cycle is to start with small learning rates, but high momentum, and then increase lr and drop mom for half the training, and reverse the trends for the second half. Doing a second cycle will initially have high momentum with a bit less history, so the start of training is not strongly tethered and can bounce out of a local minimum before stabilizing.

Yes leaving and continuing training seems to mess up the losses. I try to train 10 epochs in one go.

This problem has already been discussed and answered here.

Basically, it is because the fit_one_cycle anneals the learning_rate towards 0.

29

So, when you retrain the model, the initial learning rate is higher than what previous training end up with, resulting in bigger steps that make your gradients go out from the minimum that previous training found, thus giving you and a higher loss value.

2 Likes

Awesome. Thank you for this answer