Lesson 1 - Part 1 V3 - Why is error rate different between runs with same data and learning rate?

[yeldarb (Brad)]

This is likely due to learning rate annealing. Basically, when using fit_one_cycle, your learning rate is changing behind the scenes over the course of the run.

Because of this, comparing the 5th epoch of learn.fit_one_cycle(8) is not equivalent to the last epoch of learn.fit_one_cycle(5).

If you run them both with 5 or both with 8 epochs you should see more similar results.

There are a couple of other topics about this here:

'strange pattern' in error rate

Let me try to explain what I think happens. When you train it for the first time, weights are random, and error rate will lower if the lr is not too high. You are far from a global or local minima. But once you have trained the model once, that means you have found some minima (where you fit_one_cycle finished with a low lr). If you increase lr again (even if it’s only momentarily) error rate may go up, because you get further from that minima. I don’t know if the excellent visualization crea…

Lower accuracy when running fit_one_cycle again

I believe the idea of fit_one_cycle is to start with small learning rates, but high momentum, and then increase lr and drop mom for half the training, and reverse the trends for the second half. Doing a second cycle will initially have high momentum with a bit less history, so the start of training is not strongly tethered and can bounce out of a local minimum before stabilizing.

If you want actual 100% identical runs, there are several random seeds and a few settings to set. There’s a random_seed function here that will do the trick:

Lesson1 Reproducible Results - Setting seed not working

Another clue and partial solution. The issue is that after setting all the random seeds before creating the DataBunch, training and validation losses are inconsistent (non-reproducible) as of the first fit(). The losses have a bimodal or trimodal pattern. These observations are all after a kernel restart between runs, no transforms, and num_workers = 0. What I see is that setting these random seeds again before fit() yields consistent, reproducible loss results. However, setting them before cr…

I gave it a shot and