What is the v2 equivalent of "AccumulateScheduler"?

[pierreguillou]

@boris

This is strange as it should be almost equivalent when you update the learning rat

Just do like me: in the notebook 10_nlp.ipynb of Jeremy, put cbs=GradientAccumulation() in learn.fit_one_cycle(). You should observe the huge running training loss (see my first screen shots)…

@boris

The main difference should be the effect of optimizers (momentum) or learning rate schedulers. Not sure if we could do something smart in the callback to account for it but it’s a bit tricky to think about what should be done here, even more when we can combine so many callbacks.

In fact, when you go until the end of the learner training (always the same example: the notebook 10_nlp.ipynb, see my second screen shot), you will observe that the valid loss and accuracy are right (compared to what Jeremy got) but the training loss is well high.

What I think:

• GradientAccumulation() works well.
• but… the running training loss up to the final one shows a true value but not the average one

How to correct the last point?