Gradient accumulation for segmentation using fastai v2


How to accumulate gradient for semantic segmentation using fastai v2? I am training a model for steel defects detection and can’t create batches larger than 2 images during the experimentation. I will greatly appreciate if someone can point me to some code implementation.

Many Thanks and
Kind Regards,

Just use the gradient accumulation callback. Training callbacks | fastai

Thanks a lot @muellerzr. All I have to do is to use cbs=GradientAccumulation(n_acc=24) in the learner specification. It is very useful information. Cool.

Much appreciated.

Note that you will need to reduce the lr by the same factor used by GradientAccumulation to accumulate gradients.

1 Like

Do you mean lr = lr/24? In my use case I accumulate gradient after 24 steps.

That’s the response I received when asking the question. It seems reasonable and correct.

If you search these forums for “gradientaccumulation”, the topic is the first one found. It has a more detailed explanation.

Note that I have not personally studied the relevant fastai code nor thoroughly tested empirically. Just used the advice while training.

Thanks for the useful tip and Fast AI discussions. I will have a look.