Gradient accumulation for segmentation using fastai v2

fastai
1

Hi,

How to accumulate gradient for semantic segmentation using fastai v2? I am training a model for steel defects detection and can’t create batches larger than 2 images during the experimentation. I will greatly appreciate if someone can point me to some code implementation.

Many Thanks and
Kind Regards,
Bilal

2

Just use the gradient accumulation callback. Training callbacks | fastai

3

Thanks a lot @muellerzr. All I have to do is to use cbs=GradientAccumulation(n_acc=24) in the learner specification. It is very useful information. Cool.

Much appreciated.

4

Note that you will need to reduce the lr by the same factor used by GradientAccumulation to accumulate gradients.
:slightly_smiling_face:

1 Like
5

Do you mean lr = lr/24? In my use case I accumulate gradient after 24 steps.

6

That’s the response I received when asking the question. It seems reasonable and correct.

If you search these forums for “gradientaccumulation”, the topic is the first one found. It has a more detailed explanation.

Note that I have not personally studied the relevant fastai code nor thoroughly tested empirically. Just used the advice while training.
:slightly_smiling_face:

7

Thanks for the useful tip and Fast AI discussions. I will have a look.