Gradient Accumulation

does anyone know how to implement this on fastai?

for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()                           # Reset gradients tensors
        if (i+1) % evaluation_steps == 0:           # Evaluate the model when we...
            evaluate_model() 

Can use this for fastaiv2 http://dev.fast.ai/callback.training#GradientAccumulation

@khursani8, thanks but The project I doing uses fastai 1 and I have tried to upgrade but it has problems… do you know how to do it on version 1?

Thanks.

you can use this example code https://docs.fast.ai/train.html#AccumulateScheduler

1 Like

I found a solution, thanks to lafoss,

https://www.kaggle.com/iafoss/hypercolumns-pneumothorax-fastai-0-831-lb