Accumulating Gradients

fastai
107

Getting really high train/validation loss with AccumulateScheduler … is this normal because of the “sum” reduction?

Train Loss=1044.117188
Validation Loss=930.394165

EDIT:

The issue has to do with I’m attempting to use this in a language model classification problem … where there is a probability calculated for each actual token. I tried the below but it’s still reporting the large losses I mention above.

class AbSumAccumulateScheduler(AccumulateScheduler):
    
    def on_batch_begin(self, last_input, last_target, **kwargs):
        "accumulate samples and batches"

        self.acc_samples += last_input[0].shape[0]
        self.acc_batches += 1
        
    def on_backward_begin(self, last_target, last_loss, **kwargs:Any):
        #pdb.set_trace()
        n_predicted_tokens = len(last_target[last_target != -1])
        last_loss = last_loss / n_predicted_tokens
        return { 'last_loss': last_loss }
1 Like