Can SaveModelCallback only be passed once?

fastai
1

In fastai version 2.5.3, I’m trying to pass SaveModelCallback twice to my learn.fit() function:

  1. To save the model with the best validation loss from the training cycle (’best’).
  2. To save the model at the very end (i.e., to save the most overfit version of the model, ‘end’).

However, when I run the code below, both SaveModelCallbacks just save based on best validation (example output below).

learn.fit_one_cycle(100, lr_max = 0.003 ,
                    cbs=[
                        ActivationStats(with_hist=True),
                        SaveModelCallback(
                            monitor = 'mae',
                            comp = np.less,
                            fname = 'best', min_delta = 0.001),
                        SaveModelCallback(fname='end', at_end=True)
                    ]
                   )

Example output:

Better model found at epoch 0 with mae value: 21.10202980041504.
Better model found at epoch 0 with valid_loss value: 805.0863037109375.

So the two questions (possibly related): 1. Why are both callbacks triggering even during the first epoch - shouldn’t the at_end callback only trigger in the final iteration? 2. Why is SaveModelCallback(fname='end', at_end=True) not triggering at the end? (I should add - I have run this to completion and have confirmed that the weights for the end model are not different from those for the best model.)

Or maybe at_end (whose meaning is not described in the documentation) doesn’t do what I expected it to do.

2

Looking at the source code for SaveModelCallback setting at_end=True doesn’t alter the behavior during training and overwrites any prior saved models after training is over.

def after_epoch(self):
    "Compare the value monitored to its best score and save if best."
    if self.every_epoch:
        if (self.epoch%self.every_epoch) == 0: self._save(f'{self.fname}_{self.epoch}')
    else: #every improvement
        super().after_epoch()
        if self.new_best:
            print(f'Better model found at epoch {self.epoch} with {self.monitor} value: {self.best}.')
            self._save(f'{self.fname}')

def after_fit(self, **kwargs):
    "Load the best model."
    if self.at_end: self._save(f'{self.fname}')
    elif not self.every_epoch: self.learn.load(f'{self.fname}', with_opt=self.with_opt)
1 Like
3

Thanks, this makes sense. I think the fundamental issue for my needs is the weight reloading at the end of training, which isn’t overridable. What I am doing now is changing the order of the two SaveModelCallbacks which works for my purposes. (In my original order, the best model would be loaded, and then the end model would just be saving a copy of that re-loaded best model.) If I needed something more complex, I’d make a new callback.