UserWarning when using SaveModelCallback()

ChrisCFranca · November 23, 2020, 12:04pm

Hello, everyone.

I have a very simple learner created in the following way:

learn = cnn_learner(dls=all_data,arch=resnet18, metrics=[accuracy, mcc, auroc]).to_fp16()

Where all_data is an ImageDataLoaders, mcc is an instance of MatthewsCorrCoef() and auroc an instance of RocAucBinary() (just some metrics). When i use the learn.fine_tune() method, i pass the callback SaveModelCallback() (and some others) in the following way:

learn.fine_tune(2,
base_lr=1e-2,
cbs=[SaveModelCallback(monitor=‘matthews_corrcoef’, comp=np.greater, fname=‘resnet18_3’, with_opt=False, reset_on_fit=True),
EarlyStoppingCallback(monitor=‘matthews_corrcoef’, comp=np.greater, min_delta=0.01, patience=3),
ShowGraphCallback()])

When i do so, the training goes normally, but i keep getting this warning after every epoch:

C:\Users\chris\anaconda3\envs\fastai_env\lib\site-packages\fastai\learner.py:54: UserWarning: Saved filed doesn’t contain an optimizer state.
elif with_opt: warn(“Saved filed doesn’t contain an optimizer state.”)

I know that it is the SaveModelCallback() since the warning disappears once i remove it.
Since it’s just a warning, i’m not exactly sure if i’m doing something wrong here. Can anyone help me? Thanks!!

muellerzr · November 23, 2020, 12:14pm

Looks to be a bug. Specifically here:

github.com

fastai/fastai/blob/master/fastai/callback/tracker.py#L83


    def _save(self, name): self.last_saved_path = self.learn.save(name, with_opt=self.with_opt)
    def after_epoch(self):
        "Compare the value monitored to its best score and save if best."
        if self.every_epoch: self._save(f'{self.fname}_{self.epoch}')
        else: #every improvement
            super().after_epoch()
            if self.new_best:
                print(f'Better model found at epoch {self.epoch} with {self.monitor} value: {self.best}.')
                self._save(f'{self.fname}')
    def after_fit(self, **kwargs):
        "Load the best model."
        if not self.every_epoch: self.learn.load(f'{self.fname}')
# Cell
class ReduceLROnPlateau(TrackerCallback):
    "A `TrackerCallback` that reduces learning rate when a metric has stopped improving."
    def __init__(self, monitor='valid_loss', comp=None, min_delta=0., patience=1, factor=10., min_lr=0, reset_on_fit=True):
        super().__init__(monitor=monitor, comp=comp, min_delta=min_delta, reset_on_fit=reset_on_fit)

I’ll put a PR in and fix this now

ChrisCFranca · November 23, 2020, 12:28pm

Thank you, @muellerzr!

muellerzr · November 23, 2020, 1:41pm

When this PR gets merged the problem will be fixed

Wound up being two-fold, you couldn’t do learn.load() without the optimizer and have it not raise a warning at you (in case anyone has tried to)