SaveModelCallback reset_on_fit usage?

Part 1 (2020)
1

TLDR; I’m trying to save the best model from multiple fine tuning steps. It looks like SaveModelCallback has this functionality but I can’t figure it out.

I’m using layer unfreezing to help fine tune a model for NLP.

I use fine tuning to squeeze a little more accuracy out of the model and loop through this code several times as part of doing some cross-validation. As a result, the model can overfit at different epochs and different calls to fit_one_cycle. I use the SaveModelCallback but this seems to save the best model from the subsequent fit calls.

I thought the parameter reset_on_fit might allow the to pick up right where I left off with the previous fit but it doesn’t seem to perform this functionality.

I’ve check the modified date on the model saved at it always changes between calls to fit_one_cycle.

My code looks something like this

cbs = SaveModelCallback(monitor='_rmse', comp=np.less, min_delta=0.005, reset_on_fit=True)

learn.fit_one_cycle(7, lrs.lr_min, cbs=cbs)

learn.freeze_to(-2)
learn.fit_one_cycle(1, slice(lrs.lr_min/2/2.5**4, lrs.lr_min/2), cbs=cbs)

learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(lrs.lr_min/2/2/2.5**4, lrs.lr_min/2/2), cbs=cbs )

learn.unfreeze()
learn.fit_one_cycle(5, slice(lrs.lr_min/2/2/2/2.5**4, lrs.lr_min/2/2.2), cbs=cbs)
2

It does only if you pass SaveModelCallback into Learner not to the fit call I believe

1 Like
3

Interesting. I’ve always use the callbacks with the fit and not assigned it to the learner. I’ll check that out.

4

There’s use cases for both. 99.999% of the time you’d want it passed through fit. You want things passed through the Learner when you want some callback that stays for multiple fits and maintains a state. (As a result they may also be called during .predict and .get_preds, so when exporting make sure you get rid of those callbacks! :slight_smile: )

1 Like
5

:slight_smile: That worked like a charm. Only now I have to investigate what will happen if I don’t delete my callbacks and subsequently call .predict or .get_preds (which of course I do).

6

For anyone looking at this later, one thing I figured out with some testing is that SaveModelCallback is going to look in the default directory for the saved model on each fit call.

As a result if you are doing cross validation, it is just loading the old model and trying to improve on that one (which is going to get very hard).

What I did is to change my callback assignment as follows…

for i in range(2):
    cbs = [
        SaveModelCallback(reset_on_fit=False, fname=f'model_{i}'),
    ]

The key here is that as you iterate through cross-validation sets you need to change the fname parameter so that when you tell your learner and pass the cbs to the learner you get a new model saved.

2 Likes
7

Another option you can do is if you have cb=SaveModelCallback(reset_on_fit=False), you can then do cb.fname = f'model_{i}' :slight_smile:

This way you never have to redeclare the callback, you just change the filename as needed

2 Likes
8

I am applying the reset_on_fit in EarlyStoppingCallback but it shows ‘TypeError: init() got an unexpected keyword argument ‘reset_on_fit’’.

callbacks = [EarlyStoppingCallback(learn,monitor=‘valid_loss’,min_delta=0.0, patience=10, reset_on_fit=True)]
learn.fit(80, lr=0.00001, callbacks=callbacks)