A code snippet to save the best model during training

(Ismaël Koné) #1

Hi everyone,
I’m sharing with you a small code that saves the best model after all epochs of
a training run.
Below is a typical screen shot while calling the fit method.

best_model1

The best model was after the 9th (8th from 0) epoch with 0.99 accuracy on the validation set.
If you use the save_cycle attribute, only the end cycle epoch will be saved. But as a cycle could have many epochs, what if you have your best score after an epoch within a cycle? You’ll miss it. I didn’t do the maths to know whether in this case the 9th epoch was within or at the end of a cycle.
Anyway to solve this problem, I used a callback. I defined a class:

class SaveBestModel(LossRecorder):
    def __init__(self, model, lr, name='best_model'):
        super().__init__(model.get_layer_opt(lr, None))
        self.name = name
        self.model = model
        self.best_loss = None
        self.best_acc = None

    def on_epoch_end(self, metrics):
        super().on_epoch_end(metrics)
        loss, acc = metrics
        if self.best_acc == None or acc > self.best_acc:
            self.best_acc = acc
            self.best_loss = loss
            self.model.save(f'{self.name}')
        elif acc == self.best_acc and  loss < self.best_loss:
            self.best_loss = loss
            self.model.save(f'{self.name}')

This class inherits from the LossRecorder which in turns inherits (not directly) from CallBack class of the fastai lib. For details, have a look in the file sgdr.py in the fastai lib.
Briefly, I check the model with best accuracy and best loss at the end of each epoch and save it. As I use the same name so the last one saved will be the best of the training run.
Then we have to call it with the fit method like:

lr = np.array([lrf/25., lrf/5., lrf])
my_cb = SaveBestModel(learn, lr, name='best_sgdr')
learn.fit(lr, 2, cycle_len=2, cycle_mult=1, callbacks=[my_cb])

Finally, we can load our best model after the training run as usual:

learn.load('best_sgdr')
learn.TTA()

And that’s all.
Hopefully this will be helpful for some of you.
A final word is that, I’m really appreciating now the good design of the fastai lib. Because when I had this idea, I thought I had to do surgery through the lib to get it work and finally it was not the case. Really neat design @jeremy !

15 Likes

Free GPU credits for Fast.ai Courses
Saving the best models during fit()
Introduce yourself here!
(RobG) #2

I believe you can use load_cycle() though it is a manual selection - Lesson 4 IMDB has an example:
m3.fit(lrs, 7, metrics=[accuracy], cycle_len=2, cycle_save_name=‘imdb2’)
[ 0. 0.29053 0.18292 0.93241]
[ 1. 0.24058 0.18233 0.93313]
[ 2. 0.24244 0.17261 0.93714]
[ 3. 0.21166 0.17143 0.93866]
[ 4. 0.2062 0.17143 0.94042]
[ 5. 0.18951 0.16591 0.94083]
[ 6. 0.20527 0.16631 0.9393 ]
[ 7. 0.17372 0.16162 0.94159]
[ 8. 0.17434 0.17213 0.94063]
[ 9. 0.16285 0.16073 0.94311]
m3.load_cycle(‘imdb2’, 4)

Maybe your code to recall the ‘best’ could be made into a PR?

2 Likes

(Ismaël Koné) #3

Thank you for your reply.
In fact the ‘load_cycle’ method will load saved models at the end of cycles. A cycle could have many epochs. And what is printed are the metrics(losses, accuracies) after each epoch of total number of cycles specified when calling fit method. So this method is guaranteed to save your best performance on the valid set while save_cycle is not.
Thank you for the PR suggestion, I’ll try.

3 Likes

Language Model Zoo :gorilla:
Lesson 4 Advanced Discussion ✅
(Ismaël Koné) #4

The PR has been merged.
@digitalspecialists Thank you again for the suggestion.
Indeed, it’s my first time to submit a PR.
Now, this is how it works:

learn.fit(lr, 2, cycle_len=2, cycle_mult=1, best_save_name=‘mybestmodel’)

learn.load(‘mybestmodel’)

Note: You must change the name from one call to another otherwise you’ll erase the previous weight saved. Because the method starts by saving the first epoch model and then compares to next ones and update accordingly.

12 Likes

(Matthew Teschke) #5

@iskode Thank you for taking the time to make this wonderful addition to fastai!

I had a question regarding how your change works when supplying custom metrics. For example, I am fitting my model like: learn.fit(lr, 10, metrics = [custom_metric], best_save_name = best_model_name)

I tried looking at your code, but I couldn’t figure out if the best model being saved is determined by my custom metric or by the loss function (in my current case, cross_entropy). Could you let me know if it’s the metric or the loss that determines the best model? In my case, I’d like it to be the metric.

0 Likes

(Ismaël Koné) #6

Hi Matthew,
thank you for your comment.
The code looks for the best accuracy by comparing previous and current accuracies at each iteration.
Whenever the current accuracy equals the best previous, it saves the one of lesser loss.
Take a close look at the method “on_epoch_end” and read it line by line it will be clear.
A final note is that the metrics inside this method is composed of the metric (in your case custom_metric) and the loss function (this one can also be specified) :
loss, acc = metrics
In your case will be: loss, custom_metric = metrics.
hopefully, it helps.

2 Likes

(Paul ) #7

Hi,

Thanks for making this addition. It is a really useful feature to allow more exploratory training and still “catch” the best epoch.

I think I may have found an issue though.

If no metrics value is passed in the fit method (such as in lesson4-imdb.ipynb when training the language model), the on_epoch_end callback fails when trying to unpack metrics into loss and acc. In such a case the validation loss should be used to determine the best model as a fallback.

On the other hand, it is also possible to pass a list of metrics to the fit method (eg. “… metrics=[accuracy, my_metric]”) should you wish to calculate multiple metrics, and I am unclear how or if this would be handled correctly or what the correct logic should be.

Finally, when you find a new best accuracy you also save the current loss as the new best loss and I am not sure that best accuracy will necessarily equate to the lowest loss in all cases.

2 Likes

(Ismaël Koné) #8

Hi Paul,
thank you for your feedback! That’s a great contribution.

If no metrics value is passed in the fit method (such as in lesson4-imdb.ipynb when training the language model), the on_epoch_end callback fails when trying to unpack metrics into loss and acc. In such a case the validation loss should be used to determine the best model as a fallback.

Indeed I didn’t test this feature in the case of RNN (RNN_Learner class in rnn_train.py). I’ve tested only on Convolutional NN (ConvLearner class) as this one sets a default accuracy while the former not. So I’ll update it.

On the other hand, it is also possible to pass a list of metrics to the fit method (eg. “… metrics=[accuracy, my_metric]”) should you wish to calculate multiple metrics, and I am unclear how or if this would be handled correctly or what the correct logic should be.

That’s also a good catch as this feature currently is not intended for multiple metrics. So the update will take this case into account. For the best model choice logic, I think keeping it simple will be good, I mean choosing the best model based on first metric of the list. So anyone wanting the best model based on a particular metric among his list must put it first.
If no metrics, choose the best model based on the loss as you pointed.

Finally, when you find a new best accuracy you also save the current loss as the new best loss and I am not sure that best accuracy will necessarily equate to the lowest loss in all cases.

You’ve noticed another interesting point. This is illustrated in my first post of this thread, at epoch 5: val_loss = 0.09247 with val_accuracy = 0.97 while at epoch 9 val_loss = 0.0704 with val_accuracy = 0.95.
So here we suppose the best model criterion is the accuracy provided and only we look at the loss to break ties when accuracies are equal. I think it makes sense that among equal accuracies, the best one is the least lossy.

So overall thank you again for such good remarks and catches.

1 Like

(Paul ) #9

Hi,

I think your suggestion to use the first metric, in the case that multiple are passed, as the one to use to choose the best model makes a lot of sense and keeps things simple. I look forward to testing out the changes when they are released.

Thanks again for a useful addition to the fastai library.

2 Likes

(Said Aspen) #10

Awesome!

0 Likes

(Ismaël Koné) #11

Hi Paul,
the correction has been merged to the lib.
Thank you so much.

0 Likes

(Hugh Runyan) #12

Are any of the cycle_save_name, callbacks, etc called by default? Here is the basic outline of my situation: I ran a model overnight. At some point in the middle of a cycle, it started to overfit (training loss continuing to decrease, validation loss increasing). I want to 1) store/access the training and validation loss as a function of epoch for easy analysis/plotting, and 2) be able to access the weights corresponding to each epoch. I initialized training with learn.fit(lrs, num_epochs, cycle_len, cycle_mult), but no explicit flags for saving/storing anything along the way. I’ve looked through the library: it looks like most of those things default to None. Do I have to re-train with the extra parameters explicitly declared?

0 Likes

(Will) #13

just a quick clarification in the docs.

It says if no metrics are passed then the measurement used is the loss. Is this the loss on the training set or validation set, just so that I’m clear how it’s working.

Thanks!

0 Likes

Using fast.ai in the wild - a report from the Kaggle frontline
(Ismaël Koné) #14

It’s the loss of the validation set. In general, our best model is always chosen based on the performance on the validation set.

1 Like

(Mark Worrall) #15

Hi @iskode,

I was looking into this as I happened to be experiencing odd behaviour with this feature so wondered if you were able to confirm my thoughts. The situation is that I wish to monitor multiple metrics during training but still save my model based on validation loss, so I thought to get around the model being judged on the first metric passed I would pass the loss function again as the first metric (here it’s BCE loss). Unfortunately (and please correct me if I’m wrong, I’m no Python pro) in this case we default to the save_when_acc method which expects the second metric to be maximized - hence I’m actually saving my worst model?!

I know this is easy to fix with either a custom class or defining 1-BCEloss as a new metric but if this is the actual behaviour it’s a little confusing.

What do you think? Happy to hear your thoughts.

Mark

1 Like

(Ismaël Koné) #16

Hello @maw501,
good catch !
I agree with you it’s weird and constrained. Because to use the validation loss for the best model selection one is obliged to not add any metric. Whenever the parameter metrics is not empty, the method uses automatically the first metric and consider it as an accuracy. Thus the best model corresponds to the higher value. That means if your metric is a loss then the best model will save the worst model in fact as in this case the lower the better.
To solve this problem and allow anyone to print many metrics and to freely choose which one to select its best model, I think of adding a parameter which is a tuple of 2 values: one for the metrics index and the order is a boolean indicating whether the selected metric (for the best model) is a loss or not. Not loss means it’s an accuracy. With these parameters, hopefully we’ll be able to decouple the metrics and best model choice.

0 Likes

(Mark Worrall) #17

Hi @iskode,

Thanks for the reply. Yes, there are many ways to approach and really depends on how much complexity/overhead we want for the call. I think decoupling saving the model from monitoring metrics is important though.

To your point about passing in a tuple, it might be easier to make the second term a boolean for whether the metric is being minimized or not. But as I said, many options for this and given dev1 is in progress not sure it’s realistic for many changes now…?

Thanks again,
Mark

0 Likes