How to make SaveModelCallback() use accuracy in the filename?

[aviopene (Avio)]

At the moment I’m saving my models during training like this:

learn.fit_one_cycle(20, max_lr=slice(1e-6, 1e-3), callbacks=[SaveModelCallback(learn, every=‘epoch’, monitor=‘accuracy’, name=save_dir + model_name + ‘-data-aug-no-flips-’ + currtime)])

This works fine on Colab (it’s the dog breed classification challenge on Kaggle), I launched this yesterday evening and now, in the morning, I have a nice 90.9% accuracy model. Wonderful.

I was just wondering if there was a way to make SaveModelCallback() use the accuracy information writing the filename. Can I do that “externally” like I do above adding date/time and so on, or I have to modify Fast.ai callbacks/tracker.py?

In Keras it can be done as explained here:

[lokeshkvn (Lokesh)]

@aviopene have you got the solution to name model using accuracy?

[aviopene (Avio)]

Yep, now I’m using this callback:

class SaveBestAccuracyCallback(TrackerCallback):
    "A `TrackerCallback` that stops the training when the accuracy reaches the desired level."
    def __init__(self, learn:Learner, save_str:str='best-accuracy-model'):
        super().__init__(learn, monitor='accuracy', mode='auto')
        self.prev_acc = 0.0
        self.save_str = save_str

    def on_epoch_end(self, epoch:int, **kwargs:Any)->None:
        "Check the monitored value and save the model if above prev_acc."
        current = self.get_monitor_value()
        if current is not None and self.operator(current, self.prev_acc):
            print(f'Accuracy {current} is above previous accuracy {self.prev_acc} at epoch {epoch}. Saving model...')
            self.prev_acc = current
            self.learn.save(f'{self.save_str}-best-acc-{current}-epoch-{epoch}')

I’ve tested it only with the accuracy metric and for sure it will explode with many other metrics, e.g. with error_rate because self.prev_acc should be initialized to 100.0 and I’m not sure that self.operator() will remap itself correctly for other metrics (but maybe it will, it wouldn’t make much sense otherwise). So use it at your own risk, but it’s a starting point

[lokeshkvn (Lokesh)]

Thanks!