How to retrieve Callback generated by Callback_fns after train?

fastai
1

Hi everyone, I would like to be able to retrieve my losses and metrics at the end of my training. When calling
learner.fit(...) a look in basic_train shows that for each element of callback_fns, a callback is generated and passed to fit. Since Recorder is in callback_fns, I would like to extract the Recorder at the end of training to be able to access the loss/metrics. However, after doing learn.fit, learn.callbacks doesn’t contain the Recorder.
Indeed, when you look at the source code of fit, you see
callbacks = [cb(self) for cb in self.callback_fns] + listify(callbacks)
fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics, callbacks=self.callbacks+callbacks)
which means the instance of Recorder is created when fit is called, but from what I see, is not saved in Learner.callbacks.

I found a solution to my problem. I created a fit2 method for Learner in which I return the Callbacks at the end of training(basically copy the fit method and add a return callbacks line at the end), but this seems a little convoluted. Can you think of a better solution to get the Recorder (or any Callback generated from Callback_fns) at the end of Training?

2

Any callback function you pass in callback_fn becomes an attribute of your learner (lower case, with _ separation). For instance the recorder is in learn.recorder.

2 Likes
3

That’s great! Thank you very much for your answer. Could you point me to where learn gets an attribute for each callback_fns in the source code? I feel like I’m asking ‘trivial’ questions that could be answered by thorough look in the source code. Before asking this question, I tried to check learn attributes to find the callback, and checked the source of learn.fit to see where the callback associated to callbacks_fn were saved, but without success. I also checked the doc of fit and callbacks but couldn’t find info on where callbacks_fn callbacks were saved. Do you have any advice on how to better check fastai source/doc while looking for this kind of information? I would love to answer those questions by myself and not waste your time.

4

In this case, it’s defined here, in the class LearneCallback. By definition, callback_fns are all LearnerCallback since they take a learner when you create them, which is why it’s defined here.
Note that any other LearnerCallback you define, even outside of callback_fns becomes an attribute of the associated Learner.

2 Likes
5

I tried to access LearnerTensorboardWriter callback with learner.learner_tensorboard_writer but it said:

tai/script.py", line 40, in call_parse
    func(**args.__dict__)
  File "examples/train_penn.py", line 104, in main
    print(learn.learner_tensorboard_writer)
AttributeError: 'LanguageLearner' object has no attribute 'learner_tensorboard_writer'