Order of callbacks for learner object might be incorrect?

The order of callback functions defined here: https://github.com/fastai/fastai_pytorch/blob/master/fastai/basic_train.py#L113 is Recorder then every other callback.

However, say I have another Callback which does some computation on_epoch_end and returns some metrics, it will not be displayed by the recorder, since the recorder captures the results before the new callbacks are being executed.

Please correct me if I am misunderstanding anything.

Yup, this is correct. I have been thinking about it, especially because some callbacks might be computing a metric (like precision, recall, f2) that you can’t average over the batches, and need to pass new metrics. So this is an unsolved issue yet.

I was stuck at the same place, since I wanted to pass a newly evaluated metric.

Shouldn’t adding Recorder at the end of the list solve the issue?

More specifically: Don’t add recorder at this line: https://github.com/fastai/fastai_pytorch/blob/master/fastai/basic_train.py#L113 instead add recorder at the end here https://github.com/fastai/fastai_pytorch/blob/master/fastai/basic_train.py#L130

Not sure if this breaks down something else though

I think we need a better system, with the possibility to add a new callback where we want it (end/beginning or in between). I think callbacks in general need an order attribute so that the callbackhandler can sort them out, and this is what will solve this.
But no more dev for now, only doc ;). As soon as I get the time, this will be on the first new things implemented.

1 Like

IIRC some things rely on Recorder being present. It may be necessary to split it into multiple callbacks.