Potential bug with LearnerCallback?

On the documentation page for creating your own metric it says in regards to the Callback class

The callback metrics are only called during the validation phase

After some experimentation with LearnerCallback it seems that anything implementing this interface is called during both training and validation, although I had assumed it would only be called during validation similar to the Learner class.

In my case this mixed up training data and validation data, leading to a metric which was evaluating training and validation data together.

Has anyone come across this while implementing a LearnerCallback?
Is this the expected behavior?
Is there anyway to specify that a LearnerCallback should only be called during validation?

I see in the fastai that MultiLabelFBeta is also based on the LearnerCallback, so this might also be affected, though I haven’t looked into this.

A LearnerCallback or a Callback is called during training and validation if you pass it as a callback, during validation only if you pass it as a metric. The class is irrelevant, it’s how it passed to Learner that matters.

1 Like

Ah ok, I didn’t realise you could pass a LearnerCallback as a metric. That clears it up

Is there an example of this somewhere? Perhaps an example of using MultiLabelFBeta?

I’ve made a few attempts to pass the LearnerCallback to a learner using the metrics parameter, and also the Learner as a callback_fns, but nothing seems to work

Oh you won’t be able to pass a LearnerCallback at creation since the learner dosn’t exist yet. You can append it to the metrics after the learner object has been created.

Thanks for your help, that’s working now.

Here’s an example in case it’s useful for anyone else

1 Like