The callback metrics are only called during the validation phase
After some experimentation with LearnerCallback it seems that anything implementing this interface is called during both training and validation, although I had assumed it would only be called during validation similar to the Learner class.
In my case this mixed up training data and validation data, leading to a metric which was evaluating training and validation data together.
Has anyone come across this while implementing a LearnerCallback?
Is this the expected behavior?
Is there anyway to specify that a LearnerCallback should only be called during validation?
I see in the fastai that MultiLabelFBeta is also based on the LearnerCallback, so this might also be affected, though I haven’t looked into this.
A LearnerCallback or a Callback is called during training and validation if you pass it as a callback, during validation only if you pass it as a metric. The class is irrelevant, it’s how it passed to Learner that matters.
Ah ok, I didn’t realise you could pass a LearnerCallback as a metric. That clears it up
Is there an example of this somewhere? Perhaps an example of using MultiLabelFBeta?
I’ve made a few attempts to pass the LearnerCallback to a learner using the metrics parameter, and also the Learner as a callback_fns, but nothing seems to work
Oh you won’t be able to pass a LearnerCallback at creation since the learner dosn’t exist yet. You can append it to the metrics after the learner object has been created.