Passing parameters in custom metrics in fastai2


Has anyone here had recent success specifying metrics for FastAI2 where you need to pass in the labels parameter (which gets passed to the underlying sklearn metrics function, and determines which labels are measured or averaged)? I get an error whenever I try to do this. I haven’t started debugging into the library yet, because I’m still wondering if I’m doing it wrong.

This Colab notebook shows the problem:

Looking at the stack trace, it seems like the problem may be that, within SKLearn, the set of labels in labels is what’s driving the creation of a LabelEncoder which presumably handles inter-conversion between textual labels and numbers to go into numpy arrays. That sounds like it might be wrong, as I am passing in only one label because it’s the label of interest. And the direct SKLearn call works fine, so this should be the right way to use the API, I would think.

Hmm, maybe all I’m seeing here is that sklearn and therefore the fastai2 metric functions all require that labels be numerical. :thinking:

IIRC the vocab of a dataloader has a o2i method to get the number of each label.

In case anyone else hits this question, I should add here that in another forum @shimsan pointed out the solution to me: when using the FastAI2 metrics function, you need to pass in the labels argument as numerical indexes not as the raw string labels.

So, for instance, if you’re string label is “n03394916” and you want the precision score for only that label, then you need to do this:

learnMicro = cnn_learner(dls,resnet18,metrics=Precision(labels=[dls.vocab.o2i['n03394916']],average='micro'))

This is different from the behavior of the underlying the SKLearn precision_score function, which the FastAI2 function wraps. SKLearn lets you pass in numeric labels if the dataset uses numeric labels, and string labels if the dataset uses string labels.

Maybe this is something that should be changed in how the FastAI2 metric wrapper works?

1 Like

Thanks for capturing it on the forum! And good work, making a Colab for your issue, so that I could quickly verify what worked :slight_smile: