Has anyone here had recent success specifying metrics for FastAI2 where you need to pass in the labels parameter (which gets passed to the underlying sklearn metrics function, and determines which labels are measured or averaged)? I get an error whenever I try to do this. I haven’t started debugging into the library yet, because I’m still wondering if I’m doing it wrong.
Looking at the stack trace, it seems like the problem may be that, within SKLearn, the set of labels in labels is what’s driving the creation of a LabelEncoder which presumably handles inter-conversion between textual labels and numbers to go into numpy arrays. That sounds like it might be wrong, as I am passing in only one label because it’s the label of interest. And the direct SKLearn call works fine, so this should be the right way to use the API, I would think.
In case anyone else hits this question, I should add here that in another forum @shimsan pointed out the solution to me: when using the FastAI2 metrics function, you need to pass in the labels argument as numerical indexes not as the raw string labels.
So, for instance, if you’re string label is “n03394916” and you want the precision score for only that label, then you need to do this:
This is different from the behavior of the underlying the SKLearn precision_score function, which the FastAI2 function wraps. SKLearn lets you pass in numeric labels if the dataset uses numeric labels, and string labels if the dataset uses string labels.
Maybe this is something that should be changed in how the FastAI2 metric wrapper works?
