FBeta score in fastai: 'macro' vs 'weighted'

Part 1 (2019)
1

I’m looking for a metric that is more representative of the imbalanced image dataset I’m working on. In sklearn the fbeta metric has ‘macro’ and ‘average’ as potential input for the ‘average’ parameter. (link)

With fastai passing nothing to FBeta() results in a notification that the macro will be used. Passing the average instead as a parameter results in nan for the FBeta metric output.

learn = create_cnn(data, models.resnet50, metrics=[error_rate, FBeta(average='macro')], callback_fns=ShowGraph)

Is there something I missed here?

I’m happy with any advice regarding imbalanced datasets, especially regarding oversampling, different thresholds, metrics.

1 Like
2

I think you need to pass a function for the metric, not a function call. Take a look at how to define a partial function in Python.

3

Hi,

I changed it in the following way, without the intended effect though. It still returns nan.

fbeta = FBeta(average='weighted')

learn = create_cnn(data, models.resnet50, metrics=[error_rate, fbeta],
                   callback_fns=ShowGraph)

This is what you mean with passing a function right?

By the way, passing macro instead of weighted does work. So maybe its due to this argument. I’m interested in the weighted error though!

4

Oops, my mistake. I thought you were trying to pass the fbeta metric function rather then the FBeta class instance.

Let me try tracing the code when I get back to my computer. If I can’t figure it out, we’ll need to ask an expert.

Later…
I don’t see anything wrong with your code, and it returns a reasonable answer when I run it.

There are a couple of places where the code could blow up into NaN, here:
self.metric = (1 + self.beta2) prec rec / (prec * self.beta2 + rec + self.eps)

and
elif avg == “weighted”:
return self.cm.sum(dim=1) / self.cm.sum()

But I hesitate to diagnose without the actual error in front of me, and without knowing the intent of the FBeta code.

You might post a minimal example of the failure. Short of that, fbeta.eps = 1e-2 might localize the overflow and show that precision and recall are both zero.

HTH, Malcolm

5

Hi. This post might show the cause of the error you are getting.