Problem with F1ScoreMulti metric

fastai fastai dev
1

I’m trying to use the F1ScoreMulti metric with a multi-classification problem, however its giving me an error:

/opt/conda/lib/python3.6/site-packages/fastai2/learner.py in accumulate(self, learn)
    441     def accumulate(self, learn):
    442         bs = find_bs(learn.yb)
--> 443         self.total += to_detach(self.func(learn.pred, *learn.yb))*bs
    444         self.count += bs
    445     @property

TypeError: unsupported operand type(s) for *: 'AccumMetric' and 'int'

This occurs when running the following lines:

learn = cnn_learner(dls, resnet50, pretrained=True, metrics=[F1ScoreMulti])
learn.fit_one_cycle(1)

If I instead switch the metrics to use ‘accuracy_multi’ instead of ‘F1ScoreMulti’ everything works fine.
Do I need to do something different when using the Sklearn metrics?

5 Likes
2

You may need to call an instance of it first, IE: F1ScoreMulti(). sklearn metrics are all supported (as detailed in this notebook: https://github.com/fastai/fastai2/blob/master/nbs/13a_metrics.ipynb)

6 Likes
3

Thanks again for the very swift help.

Creating an instance does fix the problem, so my code now looks like:

f1score_multi = F1ScoreMulti()
learn = cnn_learner(dls, resnet50, pretrained=True, metrics=[f1score_multi])

I’d checked out the documentation page, created from the notebook you added the link for, but hadn’t seen it mentioned that an instance was required for the Sklearn derived metrics (although I may well have missed this!)

5 Likes
4

It does not explicitly state this, the guess was due to your error saying it was the parent object type and not the function itself, as it was trying to grab part of it (I’ve faced this issue quite often to where it’s just a “know” thing)

5

In general, everything that begins with a capital needs to be instantiated (we are naming functions that needs to be called with a capital too, to help with that).

6 Likes
7

F1Score needs to be instantiated.

pred = torch.randint(0,10,(64,))
targ = torch.randint(0,10,(64,))
f1score_ins = F1Score(average="macro")
f1score_ins(pred,targ)

Learn needs a metric function. In this case f1score_ins.

It looks like a function but is actually a class and needs to be
instantiated. F1Score is a class based on the skm_to_fastai
functions which returns a Class object (AccumMetric). AccumMetric
has function __call__ which basically allows you to call the
object as a function

So It will be:

f1_score_multi = F1Score(average="macro") ## convert class to functie
learn = cnn_learner(dls,resnet18,metrics=f1_score_multi)

P.S
Depending on your problem you need to choose average or leave it to default binary.

6 Likes