Metric F1 score macro on classification ( single label multiclass )

Part 1 (2019)
1

For the information I have gathered on
https://docs.fast.ai/metrics.html#FBeta
https://forums.fast.ai/t/f1-score-as-metric/30370/52
and other places, FBeta is the right choice for multiclass classification (single label).

But for some reason it is not working for me and fbeta does.

I am trying

F1_macro = partial(FBeta, average='macro', beta = 1 )
learn_metrics=[accuracy,  F1_macro]
learn = cnn_learner(data, arch, metrics=learn_metrics)

And I get

TypeError: __init__() got multiple values for argument 'average'

If I do

F1_macro = partial(FBeta,  beta = 1 )
learn_metrics=[accuracy,  F1_macro]
learn = cnn_learner(data, arch, metrics=learn_metrics)

I get

AttributeError: 'FBeta' object has no attribute 'detach'

If I use

F1_score = partial(fbeta,thresh=0.5, beta = 1)

it works well (no errors).

I have read a variety of posts and couldn’t get any solution there.

And not only that, I do need the macro F1 score as in

image
image712Ă—294 6.53 KB

2

@rgarcia FBeta is a class. So, it must be instantiated before use.

fscore = FBeta(average='macro', beta=1)
learn = cnn_learner(data, arch, metrics=[fscore])

It worked well for me.

3 Likes
3

Thanks Nikhil.

I tried that before and now, and the error changed to

RuntimeError: The size of tensor a (50) must match the size of tensor b (12) at non-singleton dimension 2

I must be doing something silly/wrong but can’t figure what.

However using
F1_score = partial(fbeta,thresh=0.5, beta = 1)
works well, which I believe shoudn’t as it is for multilabel classification.

But

F1_macro = FBeta(  beta=1)
or
F1_macro = FBeta(average='macro',  beta=1)

doesn’t work.

I have read many post regarding

RuntimeError: The size of tensor a (50) must match the size of tensor b (12) at non-singleton dimension 2

but also couldn’t figure out what is really the problem since some people resolved it changing the bs (batch size) of the databunch.

I tried different bs values and that affected the first number on the error message.

There is something (or many things) I haven’t understood yet.

4

I am trying to use F1ScoreMulti for multiclass classification. But I am getting below error.

<ipython-input-5-474cb0b9ebfb> in get_learner(model)
      1 def get_learner(model):
----> 2     f1 = F1ScoreMulti(average="weighted")
      3     learn = cnn_learner(data, model, metrics=f1)
      4     return learn

NameError: name 'F1ScoreMulti' is not defined

I have tried other topic on F1Score but still I am getting this error.
Thanks

5

On lesson 10 of the NLP course I found this.

Taking the f1 from sklearn + transforming it to work with Tensors instead of np arrays.

from sklearn.metrics import f1_score
@np_func
def f1(inp,targ): return f1_score(targ, np.argmax(inp, axis=-1))
6

F1Score needs to be instantiated.

It looks like a function but is actually a class and needs to be
instantiated. F1Score is a class based on the skm_to_fastai
functions which returns a Class object (AccumMetric). AccumMetric
has function __call__ which basically allows you to call the
object as a function

So It will be:

f1_score_multi = F1Score(average="macro") ## convert class to functie
learn = cnn_learner(dls,resnet18,metrics=f1_score_multi)