I’m trying to use the F1ScoreMulti metric with a multi-classification problem, however its giving me an error:
/opt/conda/lib/python3.6/site-packages/fastai2/learner.py in accumulate(self, learn)
441 def accumulate(self, learn):
442 bs = find_bs(learn.yb)
--> 443 self.total += to_detach(self.func(learn.pred, *learn.yb))*bs
444 self.count += bs
TypeError: unsupported operand type(s) for *: 'AccumMetric' and 'int'
This occurs when running the following lines:
learn = cnn_learner(dls, resnet50, pretrained=True, metrics=[F1ScoreMulti])
If I instead switch the metrics to use ‘accuracy_multi’ instead of ‘F1ScoreMulti’ everything works fine.
Do I need to do something different when using the Sklearn metrics?
You may need to call an instance of it first, IE:
F1ScoreMulti(). sklearn metrics are all supported (as detailed in this notebook: https://github.com/fastai/fastai2/blob/master/nbs/13a_metrics.ipynb)
Thanks again for the very swift help.
Creating an instance does fix the problem, so my code now looks like:
f1score_multi = F1ScoreMulti()
learn = cnn_learner(dls, resnet50, pretrained=True, metrics=[f1score_multi])
I’d checked out the documentation page, created from the notebook you added the link for, but hadn’t seen it mentioned that an instance was required for the Sklearn derived metrics (although I may well have missed this!)
It does not explicitly state this, the guess was due to your error saying it was the parent object type and not the function itself, as it was trying to grab part of it (I’ve faced this issue quite often to where it’s just a “know” thing)
In general, everything that begins with a capital needs to be instantiated (we are naming functions that needs to be called with a capital too, to help with that).
F1Score needs to be instantiated.
pred = torch.randint(0,10,(64,))
targ = torch.randint(0,10,(64,))
f1score_ins = F1Score(average="macro")
Learn needs a
metric function. In this case
It looks like a function but is actually a class and needs to be
F1Score is a class based on the
functions which returns a Class object (
__call__ which basically allows you to call the
object as a
So It will be:
f1_score_multi = F1Score(average="macro") ## convert class to functie
learn = cnn_learner(dls,resnet18,metrics=f1_score_multi)
Depending on your problem you need to choose
average or leave it to default