Using f1 score instead of accuracy

I’m stuck in lesson 1 (dogs&cats) when I try to replace accuracy with f1 score.
I do these steps:

from sklearn.metrics import f1_score
metrics=[f1_score]

arch=resnet34
data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
learn = ConvLearner.pretrained(arch, data, precompute=True, metrics = metrics)
learn.fit(0.01, 2)

However I get this error:

ValueError: Classification metrics can't handle a mix of binary and multilabel-indicator targets

Here I can see an example of f1 score function:

def f1(preds, targs, start=0.17, end=0.24, step=0.01):
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        return max([f1_score(targs, (preds>th), average='micro')
                    for th in np.arange(start,end,step)])

But I get the same error.
What could be the best way to use f1_score for unbalanced datasets?

1 Like

If I use fbeta without any arguments:

arch=resnet34
data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.metrics = [accuracy,fbeta]
learn.fit(0.01, 2)

I get this:

TypeError: fbeta() missing 1 required positional argument: 'beta'

okay, got a notebook that works. Sorry, don’t mind if I clean the mess I made in this thread.

2 Likes

Oh, that’s amazing! Thank you so much! Yes, we should clean this )