F1 measure not improving with epochs

Part 1 (2018)
1

Hi there,

I am running a standard image classifier with an f1 measure. I am not seeing an improvement to my f1 score

Here is my measure:

def f1(preds, targs):
     preds = np.argmax(preds, 1)
     targs = np.argmax(targs, 1)
     return sklearn.metrics.f1_score(targs, preds, average='micro')

My classifier:

arch = resnet34
sz = 224

def get_data(sz):
    tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.05)
    return ImageClassifierData.from_csv('/home/paperspace/images', 'train_images', label_csv, tfms=tfms,
                        val_idxs=get_cv_idxs(images_count), suffix='.jpeg', test_name='test_images')

learn = ConvLearner.pretrained(arch, get_data(sz), precompute=True, metrics=[ f1 ])

Here is some output:

1
1.png1260×588 56.9 KB

2
2.png974×612 34.9 KB

3
3.png1244×790 82.7 KB

What could I do differently?

Thanks.

2

Hello,

your F1 score function is not working.
Also no threshold is given to get 0/1 outputs for calculating the F1 score.

I used this F1 score function (based on the F2 function in fastai) which delivers comparable results to one in a kaggle competition:

from sklearn.metrics import f1_score

def f1(preds, targs, start=0.17, end=0.24, step=0.01):
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        return max([f1_score(targs, (preds>th), average='micro')
                    for th in np.arange(start,end,step)])

Be aware that the function checks for the best threshold and the best threshold is not necessary lying in the defined start and end window.

To get the optimal threshold in your code you can use the ‘opt_th’ function from planet.py which is in the course directory.

Best regards
Michael

3 Likes