Potential bug: Inconsistent predictions and class labels

Part 1 (2019)
1

I use a resnet16 for a binary image classification task with fastai. I use my learner to get predictions on my test set,

test_preds_raw = learn.get_preds(ds_type=DatasetType.Test)

Then I zip these predictions together with the corresponding image filenames,

test_fnames = list(map(lambda path: str(path).split('/')[-1], learn.data.test_ds.items))
unzipped_preds = list(zip(*test_preds_raw))
fnames_and_preds = list(zip(test_fnames, unzipped_preds))

Finally, I filter out entries where the predictions (ie. ‘probabilities’) and class labels do not match.

def is_pred_inconsistent(tpl):
    tpl = tpl[1]
    return np.argmax(tpl[0]) != tpl[1]
inconsistent_preds = list(filter(is_pred_inconsistent, fnames_and_preds))

Now I get a bunch of entries where the predicted class label and the ‘probabilities’ are inconsistent…

[('01_841_act_l1_n08.jpg', (tensor([0.4679, 0.5321]), tensor(0))),
('01_841_act_l1_k20.jpg', (tensor([0.1810, 0.8190]), tensor(0))),
('05_841_act_l5_m13.jpg', (tensor([0.2760, 0.7240]), tensor(0))),
('02_841_act_l2_i07.jpg', (tensor([0.4829, 0.5171]), tensor(0))),
('05_841_act_l5_h15.jpg', (tensor([0.2376, 0.7624]), tensor(0))),
('05_841_act_l5_f17.jpg', (tensor([0.4292, 0.5708]), tensor(0))),
('05_841_act_l5_c08.jpg', (tensor([0.1060, 0.8940]), tensor(0)))]

Unless I did something stupid in the above snippets, I believe this isn’t supposed to happen. I’m looking for an explanation for this, maybe I misunderstand what learner.get_preds() is supposed to return…

2

To get the actual label, you need to take the argmax. Eg:

y, _ = learn.get_preds(DatasetType.Test)
y = torch.argmax(y, dim=1)
preds = [learn.data.classes[int(x)] for x in y]

This will give you the actual classes labeled.

1 Like
3

That’s exactly what happens in my method is_pred_incosistent. I’m still looking for an answer.

I guess my point is this: if we have,

probs, labels = learn.get_preds(ds_type=DatasetType.Test)

shouldn’t torch.argmax(probs, dim=1) be the same as labels? Why?

4

Assuming ('01_841_act_l1_n08.jpg', (tensor([0.4679, 0.5321]), tensor(0))) is the input to is_pred_inconsistent then by doing tpl = tpl[1] you replace tpl with (tensor([0.4679, 0.5321]). This is bad however as in the next line np.argmax(tpl[0]) returns 1 while tpl[1] is 0.5321. This always will return false.

5

@jdb100:

Assuming ('01_841_act_l1_n08.jpg', (tensor([0.4679, 0.5321]), tensor(0))) is the input to is_pred_inconsistent then by doing tpl = tpl[1] you replace tpl with (tensor([0.4679, 0.5321]) .

This is not the case, you can see it for yourself here: https://repl.it/repls/SatisfiedEarnestMachinecodeinstruction

7

@mkd It’s not, as the test set is always labeled as 0’s, you can see that yourself :slight_smile: the target labels don’t really ‘exist’ in the test set. See the docs:

https://docs.fast.ai/basic_train.html#Learner.get_

Preds will have the format of predictions, ground labels.

8

@muellerzr, you are right and this is such a stupid mistake I did there :confused: Thanks for pointing out.

However, even with the validation data set (ie. running learn.get_preds(ds_type=DatasetType.Valid) instead) my is_pred_inconsistent filter returns an image:

[('03_841_act_l3_h03.jpg', (tensor([0.5668, 0.4332]), tensor(1)))]

Or is the last entry of the second tuple supposed to be the ground truth here?

9

All good! :slight_smile: It looks like that second value (the tuple) still contains all the raw probabilities. Can I see your updated code? And then the last one (position 3) is your actual label in this case.