Confused by "outputs" from learn.predict(img)

So running this on my own dataset …

pred_class, pred_idx, outputs = learn.predict(img)
# outputs => tensor([15.9323,  0.1261])

The documentation describes outputs as probabilities … but these don’t look like probabilities.

Am I missing something?

1 Like

From the code snippet of predict it only looks like raw scores -

You may need to do -

import torch.nn.functional as F
F.softmax(outputs)

To get probabilities. But I agree either the code in the predict function needs to be updated or the doc string in the function needs to be updated. Will wait for what others think or if I am missing something.

5 Likes

Here is where it gets more strange …

When I do fastai the hard way (FATHW), I get different probabilities than when I do it using the provided helper methods.

FATHW:

train_tfms, val_tfms = get_transforms()
img = apply_tfms(val_tfms.append(normalize_funcs(*imagenet_stats)), img, size=224)
backbone = create_body(models.resnet34(pretrained=False), -2)
head = create_head(num_features_model(backbone) * 2, 2)
m = nn.Sequential(backbone, head)
m.load_state_dict(torch.load(PATH/'models/stage-2-34.pth'))
m.eval()
log_probs = m(img.data.unsqueeze_(0))
print(log_probs.data)
# => tensor([[-1.9587,  2.3774]])

torch.softmax(log_probs.data, -1).numpy()
# => array([[0.012919, 0.987081]], dtype=float32)

… when I do it with the helper methods I see

classes = ['in', 'out']

data2 = ImageDataBunch.single_from_classes(PATH, classes, tfms=get_transforms(), size=224).normalize(imagenet_stats)
learn = create_cnn(data2, models.resnet34, pretrained=False)
learn.load('stage-2-34')

pred_class, pred_idx, outputs = learn.predict(img)
pred_class, pred_idx, outputs
# => ('out', tensor(1), tensor([2.0508e-02, 2.6639e+01]))

torch.softmax(outputs, -1)
# => tensor([2.7533e-12, 1.0000e+00])

Any ideas as to what is going on?

-wg

Could it just be random weights? Like if others start closer to real result and that way converge faster in those steps.

The only thing I can think of is that the image is augmented differently each time it is passed to a model. The two cases you present got different augmented versions of the image.

You could check this by running the same image through the same model version several times.

But there shouldn’t be any augmentation at inference time … only during training.

There are no random weights … its using a fully trained model via learn.load

I do that but the different results still don’t make sense to me yet …

outputs 
#=> tensor([3.1907e+02, 9.0539e-03])` ... 

F.softmax(outputs, -1)
#=> tensor([1., 0.])

… but, when I run my custom code, I get values that look more sensible to me

m.eval()
log_probs = m(img.data.unsqueeze_(0))
#=> tensor([[ 4.1487, -3.3426]])

torch.softmax(log_probs.data, -1)
#=> tensor([9.994424e-01, 5.576377e-04])

Still can’t figure out why they are different or which to trust :slight_smile:

Not true: search for TTA() (however it’s not the present case).

As you guys correctly figured out, you were looking at logarthms of probabilities. No wonder that to get the actual probs, you need to exponentiate those values (softmax & co).

1 Like

At inference time you are predicting a single image, why would you apply any any transformation whatsoever? The goal of transforms is to provide more looks at your training examples during training.

Predicting over slight variations of the test image, and then average those predictions. It gives better results than a single prediction.

I’m interested to know if you have resolved the discrepancy between your custom code and the other code. Are you able to get the same answer?

No I haven’t.

The results I’m seeing from my code look more like what I am expecting. Perhaps @jeremy can shed some light on what is going on and which is to be trusted.

As @balnazzar mentioned, I suspect you’re just missing a log().

Yah I tried …

classes = ['in', 'out']

data2 = ImageDataBunch.single_from_classes(PATH, classes, tfms=get_transforms(), size=224).normalize(imagenet_stats)
learn = create_cnn(data2, models.resnet34, pretrained=False)
learn.load('stage-2-34')
pred_class, pred_idx, outputs = learn.predict(img)

torch.softmax(log(outputs), -1)

The results are still inconsistent (although much closer). My OCD self is driving me crazy because it just seems like either approach should yield the same result … but perhaps that isn’t the case for some reason unbeknownst to me.

2 Likes

Do not tag @balnazzar!!

Ahahah! Great joke! :rofl::rofl::stuck_out_tongue_winking_eye:

Probably a silly question with a simple answer, but when I do learn.predict(img) on single images it calls up a pbar and I can’t seem to find how to prevent that. Is there a way?

Oh wow. I think I missed the log() when using output and torch softmax, too! Now the results look much more what I would expect!

If have this now in my code:

pred_class, pred_idx, outputs = learner.predict(img)
probs = torch.nn.functional.softmax(np.log(outputs), dim=0)]
2 Likes

I don’t know, embarrassingly I’m not on fastai.v1 yet. You’re probably way ahead of me and I need to catch up :sweat: