Expected prediction probability to be higher, but it's super low

Part 1 2022
1

Hello everyone, I am trying to fine tune resnet18, to differentiate between images of Marvel and DC super heroes. Here is a screenshot from the Jupiter notebook,

image
image1666×1144 71.8 KB

Anyone know why the prob for Marvel prediction is so low ??
Datablock has 600 super hero images, half’s between DC and Marvel, and is then used to fine_tune resnet18 for 3 epocs.

Notebook: https://www.kaggle.com/code/vipulbhj/marvel-or-dc/notebook

Some Context:

I know super hero images look extremely similar, and the data has features that meshed together, it’s by design. I wanted to see how models would behave when domains are not too clearly separate. I would try to add more images and see what happens, but what I am curious about is, why is it 0.0007 and still identifies it as marvel, which would mean, the DC prob is even lower, so if not any, what does it think that image looks like ?

What I would have expected it to do instead is, have a high prob and get things wrong, since both are so same, as in a general sense, they have same features so both would match the features in some way and form, and a large number of features will intersect, meaning higher probs with some wrong results.

Thanks for the help :smile:

2

There’s a tiny bug in your code: You are printing the wrong probability. Instead of taking probs[0] you need to take the probability of the predicted class.

Here’s the fix, I’ve printed the probs as well:

label,ll,probs = learn.predict(PILImage.create('marvel.jpg'))
print(f"This is a: {label} character.")
print(ll)
print(probs) # Added printing probs
print(f"Confidence Probality: {probs[ll.item()]:.4f}") #print the prob of the predicted label

The output on my machine:

This is a: marvel character.
tensor(1)
tensor([0.0028, 0.9972])
Confidence Probality: 0.9972
2 Likes
3

Thanks a lot, this was very helpful.

1 Like