Different probabilities for the same picture

Part 1 (2018)
1

Hi everyone,

I have noticed a very strange behavior in my model. When I print the probabilities for photos from the validation set using :
plot_val_with_title(rand_by_correct(True), "Correctly classified")

I get different probabilities, than when I score the exactly same pictures myself, using the following code:
image_path = '/home/ubuntu/notebooks/photos/valid/group1/979794_3.jpg'
trn_tfms, val_tfms = tfms_from_model(arch,sz,aug_tfms=transforms_side_on, max_zoom=1) # get transformations
image = open_image(image_path)
im = val_tfms(image)
learn.precompute=False # We'll pass in a raw image, not activations
log_pred = learn.predict_array(im[None])
prob =np.exp(log_pred)
pred = np.argmax(prob, axis=1)
print(pred[0])

For example one probability is equal to 0.9, and the second 0.99, or 0.74 vs 0.57.

Its all happening in one notebook, on the same model instance and using the same settings.

I’ll be very grateful for some feedback as this is totally unexpected behavior and leads to incorrect probabilities for new (out of the validation/train sets) pictures.

Regards,
Magda

2

Interesting. Would it be possible to put your notebook with outputs that show different prediction results in gist.github.com and share here? To create a gist, drag your notebook in a new gist.github.com

3

I cannot get it to work with ipython notebook, so I added python script.

Below print screen of different results:

image
image.png1602×504 55.8 KB

4

Hi @MagdaG - Thanks for posting this. I think you found a fast.ai bug. Basically, when the model is in Training mode, it has Dropouts enabled. During Evaluations, dropouts are not performed. To do this, we switch the model to model.train() and model.eval().

This is taken care by Fast.ai Learner during predict and fit process. Typically we use predict_dl or TTA to predict and it’s takes care of switching the model to model.eval(). This might have been missed in predict_array method of learner.

If my hypothesis is correct, you could fix it by -

learn.model.eval()
log_pred = learn.predict_array(im[None])

Please confirm if this fixes your problem of getting different results. I will create a pull request to add this inside the predict_array method.

2 Likes
5

Pull request created to fix this problem - https://github.com/fastai/fastai/pull/224

Appreciate if you can confirm if it fixes on your end using the above method.

2 Likes
6

Hello Magda,

I have just realised that this one is not correct
pred = np.argmax(prob, axis=1)

You need to have pred = np.argmax(prob, axis=0)
if you want an array of size (val_entries x nr_classes)

With axis=1, you will have always
5 x nr_classes, where 5 is the augmentations created by TTA.

Your initial probs.shape is 5 x val_entries x nr_classes and you want the maximum across the first size.

7

Actually you need to make a mean of the log_preds, before the argmax, so finally the axis will stay 1

Here it is the code

log_preds,y = learn.TTA()
probs = np.mean(np.exp(log_preds),0)
preds = np.argmax(probs, axis=1)

Regarding the probabilities results, they seem kind of the same with small differences - which maybe are coming from the augmentation and meaning

image