Cats vs Dog with Multi Cat

After going through the mult-cat tutorial, I decided to create a multi-cat model for our cats vs dogs dataset.

Here it is.

I thought it will give no predictions (lower probabilities), if I uploaded non cat/dog image. But it seems like that’s not the case.

Is it because it haven’t see any other images?
Will it work if I create a new category called others and use images from a dataset like ATLAS?
(Anyway, going to try this soon)

Not sure why it’s giving high probs for cat/dog. Perhaps you can try hooking in GradCAM to see which part of your test image (non-dog/cat) is giving high activations for dog/cat?

As for the predictions, I think you will need to do some trial-and-error to decide on a good ‘threshold’ probability to use. The threshold should also form part of the accuracy metric you look at, to inform your training. @muellerzr’s notebook here shows an example of the threshold accuracy metric.

Yijin

Although it is giving you 0.7458 it is still nowhere near as high as your values for the dog and cat in your notebook and I would guess nowhere near as high as if you used a CatergoryBlock with a softmax. Have you tried looking at the probabilities you get when classifying your validation set to get an appropriate threshold?

1 Like

Change the loss function to bcewithlogits, this has been discussed before in the forums also in the previous version of the course. Zack also had posted a notebook to showcase this.

1 Like

Cool. Will try that in a bit.

Yeah! That’s a good suggestion.
With earlier prop is almost close to 1.

Will look into GradCAM a bit.

Yeah! playing with threshold is a good idea.
Thanks for the notebook reference.

Using MultiCategoryBlock should do this automatically. It definitely used to and from the output I think it still does (if not then the probabilities (0.7006 and 0.7458) would add to 1).

I did some experiments on this and I think threshold can make a big difference.
But I think the notebook posted by @utkb is somewhat incorrect because it was only used in accuracy. Even that, I think threshold of 0.1 is not correct.

I think we need to do use the threshold when predicting. But I couldn’t find a way to do it. So, I did this.

(Click here for the full notebook)

Then I tried to load the pascal dataset and try to predict on that.
I couldn’t find a way to do a batch prediction and so I read some source code and did something :smiley:

See, there’s something wrong here. Could you help me to dig into this?

Thanks.

IIRC there’s a with_decoded parameter to predict. This should apply the LossLogits decode which should help. But I don’t believe I found you need to. It should stick to the thresh you set during training (as it’s on the metrics side not the loss function side)

1 Like

I had a quick look through the notebook and I don’t think there is problem with the predictor, but it is not clear as you have both

has_predictions = (preds > 0.9).float().sum(dim=1) != 0

and

has_predictions = (preds > 0.9).float().sum(dim=1) == 0

and it is not clear which one is run before you get your predictions. Either way I think

preds[0]

should be

preds[img_preds_available[0]]

1 Like

Oh! This seems like a typo. Will check more.
Thanks.

This is something new to me. Will look at it.

Thanks.

I fixed that typo and it looks great.
Now images with predictions in the PASCAL dataset looks decent.

I think, I have to train a bit more and increase the threshold.
Thanks all.

Here’s the Notebook.

1 Like