Plot_multi_top_losses() Pull Request

fastai fastai dev
41

Hi Nilesh, welcome in our community!

It is the probability that the model assigns to the “best” predicted class, that is, the class characterized by the highest probability. I used quotation marks since it is to be considered best in a somewhat bad sense.

I didn’t quite understand what you meant by “it seems like the probability of the actual class in the prediction”, that is, I’m not catching the difference between:

  1. probability of the predicted class
  2. probability of the actual class in the prediction

But just to clarify, let me say that, as you probably know, we don’t use a softmax in a multi-labeled setting. We use sigmoids cut off with a threshold. Every class that goes above that threshold should be accepted as a predicted class.
But as I implemented plot_multi_top_losses I somewhat thought it was best to pick the class with the maximum absolute probability, since I always intended to use it as a debugging tool, and the threshold is (regrettably) specified arbitrarily. If that “best” class is not amongst the actual ones (whatever their number), then that’s the key information for debugging, and it doesn’t matter so much if other classes would go above the threshold. It is not difficult to make it show the probs relative to all the classes above the thresh. Feel free to submit a PR, making it an optional flag, if you want.

Thanks!

42

Thank you Andrea for the detailed explanation. I understood your point.
Just to clarify what i meant by " actual class in the prediction", by actual class i meant the true label which would be different than the predicted label in case of wrong prediction.
So in the weather example of multilabel classification,
suppose our classes are [‘cloudy’, ‘rainy’, ‘sunny’, ‘hazy’].
the actual label is [0,0,1,0]
but our prediction logits are [0.9, 0.1,0.2,0.7]
here the actual class in the prediction is ‘sunny’ but predicted class in prediction is ‘cloudy, hazy’.

1 Like
43

Thanks @nilesh0109, it’s perfectly clear now.

In that case, plot_multi_top_losses would give you the prob relative to “cloudy”, even if “hazy” is well above the threshold and the actual class is “sunny”.

Another example can be found in this thread (post #24), where you have the probability relative to the (wrongly) predicted “healthy”.

1 Like