Is there a way to get the pure probabilities for categories (without SoftMax)

Part 1 (2020)
1

I do not see the way to get raw(pure) probabilities for resnetXX based networks, for example Single Label classification. Is it possible?

2

I am not sure that what you are asking for makes sense. After softmax you have the probability that the image is of class x, given that the image falls into one of the given classes. You can extract the probabilities before softmax normalizes them. But the unnormalized probabilities do not mean “the chance this is a dog relative to all other images in the Universe.” They are still relative to the other classes.

What exactly do you mean by “raw (pure) probabilities”, and what do you want to understand from them?

3

I am looking a way to fight noise. When an image does not contain appropriate categories the net anyway gives me ‘correct’ probabilities. Getting ‘probabilities’ before softmax layer can help.

How to?

4

Take a look at Pytorch’s “forward hooks”. I have used them with success.

fastai also has a Hooks class with more sophistication. I think the fastai CAM visualization uses Hook’s to grab the activations of an inner layer. You might be able to find a code snippet that already does what you want.

Oh, here’s perhaps an easier way given that you want the last layer output. Replace learn.loss_fn with your own loss function that retains the model output activations and then passes them on to the original loss function. Exponentiate and you will have the “raw” probabilities.

However, I don’t think this approach will give you what you want. A fundamental assumption of machine learning (this type) is that the test distribution will be like the training distribution. If you give the model a category that it has never been trained on, there is no telling which of its features it will have learned are important during previous training. This is quite a deep topic. You can find several discussions on the forums about methods to detect out of distribution samples.

5

A simpler way of doing so is replacing it with BCELossLogits. Or, the simplest way is to make a fake activation function that just returns. This will override it using a softmax on get_preds. Something like:

def myFunc(x): return x

preds = learn.get_preds(act=myFunc)

The softmax is occurring because CrossEntopyLossFlat has an activation function inside of it which softmax’s. get_preds calls this, but if we override it it’ll be ignored.

1 Like