About the relationship between losses and probabilities (and other related stuff)

Hi. I’m playing with a multilabeled dataset, and since the interpretation api seems not to work with this particular kind of classification, I’m trying to implement it from scratch, but I need to know a bit more about fastai internals. Let me elucidate:

We start with an instance of ClassificationInterpretation and save the top losses along with their indexes.

a = learn.interpret(tta=True)
losses, idxs = a.top_losses(10)

Both are matrices with the number of classes as row dimension, and the number of samples in the validation set as row dimension. In my case this amounts to 752x15.

Let’s look at losses's first rows:

As you may see, the matrix is sorted along the row axis (according to the indexes in idxs, see below), but there it is not sorted along the column axis and that made me think that its row are in correspondence with the rows in the validation set valid_ds.

question: does this correspondence actually stand?

Now, let’s select one row both losses and idxs:

As you may see, it is reporting the best loss on class 1, and thw worst on class 14.

question: how should we construe that information?

One possible answer would be that since we got the best (least) loss upon class 1, the model is predicting that sample belonging to class 1.

But look at what we got in the corresponding entries of the probabilities matrix as well as the array of predictions:

The model is predicting the 14th class for that particular sample, and this seems to contradict my conjecture above. Note also that, on the other hand, class 1 got a non-negligible prob, with a good 12%.

Either the top losses matrix and the probabilities matrix are not in correspondence by rows, or I’m missing something fundamental.

May you clarify?

1 Like