Hi,
Chapter 6 of fastbook (fastbook/06_multicat.ipynb at master · fastai/fastbook · GitHub) is suggesting accuracy_multi as a metric for a multi-label classification model, which is capable of assigning 0 or multiple labels to the same input image.
You can see the accuracy reaching 0.950877 at the end of fine_tune() for thresh = 0.2. And it can go higher into 0.96+ range if thresh = 0.5 is selected.
I am struggling to interpret the significance of these relatively high accuracy values. Does 0.96 supposed to mean “96% accurate”?
Looking at the implementation of accuracy_multi:
def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
"Compute accuracy when `inp` and `targ` are the same size."
inp,targ = flatten_check(inp,targ)
if sigmoid: inp = inp.sigmoid()
return ((inp>thresh)==targ.bool()).float().mean()
… the result of the function will be going up in two cases:
-
if targ[i].bool() == True, indicating that a corresponding label is assigned in the input dataset AND inp>thresh, indicating that the model prediction also assigned a corresponding label
-
if targ[i].bool() == False, indicating that a corresponding label is NOT assigned in the input dataset AND inp<=thresh, indicating that the model prediction also did NOT assign a corresponding label
There are 20 labels in this dataset:
len(dls.vocab)
--
20
Most images in PASCAL_2007 the dataset have just a single label assigned. Some have a few. None have anywhere near 20. This means that the accuracy will be quite high numerically (close to 1) as far as the model doesn’t assign too many labels to each image.
Here is, for example, the impact on accuracy of setting all predictions to 0, which would be equivalent to not assigning any labels to any images:
# First, calculate accuracy of the model predictions after fine_tune
preds, targs, decoded = learn.get_preds(with_decoded=True)
accuracy_multi(preds, targs, thresh=0.5, sigmoid=False)
---
TensorBase(0.9643)
zero_preds = torch.zeros(preds.shape)
accuracy_multi(zero_preds, targs, thresh=0.5, sigmoid=False)
---
TensorBase(0.9224)
The accuracy did go down from 0.9643 to 0.9224. But intuitively 0.92 is still a relatively high, indicating a decent prediction, which it is clearly not!
Am I misinterpreting the accuracy_multi approaching 1 as an indication of the model converging?
And how do I deal with the multi-label classification model not assigning any labels to my images, while the accuracy is close to 1 as I illustrated above?
PS: I am contemplating this because my own multi-label classification model (for a different dataset) is producing seemingly great results by quickly converging on accuracy of 0.97-0.98. But the classification results shown by learn.show_results() are awful, with most images not getting any labels assigned.