Hello all, I built a model in fast.ai, which is classifying spare parts for big machines. To account for uncertainty or a ‘no category identified’-class I implemented this as a multi label problem (sigmoid) as recommended by @jeremy here.
To check the accuracy on the test set I first train my model using the real train/val sets. Then I create a new databunch in which I define my labeled test set as the new “fastai val set”, load my trained model and do prediction & accuracy etc. on that test set (now for fastai purposes defined as the val set) as recommended here.
Now I would like to check which images exactly are being misclassified to get an idea of which images the model is having problems with (right now I am at about 85% accuracy for the test set with 110 pictures). @muellerzr shows how this can be done quite easily with the ClassConfusion widget in this thread for softmax. Unfortunately, this doesn’t work with multi label problems and sigmoid. I also get an error if I want to plot the top losses:
---------------------------------------------------------------------------IndexError Traceback (most recent call last)
in ()
----> 1 interp.plot_top_losses(9)4 frames
/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in get(self, i)
73 def get(self, i)->Any:
74 “Subclass if you want to customize how to create itemifromself.items.”
—> 75 return self.items[i]
76 def repr(self)->str:
77 items = [self[i] for i in range(min(5,len(self.items)))]IndexError: index 640 is out of bounds for axis 0 with size 110
Is there any way to work around this and either print a list containing the filenames of the misclassified images or show the misclassified pictures for multi label classification?
I already saw in notebook 23_tutorial.vision that with fast.ai v2 we can do this quite easily with
interp = Interpretation.from_learner(learn)
interp.plot_top_losses()
but I would like to stay with fast.ai v1 for right now, if possible.
Thanks in advance and you help would be greatly appreciated!