Retraining the bear classifier using multi-label classification

Hi there,

I’m working through the fastbook notebooks on GitHub. Overall I’m really enjoying it, but I am having difficulties with the second “Further Research” exercise in Chapter 06:

  • Retrain the bear classifier using multi-label classification. See if you can make it work effectively with images that don’t contain any bears, including showing that information in the web application. Try an image with two different kinds of bears. Check whether the accuracy on the single-label dataset is impacted using multi-label classification.

Has anyone been able to successfully complete this exercise? And if so, could they share how they did it?

I’ve been able to convert the original bear classifier to a multi-label classifier; but I can’t get it to work correctly with images that don’t contain bears: It will consistently classify everything as a bear. I’ve tried it with pictures that look nothing like bears (e.g., bicycles), but the classifier still thinks there’s a bear there, and with quite high confidence (often around 80-90%).

This is how I adapted the original DataBlock:

# cf. https://forums.fast.ai/t/how-to-use-bcewithlogitslossflat-in-lesson1-pets-ipynb/59146/5

def multi_l(l): return [l]

# amended code
bears = DataBlock(
    blocks=(ImageBlock, MultiCategoryBlock), # changed to MultiCategoryBlock
    get_items=get_image_files, 
    splitter=RandomSplitter(valid_pct=0.2, seed=42),
    get_y=Pipeline([RegexLabeller(pat = r'\w+\/([a-z]+)\/\d+\.[jpg|jpeg|JPG]'), multi_l]),
    item_tfms=Resize(128))

And this is the basic learner I used:

def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
    "Compute accuracy when `inp` and `targ` are the same size."
    if sigmoid: inp = inp.sigmoid()
    return ((inp>thresh)==targ.bool()).float().mean()

learn = cnn_learner(dls, resnet18, metrics=partial(accuracy_multi, thresh=0.9))
learn.fine_tune(4)

The main things I’ve tried are: changing the threshold of accuracy_multi (all the way up to .9999); using deeper versions of ResNet; and freezing the model for multiple epochs, unfreezing it, and then training for one more epoch.

However, nothing seems to work, and my model is unable to recognize the null class - i.e., images without any bears.

Does anyone have any suggestions?

Cheers,

MC

1 Like

I think that this is only possible by

  • adding a fourth non-bear/garbage class to the training of the model or
  • increasing the threshold during training the model

My attempt of training a multilabel model on the three classes (black, grizzly, teddy) and then predicting

  • a test case picture that was completely white gave me the forecast teddy bear with 98% probability (i.e. something that I expected because teddy bears are photographed in a white box and real bears in nature).
  • a test case picture of a house gave me [False, False, False]), tensor([0.0227, 0.3498, 0.1592]) when trained with a threshold of 0.8 and a black bear forecast tensor([ True, False, False]), tensor([0.5616, 0.2759, 0.1570])) when trained with a threshold of 0.2.

This should be one of the most important issues. After having trained a bear classifier, I cannot assume to be given images of bears only. I was very surprised that the Howard/Gugger “somewhat” claim in the introduction of chapter 6 of their book that multilabel classification would be a method for novelty detection, i.e. classifying a class that the model has not been trained on. How this should be possible is not referred to at all.

Thanks for your reply.

I haven’t worked on this problem for a while, so my memory of it isn’t so fresh.

In regards to your first point, from what I remember, it’s not common practice to train a null class (i.e., a non-bear class). I think the issue is what to populate this class with - obviously the answer is anything but bears; but just using random images isn’t going to teach the classifier anything, since there won’t be any patterns in these random images for the classifier to learn.

Adjusting the threshold seems like a more reasonable approach; but the classifier I trained often assigned high probabilities to everything it saw - bear or not, so this didn’t work for me, either.

I haven’t worked on this problem for some time, so I’m afraid I don’t have much to add. If I come back to it and discover anything new, I’ll add more to this thread.

Thanks again for your response.

Hey @Sturzgefahr, I need help with this part:

My data is in folders, i.e:

  • teddy
  • black
  • grizzly

the function
get_y = parent_label

doesn’t work anymore with MulticategoryBlock

which function should I use for get_y to get the parent names from the folders?

@jimmiemunyi

Using Pipeline and RegexLabeller worked for me. I just got the path for one of my images and went to https://regex101.com/ to find a regex that would extract the label from the path. If you’re using a Windows machine don’t forget to change the back slashes to forward slashes.

You might also want to check out this thread, which specifically talks about formatting labels for MultiCategoryBlock. MultiCategoryBlock needs the labels to be in lists, which is why I used the function def multi_l(l): return [l] within the Pipeline.

I actually haven’t worked on this project for over a month, so I’m having difficulties remembering exactly what I did. But my bet is that you just need to use a regex.

2 Likes

Hello. That’s pretty cool how Sturzgefahr figured out a way to use Pipeline and RegexLabeller for the task. His approach is really flexible.
But there is a simpler way do get labels by getting parent folder name from Path object. Just like it’s done in parent_label function. You need only small modification to make it return list:

def parent_label_multi(o):
    return [Path(o).parent.name]

Now you can use it setting:

get_y = parent_label_multi

PS: You can check out source code of original parent_label typing parent_label?? in jupyter. It’s often useful when you need to tweak existing code for your purpose.

3 Likes

thanks, its actually pretty easier.

@arampacha

I’m glad you found a simpler way to do this. Dealing with regexes is always a hassle for me.

1 Like

Hello. I used the same approach. One thing I would add is to make sure to parent_label_multi returns a list. Otherwise, the DataBlock with MultiCategoryBlock will not understand what to do.

Hello everyone :grinning:

I have same problem. My multi-classifier can not recognize ‘not bear’. For my classifier, everything looks like bears with high confidence :joy:

I also tried two things.
1.High thresh in multi-accuracy(0.99)
2.Adding ‘na’ class by multicategoryblock(addna=True)

Please tell me what can I do for :slightly_smiling_face:

thanks!!

1 Like