Retraining the bear classifier using multi-label classification

[Sturzgefahr]

Hi there,

I’m working through the fastbook notebooks on GitHub. Overall I’m really enjoying it, but I am having difficulties with the second “Further Research” exercise in Chapter 06:

• Retrain the bear classifier using multi-label classification. See if you can make it work effectively with images that don’t contain any bears, including showing that information in the web application. Try an image with two different kinds of bears. Check whether the accuracy on the single-label dataset is impacted using multi-label classification.

Has anyone been able to successfully complete this exercise? And if so, could they share how they did it?

I’ve been able to convert the original bear classifier to a multi-label classifier; but I can’t get it to work correctly with images that don’t contain bears: It will consistently classify everything as a bear. I’ve tried it with pictures that look nothing like bears (e.g., bicycles), but the classifier still thinks there’s a bear there, and with quite high confidence (often around 80-90%).

This is how I adapted the original DataBlock:

# cf. https://forums.fast.ai/t/how-to-use-bcewithlogitslossflat-in-lesson1-pets-ipynb/59146/5

def multi_l(l): return [l]

# amended code
bears = DataBlock(
    blocks=(ImageBlock, MultiCategoryBlock), # changed to MultiCategoryBlock
    get_items=get_image_files, 
    splitter=RandomSplitter(valid_pct=0.2, seed=42),
    get_y=Pipeline([RegexLabeller(pat = r'\w+\/([a-z]+)\/\d+\.[jpg|jpeg|JPG]'), multi_l]),
    item_tfms=Resize(128))

And this is the basic learner I used:

def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
    "Compute accuracy when `inp` and `targ` are the same size."
    if sigmoid: inp = inp.sigmoid()
    return ((inp>thresh)==targ.bool()).float().mean()

learn = cnn_learner(dls, resnet18, metrics=partial(accuracy_multi, thresh=0.9))
learn.fine_tune(4)

The main things I’ve tried are: changing the threshold of accuracy_multi (all the way up to .9999); using deeper versions of ResNet; and freezing the model for multiple epochs, unfreezing it, and then training for one more epoch.

However, nothing seems to work, and my model is unable to recognize the null class - i.e., images without any bears.

Does anyone have any suggestions?

Cheers,

MC

[John_F_Nash_Jr (John)]

I think that this is only possible by

• adding a fourth non-bear/garbage class to the training of the model or
• increasing the threshold during training the model

My attempt of training a multilabel model on the three classes (black, grizzly, teddy) and then predicting

• a test case picture that was completely white gave me the forecast teddy bear with 98% probability (i.e. something that I expected because teddy bears are photographed in a white box and real bears in nature).
• a test case picture of a house gave me [False, False, False]), tensor([0.0227, 0.3498, 0.1592]) when trained with a threshold of 0.8 and a black bear forecast tensor([ True, False, False]), tensor([0.5616, 0.2759, 0.1570])) when trained with a threshold of 0.2.

This should be one of the most important issues. After having trained a bear classifier, I cannot assume to be given images of bears only. I was very surprised that the Howard/Gugger “somewhat” claim in the introduction of chapter 6 of their book that multilabel classification would be a method for novelty detection, i.e. classifying a class that the model has not been trained on. How this should be possible is not referred to at all.

[Sturzgefahr]

Thanks for your reply.

I haven’t worked on this problem for a while, so my memory of it isn’t so fresh.

In regards to your first point, from what I remember, it’s not common practice to train a null class (i.e., a non-bear class). I think the issue is what to populate this class with - obviously the answer is anything but bears; but just using random images isn’t going to teach the classifier anything, since there won’t be any patterns in these random images for the classifier to learn.

Adjusting the threshold seems like a more reasonable approach; but the classifier I trained often assigned high probabilities to everything it saw - bear or not, so this didn’t work for me, either.

I haven’t worked on this problem for some time, so I’m afraid I don’t have much to add. If I come back to it and discover anything new, I’ll add more to this thread.

Thanks again for your response.

[jimmiemunyi]

Hey @Sturzgefahr, I need help with this part:

@Sturzgefahr

get_y=Pipeline([RegexLabeller(pat = r'\w+\/([a-z]+)\/\d+\.[jpg|jpeg|JPG]'), multi_l]),

My data is in folders, i.e:

• teddy
• black
• grizzly

the function
get_y = parent_label

doesn’t work anymore with MulticategoryBlock

which function should I use for get_y to get the parent names from the folders?

[Sturzgefahr]

@jimmiemunyi

Using Pipeline and RegexLabeller worked for me. I just got the path for one of my images and went to https://regex101.com/ to find a regex that would extract the label from the path. If you’re using a Windows machine don’t forget to change the back slashes to forward slashes.

You might also want to check out this thread, which specifically talks about formatting labels for MultiCategoryBlock. MultiCategoryBlock needs the labels to be in lists, which is why I used the function def multi_l(l): return [l] within the Pipeline.

I actually haven’t worked on this project for over a month, so I’m having difficulties remembering exactly what I did. But my bet is that you just need to use a regex.

[arampacha]

Hello. That’s pretty cool how Sturzgefahr figured out a way to use Pipeline and RegexLabeller for the task. His approach is really flexible.
But there is a simpler way do get labels by getting parent folder name from Path object. Just like it’s done in parent_label function. You need only small modification to make it return list:

def parent_label_multi(o):
    return [Path(o).parent.name]

Now you can use it setting:

get_y = parent_label_multi

PS: You can check out source code of original parent_label typing parent_label?? in jupyter. It’s often useful when you need to tweak existing code for your purpose.

[jimmiemunyi]

@arampacha

But there is a simpler way

thanks, its actually pretty easier.

[Sturzgefahr]

@arampacha

I’m glad you found a simpler way to do this. Dealing with regexes is always a hassle for me.

[nn.Charles]

Hello. I used the same approach. One thing I would add is to make sure to parent_label_multi returns a list. Otherwise, the DataBlock with MultiCategoryBlock will not understand what to do.

[tsumabuki1453]

Hello everyone

I have same problem. My multi-classifier can not recognize ‘not bear’. For my classifier, everything looks like bears with high confidence

I also tried two things.
1.High thresh in multi-accuracy(0.99)
2.Adding ‘na’ class by multicategoryblock(addna=True)

Please tell me what can I do for

thanks!!

[klty0988 (Kenneth)]

Hi @tsumabuki1453, thanks for bringing this up. I am similarly encountering the same problem. For an image that is clearly not a bear, I also get a high probability for it being a teddy bear (0.958) for the following whisky photo:

In addition, my threshold vs accuracy plot gives me this, where the higher threshold leads to higher accuracy all the way to threshold of 1. This feels odd to me, so just wanna check that you guys also get this curve?

Bumping this thread up as this multi-label bear classification problem is indeed rather interesting and yet to be fully explained.

Thanks!

[Rccsv (Rocco Salvetti)]

@tsumabuki1453 in the DataBlock I have used MultiCategoryBlock(addna=True) but that won’t run. Am I missing something?
Thank you.

[raulkite (Raul)]

Any advance with this. I’m in same place

Regards

Raul

[Conwyn]

Hi Paul
I can not get not-a-bear working either.
.
One thing which seems to work is to create a folder of squares. So start with a square and randomly divide into one square or four squares. Repeat recursively. Stop at one pixel. Whenever you choose one square not four squares then assign a incrementing number. At the end of the process assign a random colour to each number. These unreal squares seem to match the not a bear category. I found using 256x256 images making the 1 to 4 as 1/16 for one square,

Regards Conwyn

[jerry_frncs (Jerome Francis)]

Hi all,

I was working on this recently, and i got a few results where i was getting ’ [ ] ’ class (ie, not a bear) for a bicycle image using a resnet34 model trained on bears (with grizzly, black, teddy as classes).

For example:
Test IMAGE :

Resutlt for this image (using learn.predict()) : ([], tensor([False, False, False]), tensor([0.4861, 0.2155, 0.3676]))

But I also got this result for the below image:
Result (using learn.predict()) : ([‘black’, ‘teddy’], tensor([ True, False, True]), tensor([0.7204, 0.0986, 0.7658]))
IMAGE:

Info:
Arch: resnet34
metric : [partial(accuracy_multi, thresh=0.85), RocAuc()] # set as 0.85 using graph
Discriminative learning used
Getting 98.6667% accuracy, roc_auc_score=1.0 after 15epochs with unfreezing.

Question:

• What is the conversion rule from the probabilities to (True/False) values in label tensor? (This maybe a basic theory ques)
• if there is a rule, is there any param value that affects this rule?

Regards,
Jerome

[Kerner (Raviv Kerner)]

Hi all,
I have also tried to experiment all suggestions in the further research on a classification of different images (Tank classifier: Merkava MK4 tank, M1 Abrams tank and a water tank) and got the same results.

Actually training on the exact same data set made both models are quite similar:

1. Metrics - both models are around a 7% error rate (actually adjusting the multi label threshold increases its accuracy but I believe the difference is more connected to process randomness and different metric functions than the models).
2. Top loses - looks like the models are having problems on the same images.
3. Out of scope train images - small random check, the multi label model didn’t show any advantage in identity the images were out of scope or label two different images as we might expect (both models had the same mistakes)

I guess that at the end, the model will be able to learn as good as the examples he learn during training and as we didn’t train the multi label model with the relevant examples, in this case, it couldn’t out perform the multi class model.

Here is a post about my full solution with all the experiment results:
https://ravivkerner.github.io/blogtest/2021/11/23/tf.html

[JumpyJason]

Maybe it’s because we are all training the models using images that only contain one bear at a time. I’m facing the same problem, and maybe using good multi-labeled training data can solve this problem, but it’s too much work for a beginner such as myself.