Weird confusion matrix in chapter 6

Part 1 2022
1

Hi. I’m following the “further research” question in chapter 6 of the book, trying to build a multi-label bear classifier. My code are as follows:

!pip install fastbook
from fastbook import *
from fastai.vision.all import *
bear_types = 'black', 'grizzly', 'teddy'
path = Path('bears')
path.mkdir(exist_ok=True)
for o in bear_types:
    new_path = path / o
    new_path.mkdir(exist_ok=True)
    results = search_images_ddg(f'{o} bear')
    download_images(new_path, urls=results)
    print(o, "download completed.")
fns = get_image_files(path)
failed = verify_images(fns)
failed.map(Path.unlink)

This is the same code mentioned in chapter 2 when building a simple bear classifier. But when I am trying to prove my skills by building a multi-labeled one, something troubles me:

dblock = DataBlock(blocks=(ImageBlock, MultiCategoryBlock),
                   get_items = get_image_files, 
                   get_y = lambda o: [parent_label(o)],
                   item_tfms=RandomResizedCrop(224, min_scale=0.5),
                   batch_tfms=aug_transforms(),
                   splitter=RandomSplitter(valid_pct=0.2, seed=42))
dls = dblock.dataloaders(path)
learn = vision_learner(dls, resnet18, metrics=partial(accuracy_multi, thresh=0.2))
learn.fine_tune(3)
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()

Because this is a multi-label classification problem, I slightly changed the get_y method. When I called dls.show_batch() it seems that all is right. But look at the confusion matrix!

No teddy bears! What’s happening here? Can anyone help me?

3

Hello,
first of all I’m no expert but generally your approach seems fine. If I’m understand the documentation correctly MultiCategoryBlock is for items that can have two categories. With your bear example, grizzly bears are both “Bear” and “NotForCudling” but on the other hand teddy bears are both “Bear” and “DefinitelyForCudling”. They each have two categories.

In your example try using CategoryBlock which is for single-label category. Your category is “Bear” eg. the type of bear you are trying to classify.

Let me know if it helped!

4

Thanks for your effort! I’m sure there is data in the validation set of the model, and the confusion matrix should be based on that. But why are there no actual “teddy” bears in the matrix? Being multi-labeled shouldn’t be a problem for the confusion matrix, and I’m confused on that.

5

I copied your code and did some test and I think I have figured out your problem. After some digging around I was able to identify two main problems:

MultiCategoryBlock  with combination of get_y = lambda o: [parent_label(o)]

Provides correct labels but for some reason doesn’t take into account the last one as you mentioned.

I changed then changed get_y to parent_label but that with combination of MultiCategoryBlock screwed labels and actually parsed them by each letter and turned those into categories.

Then I changed get_y to get_y = parent_label, and kept CategoryBlock, made corresponding changes to metrics and my confusion matrix is now correct and provides values for all categories. Here is a full code:

dblock3 = DataBlock(blocks=(ImageBlock, CategoryBlock),
                   get_items = get_image_files, 
                   get_y = parent_label,
                   item_tfms=RandomResizedCrop(224, min_scale=0.5),
                   batch_tfms=aug_transforms(),
                   splitter=RandomSplitter(valid_pct=0.2, seed=42))
dls3 = dblock3.dataloaders(source = path)

Leaner:

learn = vision_learner(dls3, resnet18, metrics=accuracy)
learn.fine_tune(2)
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()

I do not know why your first method didn’t work and why the labels got screwed up, but dls.train.show_batch(max_n=6, nrows=1) is really great for debugging issues with data as it shows you the data that is going to the learner.

Hope this helped

1 Like
6

I’m trying out multi-label classification and changing the MultiCategoryBlock back to CategoryBlock is not my intention. But I am genuinely grateful for you putting so much effort in this. I tried show_batch and teddy bears actually are in the training data.

7

Can you show a screenshot of how the batch looks? Are the labels messed up like in my case?

8

if this helps: (still the multi-labeled version)
image