Custom Weighted Labels for Classication

Deep Learning
1

I could have sworn I saw a post where this was mentionted a long time ago but I cannot seem to find it. I am not sure what this technique is called and have been unable to search for it effectively.

What I am looking to do is instead of having my y labels be [0,0,0,0,0,1] as the goal of prediction (and what the loss function uses), I want it to be something like [0,0,0,0.1,0.1,0.8], so that if I get a category ‘close’ to the right answer it is penalized less than if I classify it completely wrong. For example, if I am predicting pet breeds:

  • If I pick the right breed that’s ideal
  • If the image is a cat and I pick the wrong breed of cat, that is penalized some
  • If the image is a cat and I guess that it’s some breed of dog, that is penalized more heavily than bullet 2

I am trying to figure out how I can use a custom defined matrix as my labels. I can’t seem to figure out either how to specify this and how to access and change the y matrix that has to be generated for the model to work generated. In the pets dataset, what I would like to do is change my y values so that if it is a cat and I predict the wrong breed of cat, that is a more acceptable error than if I predict a dog. My thought on how to do this would be to change the y matrix, though I am open to other ideas on how to get this result.

Here is the datablock and dataloader.

from fastai2.vision.all import *
path = untar_data(URLs.PETS) #Sample dataset from fastai2
dls = pets.dataloaders(path/"images")

pets = DataBlock(
    blocks = (ImageBlock, CategoryBlock),
    get_items = get_image_files,
    splitter= RandomSplitter(valid_pct = 0.2, seed=seed),
    get_y= using_attr(RegexLabeller(r'(.+)_\d+.jpg$'),'name'),
    item_tfms=Resize(460),
    batch_tfms=aug_transforms(min_scale = 0.9,size=224)
    )
dls = pets.dataloaders(path/"images")
2

Hi Enzo
I would directly create a loss. You can wrap the Cross Entropy (CE) loss or any other loss with a custom loss class that:

  1. As argument accept a loss function like CE.
  2. Calls the original loss with reduction=‘none’) to compute the loss for every sample. Fastai has a context manager to do that. I think that LabelSmoothing loss uses it. See it’s implementation.
  3. Multiply the loss of each sample by the factor you want. For example, 0.8 when the category is close. I would define these weights in a matrix of N labels x N labels that is passed to your custom loss.
  4. Do the loss mean / sum if reduction argument is set to ‘mean’/‘sum’.
  5. Return the loss
3

Thank you, this is very helpful. I see what I need to do, but I am now struggling with implementation of this. What I am struggling with seems like something that should not be that hard - though I am a bit stuck. I am fairly new to this and I appreciate the help.

I am looking at these loss functions, specifically the LabelSmoothing one. I can see where I need to modify the multiplier for what I am looking to do, but I cannot seem to find where I can get the order of the labels. I have 37 classes, so my output has 37 predictions, 1 per class. My Target tells me which of those is the ‘correct’ answer. However, in order to weight them differently, I need to be able to see what class is each of the 37 columns.

For example if I had only 3 classes, my output could be [0.25,0.2,0.55]. Are these scores relating to [Bulldog, Terrier, Maincoon], meaning the maincoon class has the highest score. Or is the order [Terrier, Maincoon, Bulldog] meaning the bulldog class has the highest score? I thought it should be super easy to find this information, but I cannot for the life of me find how to map the columns in output to what class they are scoring. I have tried looking in my dataload as well as the cnn_learner objects, and I am not sure where else to look.

The code I am looking at from fastai is below if that is helpful. The output and target I am referring to are the arguments in the forward function.

class LabelSmoothingCrossEntropy(Module):
    y_int = True
    def __init__(self, eps:float=0.1, reduction='mean'): self.eps,self.reduction = eps,reduction

    def forward(self, output, target):
        c = output.size()[-1]
        log_preds = F.log_softmax(output, dim=-1)
        if self.reduction=='sum': loss = -log_preds.sum()
        else:
            loss = -log_preds.sum(dim=-1) #We divide by that size at the return line so sum and not mean
            if self.reduction=='mean':  loss = loss.mean()
        return loss*self.eps/c + (1-self.eps) * F.nll_loss(log_preds, target.long(), reduction=self.reduction)

    def activation(self, out): return F.softmax(out, dim=-1)
    def decodes(self, out):    return out.argmax(dim=-1)
4

I believe I found it. learn.dls.vocab gets the class names.

1 Like
5

Yep! And vocab.o2i can be used directly if you want to work work labels instead of the encoded indices. :slightly_smiling_face: