How can we use sample-wise loss weights in fastai? Using class-wise weights for loss is extremely simple and straight-forward, but using sample-wise weights for loss seems to be a bit more involved when using fastai.
I have a weight associated with each sample in my training data set that I would like to use to weight the loss for that particular sample. This is accomplished easily enough outside of the fastai framework in 3 steps.
1.) defining a custom data set:
import torch.utils.data as torchdata class SampleWeightsData(torchdata.Dataset): ''' A dataset that will return x, y, and the weight for x ''' def __init__(self, full_df, in_cols, y_col, w_col): self.full_df = full_df self.x = torch.FloatTensor(full_df[in_cols].values) self.y = torch.LongTensor(full_df[y_col].values) self.w = torch.FloatTensor(abs(full_df[w_col] / full_df[w_col].max()).values) def __len__(self): return len(self.full_df) def __getitem__(self, i): return self.x[i], self.y[i], self.w[i]
2.) Using a custom loss function:
import fastai.layers as fast_layers class CEFSampleWeightLoss(nn.Module): def __init__(self): super().__init__() self.base_loss = fast_layers.CrossEntropyFlat(reduction='none') def forward(self, x, y, w): loss = self.base_loss(x, y) loss *= (w**2) return loss.sum()
3.) a simple modification to the standard fitting process so that these weights are used by the custom loss function:
... for batch_num, row in enumerate(dl): x = row.to(device) y = row.to(device) w = row.to(device) # Clear the gradients calculated in the last mini-batch if training: opt.zero_grad() # Do forward pass, calculate loss nn_outputs = model(x).to(device) loss = loss_fx(nn_outputs, y, w) ...
I am unsure how to accomplish this same objective in the framework of a fastai learner. My custom loss function is expecting 3 parameters to its forward function instead of the traditional 2, so I don’t see how I can use the learner’s fit function. Further, I need to get my databunch to return the weights per sample along with the features and label. FYI, my data is tabular (though I am only using continuous features currently, no categorical), so I will want to be using tabular learners and tabular lists. I apologize if this is a trivial matter; I am new to using fastai (and relatively new to deep learning generally) and haven’t ever made use of fastai’s callbacks (which I have a hunch will be helpful here). I’ve also scoured the forums and been unable to find much on this topic.
I would just do this manually and use bits and pieces of fastai’s genius where applicable, but I can’t seem to replicate my baseline results without using fastai’s learner. I have a separate topic about this.