TL;DR
How can we use sample-wise loss weights in fastai? Using class-wise weights for loss is extremely simple and straight-forward, but using sample-wise weights for loss seems to be a bit more involved when using fastai.
Verbose=True
I have a weight associated with each sample in my training data set that I would like to use to weight the loss for that particular sample. This is accomplished easily enough outside of the fastai framework in 3 steps.
1.) defining a custom data set:
import torch.utils.data as torchdata
class SampleWeightsData(torchdata.Dataset):
'''
A dataset that will return x, y, and the weight for x
'''
def __init__(self, full_df, in_cols, y_col, w_col):
self.full_df = full_df
self.x = torch.FloatTensor(full_df[in_cols].values)
self.y = torch.LongTensor(full_df[y_col].values)
self.w = torch.FloatTensor(abs(full_df[w_col] / full_df[w_col].max()).values)
def __len__(self):
return len(self.full_df)
def __getitem__(self, i):
return self.x[i], self.y[i], self.w[i]
2.) Using a custom loss function:
import fastai.layers as fast_layers
class CEFSampleWeightLoss(nn.Module):
def __init__(self):
super().__init__()
self.base_loss = fast_layers.CrossEntropyFlat(reduction='none')
def forward(self, x, y, w):
loss = self.base_loss(x, y)
loss *= (w**2)
return loss.sum()
3.) a simple modification to the standard fitting process so that these weights are used by the custom loss function:
...
for batch_num, row in enumerate(dl):
x = row[0].to(device)
y = row[1].to(device)
w = row[2].to(device)
# Clear the gradients calculated in the last mini-batch
if training: opt.zero_grad()
# Do forward pass, calculate loss
nn_outputs = model(x).to(device)
loss = loss_fx(nn_outputs, y, w)
...
I am unsure how to accomplish this same objective in the framework of a fastai learner. My custom loss function is expecting 3 parameters to its forward function instead of the traditional 2, so I don’t see how I can use the learner’s fit function. Further, I need to get my databunch to return the weights per sample along with the features and label. FYI, my data is tabular (though I am only using continuous features currently, no categorical), so I will want to be using tabular learners and tabular lists. I apologize if this is a trivial matter; I am new to using fastai (and relatively new to deep learning generally) and haven’t ever made use of fastai’s callbacks (which I have a hunch will be helpful here). I’ve also scoured the forums and been unable to find much on this topic.
I would just do this manually and use bits and pieces of fastai’s genius where applicable, but I can’t seem to replicate my baseline results without using fastai’s learner. I have a separate topic about this.