Custom metric in fastai2

Part 1 (2020) Non-beginner
1

I am not sure I get how to implement a fully custom metrics in fastai2.

Here the context.
I am using this Kaggle competition as a playground, and the evaluation metric is

mean column-wise ROC AUC. In other words, the score is the average of the individual AUCs of each predicted column

I implemented the metric using the magic of callbacks, like this:

class ColumnWiseRocAuc(Callback):
    def begin_epoch(self):
        self.targets, self.probas = Tensor([]), Tensor([])    
    
    def after_batch(self): 
        preds = F.softmax(self.pred, dim=1).detach().cpu()
        y = self.y[:, None].cpu()
        y_onehot = torch.FloatTensor(len(y), self.dls.c)
        y_onehot.zero_()
        y_onehot.scatter_(1, y, 1)
        
        self.probas = torch.cat((self.probas, preds))
        self.targets = torch.cat((self.targets, y_onehot))
        print(preds[0], y_onehot.shape, self.probas.shape, self.targets.shape)
    
    def after_epoch(self):
        auc = 0.0
        for i in range(self.dls.c):
            auc += roc_auc_score(self.targets[:, i], self.probas[:, i])
        print(auc/self.dls.c)

which yields, the following:

image
image748×171 6.21 KB

How do I turn the above callback into a metric, e.g. add it to the progress bar alongside accuracy?
I am looking into AccumMetric, but I am not sure where to go next.
Can you guys provide any guidance please?

1 Like
2

For that you should not inherit Callback but instead Metric. Or most probably use AvgMetric, something like:

tst = AvgMetric(lambda x,y: (x-y).abs().mean())

Take a look here for examples

2 Likes
3

Here is my implementation of this metric:

def _accumulate(self, learn):
    #pred = learn.pred.argmax(dim=self.dim_argmax) if self.dim_argmax else learn.pred
    pred = learn.pred
    if self.sigmoid: pred = torch.nn.functional.softmax(pred) #hack for roc_auc_score
    if self.thresh:  pred = (pred >= self.thresh)
    targ = learn.y
    pred,targ = to_detach(pred),to_detach(targ)
    if self.flatten: pred,targ = flatten_check(pred,targ)
    self.preds.append(pred)
    self.targs.append(targ)

AccumMetric.accumulate = _accumulate

def RocAuc(axis=-1, average='macro', sample_weight=None, max_fpr=None,multi_class='ovr'):
    "Area Under the Receiver Operating Characteristic Curve for single-label binary classification problems"
    return skm_to_fastai(skm.roc_auc_score, axis=axis,
                         average=average, sample_weight=sample_weight, max_fpr=max_fpr,flatten=False,multi_class=multi_class,sigmoid=True)
3 Likes
4

This is great! Thanks so much.
I had completely overlooked the power of AccumMetric.
I was basically re-implementing its logic in my Callback, with no need to do that.

5

Hey guys!

When it comes to binary classification with the tabular_learner I came across some problems with the accuracy metric when using mse_loss. So here’s how I made use of the AccumMetric for the tabular learner:

Create custom metric

import sklearn.metrics as skm

def _accumulate(self, learn):
m = nn.Sigmoid()
pred = learn.pred
pred = torch.round(m(pred))
targ = learn.y
pred,targ = to_detach(pred),to_detach(targ)
self.preds.append(pred)
self.targs.append(targ)

AccumMetric.accumulate = _accumulate

def BinAccu():
return skm_to_fastai(skm.accuracy_score)#>

binaccu = BinAccu()
learn = tabular_learner(dls, layers=[128, 8, 128], n_out=1, metrics=[binaccu])

Here’s a small blogpost for how to implement this on tabular data: example

1 Like