Column-wise AUC loss function


(Feras) #1

I was wondering if anyone had tips on how to integrate the following defined loss function in the fastai framework language model “mean column-wise ROC AUC”. Should we use something like sklearn’s http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score and then for Y andY_true pass 2 matrices containing all the columns for which the above loss function applies or a different approach is needed? Seems this function supports only 2 vectors, not 2 matrices, but we could do it per column and then avg?


#2

The one you link to works with multiple labels. I’ll copy in how I use it with fastai lib when I get to my computer later today.

You need at least one positive example per class for it to work so you might need a large batch size for the validation set.


#3

This is what I pass in metrics array to fit:

 def roc_auc(preds, y):                                                                                                                                                                                         
  1     return metrics.roc_auc_score(y.data.cpu().numpy(), np.exp(preds.cpu().numpy())) 

(metrics is just from sklearn import metrics)


(Feras) #4

Got it. Thanks a lot!


(Feras) #5

This is the final code I have since it seems that fastai doesn’t automatically one hot encode the Ys:

def roc_auc(preds, y):
  preds = np.exp(preds) #conv from logs
  exp = V(y).data.cpu().numpy() #predicted category ID
  #batch X num_classes
  bs = preds.shape[0] #batch size
  nclass = preds.shape[1] #size to determine length of Y one hot encoding
  y = np.zeros((bs, nclass))
  y[np.arange(bs), exp] = 1 #one hot encode Ys
  return metrics.roc_auc_score(y, preds, average="micro")