I was wondering if anyone had tips on how to integrate the following defined loss function in the fastai framework language model “mean column-wise ROC AUC”. Should we use something like sklearn’s http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score and then for Y andY_true pass 2 matrices containing all the columns for which the above loss function applies or a different approach is needed? Seems this function supports only 2 vectors, not 2 matrices, but we could do it per column and then avg?
The one you link to works with multiple labels. I’ll copy in how I use it with fastai lib when I get to my computer later today.
You need at least one positive example per class for it to work so you might need a large batch size for the validation set.
This is what I pass in metrics array to fit:
def roc_auc(preds, y): 1 return metrics.roc_auc_score(y.data.cpu().numpy(), np.exp(preds.cpu().numpy()))
metrics is just
from sklearn import metrics)
Got it. Thanks a lot!
This is the final code I have since it seems that fastai doesn’t automatically one hot encode the Ys:
def roc_auc(preds, y): preds = np.exp(preds) #conv from logs exp = V(y).data.cpu().numpy() #predicted category ID #batch X num_classes bs = preds.shape #batch size nclass = preds.shape #size to determine length of Y one hot encoding y = np.zeros((bs, nclass)) y[np.arange(bs), exp] = 1 #one hot encode Ys return metrics.roc_auc_score(y, preds, average="micro")