0.50 roc_auc_score from fastai learner

I have some FastAI code with a DataLoaders and a learner where I set the metric to RocAucBinary for a binary classification task. However, when I run “fit_one_cycle” the roc_auc_score column always return exactly 0.50. But when it’s done training and I call get_preds() and use roc_auc_score on the results I get a different score, 0.614.

Any idea why this is happening? Am I using roc_auc_score incorrectly? I use the mid-level API to create a custom dataset with my dataloader, perhaps that may be the issue?

dls = DataLoaders.from_dsets(train_ds, valid_ds, bs=32)
dls.c = 1

learn = cnn_learner(dls, resnet18, loss_func=BCEWithLogitsLossFlat(), metrics=RocAucBinary())

learn.fit_one_cycle(2, 1e-3)
epoch 	train_loss 	valid_loss 	roc_auc_score 	time
0 	0.807268 	0.705478 	0.500000 	03:22
1 	0.645445 	0.633613 	0.500000 	01:50

from sklearn.metrics import roc_auc_score
valid_predicts, other = learn.get_preds()
roc_auc_score((valid_predicts >= .5), other)

1 Like

The issue, I believe, stems from misusing roc_auc_score: Its first argument is the target values, the second one is your model’s predictions. The correct code would thus be roc_auc_score(other, valid_predicts). You would most certainly reach the same score as fastai, but I’m unsure why your AUC remains constant while your loss is decreasing.

Hopefully this helps!