I have some FastAI code with a DataLoaders and a learner where I set the metric to RocAucBinary for a binary classification task. However, when I run “fit_one_cycle” the roc_auc_score column always return exactly 0.50. But when it’s done training and I call get_preds() and use roc_auc_score on the results I get a different score, 0.614.
Any idea why this is happening? Am I using roc_auc_score incorrectly? I use the mid-level API to create a custom dataset with my dataloader, perhaps that may be the issue?
dls = DataLoaders.from_dsets(train_ds, valid_ds, bs=32) dls.c = 1 learn = cnn_learner(dls, resnet18, loss_func=BCEWithLogitsLossFlat(), metrics=RocAucBinary()) learn.fit_one_cycle(2, 1e-3) epoch train_loss valid_loss roc_auc_score time 0 0.807268 0.705478 0.500000 03:22 1 0.645445 0.633613 0.500000 01:50 from sklearn.metrics import roc_auc_score valid_predicts, other = learn.get_preds() roc_auc_score((valid_predicts >= .5), other)