AUC-ROC curve in fastai for prediction

Deep Learning
1

Hi everyone,

I have a image dataset with 2 classes, I splitted in two for training and test set. The images are divided in two folder (train and test, both the two folder have two subfolders, one for each class) I am using transfer learning with resnet 34. I am running my analysis in colab. In synthesis:

import numpy as np
from fastai.vision import *
from fastai.metrics import error_rate
np.random.seed(42)
data = ImageDataBunch.from_folder(train_dir, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)
learn = create_cnn(data, models.resnet34, metrics=[error_rate,accuracy], wd=1e-1)
learn.fit_one_cycle(4)
learn.unfreeze()
learn.fit_one_cycle(2, max_lr=slice(1e-5,1e-4))
learn.export()

I wanted to use the test set to make prediction and ultimately to plot a ROC curve and an AUC.

learn = load_learner(dataset_dir, test=ImageList.from_folder(test_dir))
preds,y = learn.get_preds(ds_type=DatasetType.Test)
from sklearn.metrics import roc_curve, auc
# probs from log preds
probs = np.exp(preds[:,1])
# Compute ROC curve
fpr, tpr, thresholds = roc_curve(y, probs, pos_label=1)
# Compute ROC area
roc_auc = auc(fpr, tpr)
print('ROC area is {0}'.format(roc_auc))

the result was NaN. Digging a bit I noticed that basically the y (the predicted class is always the same, the CTRL) and there is something not convincing with the raw prediction.

learn.predict(im)
Instead using learn.predict is predicting correctly the image class from the test set. Basically, predicting in loop each single images I have a better results as predictions. I created a panda.dataframe to store the torch tensors, but it starts to be messy and I am not fully convinced is the best approach.

My questions are:

  1. I think they I am doing something wrong in making the prediction for the whole test set and try to use it for obtaing AUC and ROC.
  2. How I can predict and plot a ROC curve in the best way?