Using AUC as metric in fastai

Is it a binary classifier?

I think I just made the exact same mistake :slight_smile: - the code snippet José provided is slightly cut off and you have to scroll, there’s one additional line at the end of the on_epoch_end() function:

    def on_epoch_end(self, last_target, last_output, **kwargs):
        if len(self.output) > 0:
            output = torch.cat(self.output)
            target = torch.cat(self.target)
            preds = F.softmax(output, dim=1)
            metric = auroc_score(preds, target)
            self.learn.recorder.add_metrics([metric])

Hey @joshfp,

I’m trying to use your class for tabular_learner (binary classification), but I’m getting the following error:
AttributeError: 'Learner' object has no attribute 'add_metrics'
Do you have any idea why I’m getting this error?

Thanks!
David

I faced the same issue. Worked for me with the following change

def on_epoch_end(self, last_metrics, **kwargs):
        if len(self.output) > 0:
            output = torch.cat(self.output)
            target = torch.cat(self.target)
            preds = F.softmax(output, dim=1)
            metric = auroc_score(preds, target)
            return add_metrics(last_metrics, [metric])
4 Likes

Thanks! it worked

For those who find this post before the documentation is updated. ‘AUROC’, ‘auc_roc_score’, and ‘roc_curve’ were all added in fastai version 1.0.51.

8 Likes

Hi,
Can’t see the same on the documentation…can you pls share the link about the update?

Quick update on this for everyone who is as confused as I was: In another thread, Sylvain mentions that you have to instantiate and pass AUROC as a metric like so: learn = text_classifier_learner(data_clas, arch=AWD_LSTM, drop_mult=0.5, metrics=[accuracy, AUROC()]) This seems a bit inconsistent but thats how it works for me :slight_smile:

5 Likes

Hey Sebastian!

Thanks for that bit of info. Btw is it possible for you to link that discussion thread as well (I know it might be asking a bit much, but, if you have a quick reference saved, please do share)?

Hi partham, of course I can share it :slight_smile: this is it. But I think it is not that helpful in general. How to use AUROC with cnn_learner in Fast AI? As you can see, it used to be different as well, I needed a while to figure out the currently working way :slight_smile:

1 Like

Hey @joshfp !

Thanks a lot for sharing this. Helped me out a lot! Also, thank you @amitpphatak for making it work for a binary classifier.

I will be using your guy’s code for my master thesis since I need to compare my neural net on a number of performance metrics. Is there any way I could reference you guys? I would be glad to do so.

Thanks again!

I see from the docs that AUROC is restricted to binary classification. Does it make sense to extend it to multi-label classification? (or does that already work with some modification?) I believe scikit-learn’s version works with both binary classification and multi-label, so I think the multi-label case at least makes sense mathematically.

2 Likes

@joshfp

Thank you for sharing.
I used it in my code, but I encountered problem.

My result has 0.5 of AUROC in every epoch…
There is something wrong in my case.
Could you help me on how to fix this?

Thanks, in advance.

1 Like

I get this error while trying your code
AttributeError: ‘AUROC’ object has no attribute ‘output’

def on_epoch_begin(self, **kwargs): self.output, self.target = [], []

def on_batch_end(self, last_target, last_output, train, **kwargs):
if not train:
self.output.append(last_output)
self.target.append(last_target)

Is it a version issue? Thanks in advance.

Tried using AUROC() in binary image classification with cnn_learner. Even though the model was learning and validation loss was going down, the AUROC score was always either 0.5 or NAN (depending on the model). Replaced softmax with sigmoid and started seeing reasonable AUC scores. Is this a valid approach?

class MyAUROC(AUROC):
    ...       
    def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):
        last_output = torch.sigmoid(last_output)[:,-1]
        # last_output = F.softmax(last_output, dim=1)[:,-1]
        self.preds = torch.cat((self.preds, last_output.cpu()))
        self.targs = torch.cat((self.targs, last_target.cpu().long()))
    ...

data = (ImageList.from_df(df,'')
                .split_from_df(col='is_valid')
                .label_from_df(cols='label', label_cls=FloatList)
                .transform(get_transforms(), size=224)
                .databunch(bs=16))
           .normalize(imagenet_stats)

learn = cnn_learner( data, 
                     models.resnet50, 
                     pretrained=True,
                     loss_func=BCEWithLogitsFlat(),
                     metrics=[MyAUROC()] ).to_fp16()
...

Your solution worked perfectly for me. Thanks!

However, could you say why is it that AUROC() needs to be passed as a metric, even though it is a callback as mentioned in the Docs?

Also, I can’t seem to wrap my head around why do we have callback_fns as an argument for creating a Learner. Shouldn’t Callbacks be limited to only during training, and so functions like fit(), fit_one_cycle() etc only support Callbacks?

Hello, I have a question regarding the AUROC class and its usage in imbalance dataset.
I am working on a problem which has high imbalance and was trying to use the AUROC score.
When I saw the implementation of AUROC, I found out that by default it is considering the second class in case of binary classification task as the positive class(minority class) and taking its softmax output which is then passed as y_pred to the auc_roc_score function. But it might be the case that the first class can be positive class. So following the above situation, if i change the callback to take 1st class as positive class, is it correct?

Considering Second class as positive class:
def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):
last_output = F.softmax(last_output, dim=1)[:,1]
self.preds = torch.cat((self.preds, last_output.cpu()))
self.targs = torch.cat((self.targs, last_target.cpu().long()))

Changing the code if first class is the minority(positive class)
def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):
last_output = F.softmax(last_output, dim=1)[:,0]
self.preds = torch.cat((self.preds, last_output.cpu()))
self.targs = torch.cat((self.targs, last_target.cpu().long()))

Sounds right to me.

1 Like

Thank you. I have also implemented a per class accuracy callback which might be more useful in case of imbalanced datasets. Is it possible that it can be added to fastai callbacks?

I use @joshfp original callback function in my code but while using learn.get_preds() I am getting the following error:

AttributeError: ‘AUROC’ object has no attribute ‘output’

Can anybody point what’s the issue.

Please find attached code

and my network architecture