How to use AUROC with cnn_learner in Fast AI?

(Bilal) #1

Hi All,

I checked the forum and found a few threads talking about how to use AUROC with Fast AI. This is how it was mostly suggested.

acc_02 = partial(accuracy_thresh, thresh=0.2)
f_score = partial(fbeta, thresh=0.2, beta=1)
learn = cnn_learner(data, arch, metrics=[acc_02, f_score], callback_fns=AUROC)

But I got the following error:

Any idea what am I doing wrong here.

Thanks a lot
Kind Regards,
Bilal

1 Like

Using AUC as metric in fastai
#2

You need to instantiate it: pass AUROC().

0 Likes

(Bilal) #3

This is what Fast AI return when I pass the following arguments:

learn = cnn_learner(data, arch, metrics=[acc_02, f_score], callback_fns=AUROC())

0 Likes

#4

Oh, you need to pass it in the metrics, not the callbacks.

0 Likes

(Bilal) #5

Thanks for the response.

I now used the following commands:

learn = cnn_learner(data, arch, metrics=[acc_02, f_score, AUROC()])

And this generates the following output:

1 Like

#6

I know it has been some time, but did you ever get this to work?

0 Likes

(Bilal) #7

No, I tried all the options mentioned to me but all in vain. I didn’t achieve the AUROC with Fast AI.

1 Like

(Sebastian Fleck) #8

I can confirm that it does work for NLP though. I used exactly this syntax on my text classification data: learn = text_classifier_learner(data_clas, arch=AWD_LSTM, drop_mult=0.5, metrics=[accuracy, AUROC()])

1 Like

(Bilal) #9

I was using it for multi-label classification in vision task where it always raised an error. I’m not sure what led it into an error if it worked in text classification.

Still??

0 Likes

#10

It has been about a month but I will just drop this here. AUROC in the fastai library is restricted to binary classification problems. Use it for multi-classification must be the cause of the error.

0 Likes