Hi All,
I checked the forum and found a few threads talking about how to use AUROC with Fast AI. This is how it was mostly suggested.
acc_02 = partial(accuracy_thresh, thresh=0.2)
f_score = partial(fbeta, thresh=0.2, beta=1)
learn = cnn_learner(data, arch, metrics=[acc_02, f_score], callback_fns=AUROC)
But I got the following error:
Any idea what am I doing wrong here.
Thanks a lot
Kind Regards,
Bilal
You need to instantiate it: pass AUROC().
This is what Fast AI return when I pass the following arguments:
learn = cnn_learner(data, arch, metrics=[acc_02, f_score], callback_fns=AUROC())
Oh, you need to pass it in the metrics, not the callbacks.
Thanks for the response.
I now used the following commands:
learn = cnn_learner(data, arch, metrics=[acc_02, f_score, AUROC()])
And this generates the following output:
I know it has been some time, but did you ever get this to work?
No, I tried all the options mentioned to me but all in vain. I didn’t achieve the AUROC with Fast AI.
I can confirm that it does work for NLP though. I used exactly this syntax on my text classification data: learn = text_classifier_learner(data_clas, arch=AWD_LSTM, drop_mult=0.5, metrics=[accuracy, AUROC()])
I was using it for multi-label classification in vision task where it always raised an error. I’m not sure what led it into an error if it worked in text classification.
Still??
It has been about a month but I will just drop this here. AUROC in the fastai library is restricted to binary classification problems. Use it for multi-classification must be the cause of the error.
