I think I just made the exact same mistake - the code snippet José provided is slightly cut off and you have to scroll, there’s one additional line at the end of the on_epoch_end() function:
I’m trying to use your class for tabular_learner (binary classification), but I’m getting the following error: AttributeError: 'Learner' object has no attribute 'add_metrics'
Do you have any idea why I’m getting this error?
Quick update on this for everyone who is as confused as I was: In another thread, Sylvain mentions that you have to instantiate and pass AUROC as a metric like so: learn = text_classifier_learner(data_clas, arch=AWD_LSTM, drop_mult=0.5, metrics=[accuracy, AUROC()]) This seems a bit inconsistent but thats how it works for me
Thanks for that bit of info. Btw is it possible for you to link that discussion thread as well (I know it might be asking a bit much, but, if you have a quick reference saved, please do share)?
Hi partham, of course I can share it this is it. But I think it is not that helpful in general. How to use AUROC with cnn_learner in Fast AI? As you can see, it used to be different as well, I needed a while to figure out the currently working way
Thanks a lot for sharing this. Helped me out a lot! Also, thank you @amitpphatak for making it work for a binary classifier.
I will be using your guy’s code for my master thesis since I need to compare my neural net on a number of performance metrics. Is there any way I could reference you guys? I would be glad to do so.
I see from the docs that AUROC is restricted to binary classification. Does it make sense to extend it to multi-label classification? (or does that already work with some modification?) I believe scikit-learn’s version works with both binary classification and multi-label, so I think the multi-label case at least makes sense mathematically.
Tried using AUROC() in binary image classification with cnn_learner. Even though the model was learning and validation loss was going down, the AUROC score was always either 0.5 or NAN (depending on the model). Replaced softmax with sigmoid and started seeing reasonable AUC scores. Is this a valid approach?
However, could you say why is it that AUROC() needs to be passed as a metric, even though it is a callback as mentioned in the Docs?
Also, I can’t seem to wrap my head around why do we have callback_fns as an argument for creating a Learner. Shouldn’t Callbacks be limited to only during training, and so functions like fit(), fit_one_cycle() etc only support Callbacks?
Hello, I have a question regarding the AUROC class and its usage in imbalance dataset.
I am working on a problem which has high imbalance and was trying to use the AUROC score.
When I saw the implementation of AUROC, I found out that by default it is considering the second class in case of binary classification task as the positive class(minority class) and taking its softmax output which is then passed as y_pred to the auc_roc_score function. But it might be the case that the first class can be positive class. So following the above situation, if i change the callback to take 1st class as positive class, is it correct?
Considering Second class as positive class:
def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):
last_output = F.softmax(last_output, dim=1)[:,1]
self.preds = torch.cat((self.preds, last_output.cpu()))
self.targs = torch.cat((self.targs, last_target.cpu().long()))
Changing the code if first class is the minority(positive class)
def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):
last_output = F.softmax(last_output, dim=1)[:,0]
self.preds = torch.cat((self.preds, last_output.cpu()))
self.targs = torch.cat((self.targs, last_target.cpu().long()))
Thank you. I have also implemented a per class accuracy callback which might be more useful in case of imbalanced datasets. Is it possible that it can be added to fastai callbacks?