I’m working with the Planet dataset, which is a multilabel classification problem. With version 1.0.24, when calling
learn = create_cnn(data, models.resnet18, metrics=accuracy_thresh); learn.fit_one_cycle(1)
fastprogress shows an accuracy of 96%, but when I call accuracy_thresh(*learn.get_preds()), I get 17%. When downgrading to version 1.0.10, I get 96% both times. Is this intended or a bug?
For the benefit of other people running into this issue:
It seems that for multilabel problems, the output of learn.get_preds() changed between versions 1.0.10 and 1.0.24. In 1.0.10, the preds are logits, while in 1.0.24 they are probabilities, i.e. sigmoid(logit) (all outputs are binary). By default, the function accuracy_thresh applies sigmoid to each coefficient.
This means that in order to calculate accuracies for multilabel problems we need to call
accuracy_thresh(*learn.get_preds())
in version 1.0.10 (and lower, presumably), which by default uses sigmoid=True. In version 1.0.24 (and higher, presumably), we need to call
accuracy_thresh(*learn.get_preds(), sigmoid=False) .
1 Like