No longer able to reproduce fastprogress's accuracy_thresh in v1.0.24

I’m working with the Planet dataset, which is a multilabel classification problem. With version 1.0.24, when calling
learn = create_cnn(data, models.resnet18, metrics=accuracy_thresh); learn.fit_one_cycle(1)
fastprogress shows an accuracy of 96%, but when I call accuracy_thresh(*learn.get_preds()), I get 17%. When downgrading to version 1.0.10, I get 96% both times. Is this intended or a bug?

For the benefit of other people running into this issue:

It seems that for multilabel problems, the output of learn.get_preds() changed between versions 1.0.10 and 1.0.24. In 1.0.10, the preds are logits, while in 1.0.24 they are probabilities, i.e. sigmoid(logit) (all outputs are binary). By default, the function accuracy_thresh applies sigmoid to each coefficient.

This means that in order to calculate accuracies for multilabel problems we need to call
in version 1.0.10 (and lower, presumably), which by default uses sigmoid=True. In version 1.0.24 (and higher, presumably), we need to call
accuracy_thresh(*learn.get_preds(), sigmoid=False) .

1 Like