For the benefit of other people running into this issue:
It seems that for multilabel problems, the output of learn.get_preds() changed between versions 1.0.10 and 1.0.24. In 1.0.10, the preds are logits, while in 1.0.24 they are probabilities, i.e. sigmoid(logit) (all outputs are binary). By default, the function accuracy_thresh applies sigmoid to each coefficient.
This means that in order to calculate accuracies for multilabel problems we need to call
accuracy_thresh(*learn.get_preds())
in version 1.0.10 (and lower, presumably), which by default uses sigmoid=True. In version 1.0.24 (and higher, presumably), we need to call
accuracy_thresh(*learn.get_preds(), sigmoid=False) .