I’m not very familiar with this, but I’ll give it a go…
You report your error using three cutom metrics.
metrics=[partial(accuracy_multi, thresh=0.5),
fbeta_macro, fbeta_sample] )
First thing to help would be try each one separately to narrow down the issue.
You don’t say if you already had a minimal metric implementation working.
The minimal metric example is:
def custom_accuracy(preds, targets, threshold=0.5):
preds2 = (preds > threshold).float()
accuracy = (preds2 == targets).float().mean()
return accuracy
learn = vision_learner(dls, resnet34, metrics=[custom_accuracy])
I notice the FBetaMulti function definition does not have parameters preds, targets, so something might clash there.
FBetaMulti (beta, thresh=0.5, sigmoid=True, labels=None, pos_label=1,
average='macro', sample_weight=None)
I’m not familiar with the FBetaMulti function, but its documentation says its for “multi-label classification problems”, which is different from “multi-class classification” using a single label.
IIUC…
- Multi-class classification: Each instance belongs to exactly one class from a set of mutually exclusive classes.
- Multi-label classification: Each instance can have multiple labels, and the classes are not mutually exclusive.
Pay attention to the warning under Single-label classification.
So an approach to solve this would be:
- Drop back to a single custom metric
- Get the basic example working
- Wrap your basic example around your calls to FBeta, so you can sprinkle debug-prints to show type and content of data in/out between “basic example” and “Fbeta”.
.