RuntimeError: The size of tensor a (2) must match the size of tensor b (96) at non-singleton dimension 1

According to docs, accuracy_thresh is intended for one-hot-encoded targets (often in a multiclassification problem). I guess that’s why your size of tensor doesn’t match.