Speaking mathematically, softmax is not invertible, so what you want to do is impossible.
I agree with Alan that you would have to extract the activations from inside the model.
The question whether multi-class activations would be equivalent to multi-label trained on one class at a time interests me too. (If that’s in fact your idea.) I even posted here about it once, but no responses.