I am trying to see if something like randomforestclassifier._feature_importances is available for pytorch. For structured datasets would be really neat to have that accesible!
The method I used is to shuffle each column, one at the time, and compute the decrease in the loss function. Probably there are more efficient or better ways to do it but I hope it can be helpful for a starting point!