For the last week, I have been working on a multi class classification problem on columnar data. The first time I ran it, the accuracy was surprisingly 87% but I later realized this was because I was passing is_multi=True to the learner meaning it was using incorrect loss and accuracy metrics (it was using binary_cross_entropy and accuracy_multi instead of nll_loss and accuracy). Once I got that right, I realized my classes were imbalanced and then I applied undersampling and oversampling (duplicated rows) to evenly distribute the data.
For the last couple of days the accuracy hasn’t improved beyond 40% no matter what I try: change hyperparameters (embedding sizes, # of linear layers, etc) or change number of features.
I am looking for some insights on the best strategy to move forward in these situations. Is there a systemic approach that has worked for you to identify the issues? At this point, I have no idea whether to check my training data, tune hyper params (my assumption is that hyper params can only help so much and the model should be doing good with the default ones) or try to think of more features. Could it also be possible that the architecture/model has some issues?
Also, are there any strong indicators suggesting that the problem is not really suited to the architecture or that the features aren’t just enough?