I curious to get a bit more clarification about how we can tell if we are over fitting. Lets say my training accuracy is quite a bite higher than my validation accuracy after the first epoch of training (lets say 90% versus 80%). Is it over fitting if both accuracies are improving as more epochs are trained?
I am inclined to say “not overfitting” based on only on the difference between training and validation accuracies. As long as both accuracies are improving with more training, then its not over fitting. It only becomes over fitting when only your training accuracy is going up, but validation accuracy is going down. Is this correct?
Strictly speaking, yes you’re over-fitting if your training accuracy is higher than validation. But the correct level of over-fitting is not zero. As you say, as long as your validation accuracy is improving, you should keep training and keep reducing regularization!
I think so … at least that is the most convenient way to visualise signs of over fitting. However in non NN models, like in linear models, large weights on certain features can also possibly signify over fitting - hence we use regularization (@jeremy et al can y’all confirm?). Not sure if it is easy to do this in NN models with possible millions of parameters and the difficulty in correlating those params back to the input.