I have read in many blogs that having a validation strategy and the judgment of whether you can trust your validation approach/ validation score matters a lot.
Can anyone shed some light on this? What is it that’s being referred to?
The idea is that you don’t care how well you do on your training set. What is the easiest way to have 100% accuracy on the training set? Just memorize it and look up what the answer is for a particular example.
The problem is that this approach has very little value - we don’t care what our performance on the training set is, all we care is how well our algorithm can do on examples it hasn’t seen before. This is where the validation set comes in. It gives us some approximation to estimating how well our model will do on examples it has not seen before. But this comes with a twist- we use the validation set to fit hyperparameters of our model, such as learning rate, amount of epochs we take to train the model, etc. So in reality the performance on the validation set is not such a great approximation to what our model will do on data it has not seen before, but it does give us an ability to tune our model with some hopes that what it learns will generalize to new data.
I am not really sure if this answers your question. There is a lot of good material on the Internet that I think would be able to answer your question much better than I am able to - in particular, Andrew Ng’s course on coursera gives some good intuition on why we need a validation set. But to be honest, I think that a good start would also be to just play around with how @jeremy is using the validation set. I do not think that anyone really has a good, definite answer to how big of a validation set one would need and there really isn’t much more to it than the fact that having a validation set is needed - everything else is secondary (validation strategy, set size, etc) and probably not really important until we start breaking into the 1% of a kaggle competition 
1 Like
I’m well aware of this… I was hoping there is something more to this!
And also the reason people hold out an additional set called a test set other than the validation set is because of the human bias. We unknowingly induce it since we make changes to the training set (hyper-parameters) and compare how well the validation set is doing – that is, the optimal hyperparameters at the end of the day are the ones after many iterations. So keeping a separate set to see how well the model is doing despite this human induced bias is a good idea.