Is validation dataset used for anything other than reporting?

In traditional ML, I’ve seen datasets broken up into {TRAIN, VALIDATION, TEST}.

  • "TRAIN" would be use to train the weights of a given pre-defined model.
  • "VALIDATION" would be used for hyperparameter/model selection or tuning
  • "TEST" would be representative of a true holdout dataset.

In other words, it’s ideal for “TEST” to be treated as akin to out-in-the-wild data, and should never be used as part of the training/model-tuning process.

In lessons 1-3, I noticed ConvLearner and DataBunch are used a lot. I also noticed it compares results to Kaggle competitions. I have a few questions:

  1. Does ConvLearner (or DataBunch) do any sort of model-selection or tuning with the validation data? Or is the validation data in the DataBunch truly treated as a holdout where only metrics are reported?

  2. If it is truly a holdout, should we in general treat DataBunch’s validation dataset as the equivalent of the holdout evaluation sets found in say Kaggle competitions?

I do see DataBunch does have a Test dataset option, but it is forced to be unlabeled, which requires manually evaluating after, vs. doing it as part of the training reporting as is done in Lessons 1-3.

Thank you.