Hi again @muellerzr (and all!)
When you define the “data” variable you are defining the training set, the validation set (by splitting) and passing the test set as a separate df. That’s understood.
Then I train and I get something like:
epoch
train_loss
valid_loss
accuracy
time
0
0.322909
0.357733
0.835833
00:03
1
0.327918
0.359473
0.834833
00:03
2
0.335293
0.361857
0.832667
00:03
My understanding is that the accuracy is based on using the validation set… so, how is the test set being used in this case? (As originally the whole dataset is labeled, the test set is like a second validation set…)
The test set is not used until you are finished training and you want to only evaluate how you are doing. You do it at the very end. Does that make sense? I’ll also say, the above only have unlabeled test sets. You need to do things differently if we want to grade the test set, instead of just getting predictions.
So I could write some code to parse the test dataset, get the categories into 0’s and 1’s and compare with the second list…
I can see the use if this was a kaggle challenge, you had no labels in the test set and you need to submit results.
But in my case I have a single dataset with everything labeled… not sure if I should just forget about the test set and just have bigger train/validation sets and look at the accuracy after training.
Or maybe I am missing the smart way to use the test set
No absolutely! The test set is extremely important. Give me a moment and I’ll give some code on how to do the evaluation. Essentially we label the test Tabular list and we turn the valid_dl into that one.
We cannot directly pass a data loader to validate. What will happen is it’ll run the default validation set no matter what. So what I describe in the link above is how to go about overriding that with the labeled test set so we can properly use it.
But what is the purpose of the .add_test directive?
I can see you use the test dataset afterwards (as you indicate, making it the validation ser), but I don’t see why it is needed when “data” is created…
Because all three are tied together for Learner. This is mostly used for Kaggle competition where we have separate train and test CSV’s. When you finish training you just call learn.get_preds(DataSetType.Test) and it will give you the predictions for the test set.
However for labeled, we don’t add a test. It’s not needed, as shown in the link above.
So long as you are switching the dataloaders, yes, that’s what I’ve noticed too you can also verify this by calling learn.data. The validation set should now be your test set
I go into a different way of doing it in the notebook above, but that’s the same that is done in the Tabular example. Everything above it is the train, below is the validation, and in the middle is the test set.