I’ve seen several posts asking this question but haven’t found any relevant responses. After clean-up I split my training data into train, validate, and test sets. Train and validate get passed into the classifier to train the model. What I need to do is pass the test set into the trained model to compare the results to the validation results from when the model was trained. This seems like a pretty straightforward thing but my searches on the forum haven’t turned up anything.
Maybe this is covered in one of the lessons I’ve skipped over? My focus is on a specific NLP problem and I jumped from Part 1 lesson 4 to Part 2 lesson 10. If this is covered in the classes can somebody send me a pointer? Thanks.
As far as I know the standard is to call .predict on the test set and do the analysis manually. I think you’ve identified an opportunity for an addition to the library, as it would be helpful to be able to pass the same set of metrics you use for evaluation on the validation set.
@sgugger What do you think? Could .predict be modified to include metrics, or would it be better to wrap the .predict with another library function that does the evaluation? I’m guessing the latter since it returns a set of predictions.
Thanks. I’ve done this with other libraries. Unless someone responds with a how-to on the version I’m working with (the one we use for the classes) I’ll dig into the code and try to figure it out. Be cool if I can make a contribution.
The test set set in fastai is unlabelled, and used to make predictions on a lot of data at the same time (with learn.get_preds). If you want to validate on a second dataset, create a second data object that has this validation set. You can change the data of your Learner at any point with learn.data = new_data.