I’m wondering is it possible to do three-way split to split my data into non-overlapping training/testing/validation sets?
Using the following example to illustrate:
from fastai2.data.external import URLs, untar_data
from fastai2.data.transforms import get_image_files, parent_label, GrandparentSplitter
from fastai2.data.block import DataBlock, CategoryBlock
from fastai2.vision.data import ImageBlock
from fastai2.torch_core import doc
dest_path = untar_data(URLs.MNIST)
dblock = DataBlock(
blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
get_y=parent_label,
splitter=GrandparentSplitter(train_name='training', valid_name='testing')
)
dls = dblock.dataloaders(dest_path)
In the above code, we use GrandparentSplitter to split the data into training set (the “training” folder) and validation set (the “testing” folder). How do I specify another test set (which is separate from the training and validation sets) that I can use to evaluate the final model?
If your test files were originally in the items grabbed by get_image_files, you can write a custom splitter to not return two but three list of indices (look at the code of GrandParentSplitter and adapt it). That will make your test set part of your dls in the third subset (you can then access it as dls.loaders[2]).
Thanks @sut!
I was referring to the actual labels for the test data, not predicted labels.
I would expect that actual_classes and pred_classes would be PyTorch tensors, so I was expecting some utility function in the fast.ai library to calculate the metric.
There is no direct way, but there is a way to add things to what’s logged (see the metric sections, there is one for just any quantity you set) and you can add a callback making another validation on ds_idx=2 after the normal validation.
I’m stuck, a little advice would be much appreciated. I am trying to call learn.validate within a callback method, but this gets me stuck an infinite loop.
I assume this is because the .validate contains references to other events. How to get around this? Or is there an example of a callback which calls a method on the leaner that I can use as a template?
You can’t call validate in a callback without a recursive loop since validate calls after_epoch, which then calls the callbacks, again and again.
You should use the private method _do_validate after setting self.dl to avoid this.