In Chapter 6's image regression case, why do we split on just one person's image?

Part 1 (2020)
1

I get that we want to make sure our model generalizes well, and that this requires us to only use images of people that our model wasn’t trained on. As a reminder, here’s the Datablock we construct from the Biwi image set:

biwi = DataBlock(
    blocks=(ImageBlock, PointBlock),
    get_items=get_image_files,
    get_y=get_ctr,
    splitter=FuncSplitter(lambda o: o.parent.name=='13'),
    batch_tfms=[*aug_transforms(size=(240,320)), 
                Normalize.from_stats(*imagenet_stats)]
)

My question is, if we hard-code our splitter to just one person’s name, then doesn’t that mean our validation set is composed of just one person’s image? Normally we reserve something like 20% of our data to use as a validation set, but here we’re only using 1 person. Wouldn’t this imply that we’re not fully gauging whether our model generalizes well across a wide variety of never-before-seen people?

1 Like
2

In my opinion: yes.

But also:

I havent checked how many different people are in the dataset. Maybe taking one out of the dataset is pretty close to 20% of the total people/image amount.

Also this statements is given
One important point to note is that we should not just use a random splitter. The reason for this is that the same people appear in multiple images in this dataset, but we want to ensure that our model can generalize to people that it hasn't seen yet.

Given we only have a few people, then there wouldn’t be a better tradeoff to create a validation data set.

2 Likes
3

Makes sense. So in a case like this, where we have few images to choose from when populating our training vs. validation datasets, it might make sense to use k-folds cross-validation? That would allow us to entirely sidestep the issue of training vs validation datasets, while giving us arguably a better representation of how well our model generalizes? That’s my understanding, anyway.

1 Like