Am I doing k-fold cross validation right?

Deep Learning
1

So I implemented k-fold from this notebook by using the new fastai version and for a vision problem (the notebook is for tabular). So am I doing it ok or are there some things that need to be changed?

Here is my code:

train_df, test_df = train_test_split(full_df, test_size=0.1)

folds = 4
skf = StratifiedKFold(n_splits=folds, shuffle=True)

val_pct = []
test_pct = []
batch_size = 32


for train_index, val_index in skf.split(train_df.index, train_df['label']):
    
    train_block = DataBlock(
            blocks=(ImageBlock, CategoryBlock),
            get_x=get_x,
            get_y=get_y,
            splitter=IndexSplitter(val_index), # added val_index
            item_tfms=[
                Resize(384),
                FlipItem(p=0.4),
                RandomCrop(300)
            ],
            batch_tfms=[Normalize.from_stats(*imagenet_stats)]
        )
    
    test_block = DataBlock(
            blocks=(ImageBlock, CategoryBlock),
            get_x=get_x,
            get_y=get_y,
            item_tfms=[
                Resize(384)
            ],
            batch_tfms=[Normalize.from_stats(*imagenet_stats)]
        )
    
    
    train_dl = train_block.dataloaders(train_df, bs=batch_size)
    test_dl = test_block.dataloaders(test_df, bs=batch_size)
    
    
    test_dl.valid = test_dl.train
    
    # train model
    learn = train(train_dl, resnet101, epochs=10, freeze_epochs=7)
    _, val = learn.validate()
    
    learn.dls.valid = test_dl.valid
    _, test = learn.validate()
  

    print('done, appending results.. \n')
    val_pct.append(val)
    test_pct.append(test)

Also, should the augmentations(item_tfms, batch_tfms) from the train_block be used for the test_block.

Thanks a lot! :grinning:

2

For the most part, this looks good. However, the test data block should have an additional CenterCrop(300) so you can get an image size of 300x300 just like for the train block.

1 Like
3

So I need to have the same item_tfms as in the train_block?

4

Not exactly the same. You don’t want to have random transformations (unless you are doing TTA) like FlipItem or RandomCrop but you would ideally want the dimensions and scale to be the same as during training. So you can use a CenterCrop(300) instead of RandomCrop(300).

1 Like
5

That makes a lot more sense, thanks a lot! :grinning:

6

Have you tried looking at the updated version on walk with fastai? It has a vision example :slight_smile: https://walkwithfastai.com/Cross_Validation

Let’s rewrite this a bit to make it more fastai-like:

train_df, test_df = train_test_split(full_df, test_size=0.1)

folds = 4
skf = StratifiedKFold(n_splits=folds, shuffle=True)

val_pct = []
test_pct = []
batch_size = 32


for train_index, val_index in skf.split(train_df.index, train_df['label']):
    
    train_block = DataBlock(
            blocks=(ImageBlock, CategoryBlock),
            get_x=get_x,
            get_y=get_y,
            splitter=IndexSplitter(val_index), # added val_index
            item_tfms=[
                Resize(384),
                FlipItem(p=0.4),
                RandomCrop(300)
            ],
            batch_tfms=[Normalize.from_stats(*imagenet_stats)]
        )
    
    
    dls = train_block.dataloaders(train_df, bs=batch_size)
    test_dl = dls.test_dl(test_df, bs=batch_size)
    test_dl.after_item = Pipeline([Resize(384), ToTensor()]) # to get rid of the need for another DataBlock
    
    
    # train model
    learn = train(train_dl, resnet101, epochs=10, freeze_epochs=7)
    _, val = learn.validate()
    
    _, test = learn.validate(dl = test_dl)
  

    print('done, appending results.. \n')
    val_pct.append(val)
    test_pct.append(test)

So what did we do differently? We don’t need to write out a second DataBlock. Since item_tfms are done lazily, we can replace our test_dl's item transforms with whatever we want (in this case we just want to Resize and apply ToTensor()). If you want that last RandomCrop to make your image (300,300), you can leave it in there safely as fastai by default will make that RandomCrop a CenterCrop on the validation set, similar to what Resize is doing (you’ve also now introduced test-time disparity, as you’re not exactly recreating what you trained on in image size)

Along with this, learn.validate can accept a dl param, so we can pass that test_dl in.

Hope this helps! You did a great job IMO, this is just my own preference towards writing a functionality like this :slight_smile:

1 Like
7

Wow, this is really great. I didn’t know about https://walkwithfastai.com/ and will check it out :grinning:. Thanks for the help and the explanations!

1 Like