Random image cropping before rescaling

fastai
1

By default Fastai resizes images to the size defined by used architecture (for example 224x224 for resnet). The type of resizing is dictated by resize_method . This is done separately from transformations as they are applied after the resizing step.
What I would like to do and I also think this is beneficial to many vision applications, is to randomly crop input image (to the size applicable to chosen architecture) instead of resizing (downscaling it). Resizing by downscaling means loss of information, by taking random samples from the input image instead (cropping randomly), we can significantly increase size of our training set without compromising input image quality.
I understand this may not be helpful for all applications, but in some cases this is a common way (like text detection in photographs, where its not essential to feed the whole image to a network).

I found that this topic has been covered before (with previous version of the library) but there was no clear solution given. In transforms.py, crop is applied after rescale

Is there any way to make this scenario possible in fastai v1.0? Has anyone tried something like that before?

Cheers,
Marcin

2

If you select resize_method = ResizeMethod.Crop your images will be randomly cropped after transformations are applied (this is all done in one step by apply_transforms). The way it works is that the image is resized so that the smallest dimension match the desired size, then randomly cropped in the other dimension.

It’s not exactly what you want, I understand. You can also define your own transform to be applied to PIL images and pass it with after_open in your ImageList.

1 Like
3

Thanks @sgugger for your reply. I see that it is not supported in fastai v1.0, that’s too bad. Any chance this will be added in the future releases?

I understand that a custom method in after_open could be a solution but how do I take advantage of the “randomness” such that it applies to both source and targets? I mean I have the following scenario:
One ImageList (called src_itemlist) for my source data
One custom ImageList (maps_itemlist) for my target data (set of masks)
Later I construct LabelLists with src_itemlist.label_from_lists
I would like the cropping to be applied to both lists (src_itemlist and maps_itemlist) such that the random crop position is the same for xs as well as for ys. As I understand, after_open can be only applied to one list.
What’s the best way to do it?

4

I think you could just tweek fastai’s Image.apply_tfms so that it doesn’t resize (like you almost take away the whole first part of the function), then you can use a custom transform pipeline that includes your random crop resize at some point. See transform.crop_pad to understand how random transform are created in fastai. You can do basically the same function but without padding if you’d like. I’m not sure this is the best way to do it bu that should work.
Another option that could work may be to create your crop function (and all your transforms) with another name for the size argument (like crop_size for instance) that you can pass when creating your item lists. As long as size is never specified, resize will not be triggered by fastai in Image.apply_tfms, so you should be ok.

I don’t know if there’s any cleaner way to do this though.

5

Thanks @florobax for the suggestions, I’ll try them out.

6

Hello,

very intersting discussion.

Is there any update for the “2020 best practice” on this problem?

I was thinking along these lines:

crop_rand_patches = [crop(size=(PATCH_SIZE,PATCH_SIZE),  row_pct=(0,1), col_pct=(0,1))] # pct 
train_aug_tfms = # ... list of transforms
valid_aug_tfms = # ... list of transforms
tfms = [crop_rand_patches + train_aug_tfms, crop_rand_patches + valid_aug_tfms]

data = ImageDataBunch.from_folder(DEMO_FOLDER, bs = BATCH_SIZE, resize_method=ResizeMethod.NO, \
                              size=PATCH_SIZE, train = 'train', valid='valid', ds_tfms = tfms)

Note the ResizeMethod.NO.

Thank you
Best
Andy