Custom ImageList for "virtual" image patches/crops

Hi everyone, we’re testing a custom ImageItemList to enable loading “virtual” patches/crops

Wonder if anyone has needed similar? and could brainstorm some shortcomings/gotchas? :slight_smile:

assuming for example a dataframe with virtual path names like:

path,label
img1.png#10:10:30:30,1
img1.png#20:20:50:50,1
img2.png#20:20:50:50,1

(only img1.png,img2.png exist in data folder and 10:10:30:30 is x1,y1,x2,y2 for crop the image)

code is like

class PatchImageItemList(ImageList):
    def open(self, fn:PathOrStr)->Image:
        if not "#" in fn:
            return super().open(fn)
        
        filename,coords=fn.split("#")
        
        coords=[int(c) for c in coords.split(":")]
        image=image_loader(filename,coords)
            
        return vision.Image(px=image)

I had to do something similar but I have a normal image list and use a custom transform to take random patches from it

hi @juvian, thaks for helping! :slight_smile:

that’s an interesting note and will check those again better.
though would seem this is mostly for augmentation right?
currently the need is most on experimenting several patch strategies for very large images, but avoiding having to produce/replicate tons of small files just to run models.
so we just produce csv or dataframe, dont touch storage.

do you think transforms is a better option?
other thing I see with code above, prob could benefit from some caching (for image formats that don’t allow random access, some do) , some concerns also with multiple workers.

huge thanks for your thoughts!
Rui

Yeah I use it as data augmentation, instead of using full sized images to train I take random 64x64 patches from it. If you just need to split image in patches without overlap your solution is better.

1 Like

Hey there,

I am new to fastai, just finished a first part of the course. I was actually looking for the method to apply custom crops for each input (and test) images. I just thought that it might help classification problems with more than one object in the image.

I would appreciate if you can direct me to the code/git repo showing how to use this class with fastai ItemList.

I also a few questions regarding this:

  1. How do you apply the custom crops during the testing? My idea was to use cv.find contours and identify coordinates for cropping during training and testing for each image.

  2. As I understood from the example, we can augment the data by doing custom crops and extending the labels? Is that right?

Thanks,

Hey @alex6,

As per the class listed above, you should be able to replace ImageList, like :

src=PatchImageItemList.from_df(df=pd.read_csv(…), path=data_folder,cols=‘path’).split_by_rand_pct().label_from_df(cols=‘label’)

assuming in the csv you have:

path,label
image.png#0:0:224:224,1

Now, the purpose of this class is not training augmentation, for that (including random crops & others) you should see transforms in fastai I think. Class above is instead to enable “virtual images”, ex: patches from a bigger image without generating smaller patches (which would just be replicating original image anyway). And assuming you can have ground truth for each crop. makes sense?

hope it helps.

side note: as the observations will not be iid, validation split should be case/image wise and not random (code above is simplified in that sense).