Using albumentations with fastai v2

fastai fastai dev
1

I added examples on how to use albumentations in the pets tutorial (still incomplete, but this in the first section that is over. (In general, the tutorial shows how to write your own transform.)

Just be mindful that if you need to transform your targets as well as in your inputs (like in segmentation or object detection), you need to use a wrapper that is an ItemTransform (the tutorial explains why) and that will only work for this kind of task. It’s just 4 lines of codes to copy though, so not too complicated :wink:

If there is enough interest (given the fact this is CPU data augmentation and not GPU augmentation) I can work on a magic wrapper that would just work all the time, but that would be for after the course and fastai v2 release.

22 Likes
2

I assume all the albumentation transforms will be CPU only and it is not possible to do it on the GPU?

3

I didn’t know about albumentations but I saw they have a lot of very cool functions.
Even if it’s on CPU I would love to have a magic wrapper for using it!

1 Like
4

Yes, their library does not provide that.

1 Like
5

Easier albumentations support would help with converts to fastai who are used to using it with other frameworks. I enjoy albumentations. +1 from me.

1 Like
6

Definitely +1 on the interest. Thanks for putting this notebook together, I will check it out later!

7

Thanks for the clarification. Even with this limitation, I still this it is worth incorporating an albumentations wrapper for fastai v2 due to the many additional augmentations implemented in albumentations!

8

More of an understanding question, are transforms only for augmentation?

I’m playing with using canny edge detection and classifying based on those images, not the originals. I wrote the canny stuff within a transform using open cv filling the pets tutorial.

I export and later load_learner and it all seems to work within the notebook but I’m not sure how. does model.export serialize all the item transforms? and if so, does it also serialize any third party libraries, such as open cv?

2 Likes
9

@sgugger The wrapper in which you would work is going to make them work on GPU?

I am interested. Nice work!!

10

No. Again, that library has not been designed to work on the GPU. It would need to be entirely rewritten to work on GPU, which is not my intention.
The wrapper is to have something unified and not one transform classification and another for segmentation.

11

Alright, yeah I know that it worked on CPU but maybe you were going to reimplement it.

I think that I am going to try to reimplement some functions, I will let you know if I achieve that!

1 Like
13

I have tried the Albumentations wrapper on items_tfms. The augmentations are done correctly.

However, I am not getting reproducibility, I think that items_tfms is applied to validation too.

Is there any way of preventing items_tfms to being applied to validation?? @sgugger

14

You should set split_idx=0 in your Transform if you want it to be applied on the training set only. I’ll add something about this in the tutorial.

1 Like
16

I am so thankfull :smiley: of all your contribution to the documentation. I was having a hard time trying to find that!!!

So, I should set this parameter to the result obtained of the AlbumentationsWrapper??
Looking into aug_transform too it doesn’t have this parameter. So, is it doing by default??

17

You just need to write split_idx = 0 at the beginning of your class AlbumentationWrapper. See here for how it’s done for RandTransform (all the kinds returned by aug_transforms are RandTransform so inherit that behavior).

1 Like
19

Ohhh, understand!! :slightly_smiling_face:

Is there any need of adding super().__init__(**kwargs) too to the AlbumentationWrapper ?

I have tried that and I get the next error:

class SegmentationAlbumentationsTransform(ItemTransform):
    split_idx = 0
    def __init__(self, aug,**kwargs):
        super().__init__(**kwargs)
        self.aug = aug
    def encodes(self, x):
        img,mask = x
        aug = self.aug(image=np.array(img), mask=np.array(mask))
        return PILImage.create(aug["image"]), PILMask.create(aug["mask"])
    
transformPipeline=Compose([OneOf([
            OpticalDistortion(p=0.3),
            GridDistortion(p=0.1),
            ElasticTransform(p=0.3),
        ], p=0.2)],
        p=1)

transformPipeline=SegmentationAlbumentationsTransform(transformPipeline)

dataset1 = DataBlock(blocks=(ImageBlock, MaskBlock(codes)),
                   get_items=partial(get_image_files,folders=[dataset1_name]),
                   get_y=get_y_fn,
                   splitter=RandomSplitter(valid_pct=0.1,seed=2020),
                   item_tfms=[Resize((size,size)),transformPipeline],
                   batch_tfms=[*aug_transforms(mult=1.0, do_flip=True, 
                                flip_vert=True, max_rotate=10., 
                                max_zoom=1.1,max_warp=0.2, 
                                p_affine=0.75, p_lighting=0),Normalize.from_stats(*imagenet_stats)]
                  )
dataset1.summary(path_images)
dls = dataset1.dataloaders(path_images,bs=bs)
dls.show_batch(vmin=0,vmax=1,figsize=(12, 9))

Setting-up type transforms pipelines
Collecting items from ../datasets/Images
Found 621 items
2 datasets of sizes 559,62
Setting up Pipeline: PILBase.create
Setting up Pipeline: <lambda> -> PILBase.create

Building one sample
  Pipeline: PILBase.create
    starting from
      ../datasets/Images/manual/165.png
    applying PILBase.create gives
      PILImage mode=RGB size=1002x1004
  Pipeline: <lambda> -> PILBase.create
    starting from
      ../datasets/Images/manual/165.png
    applying <lambda> gives
      ../datasets/Labels/manual/165.png
    applying PILBase.create gives
      PILMask mode=L size=1002x1004

Final sample: (PILImage mode=RGB size=1002x1004, PILMask mode=L size=1002x1004)


Setting up after_item: Pipeline: AddMaskCodes -> SegmentationAlbumentationsTransform -> Resize -> ToTensor
Setting up before_batch: Pipeline: 
Setting up after_batch: Pipeline: IntToFloatTensor -> AffineCoordTfm -> LightingTfm -> Normalize
Could not do one pass in your dataloader, there is something wrong in it

Building one batch
Applying item_tfms to the first sample:
  Pipeline: AddMaskCodes -> SegmentationAlbumentationsTransform -> Resize -> ToTensor
    starting from
      (PILImage mode=RGB size=1002x1004, PILMask mode=L size=1002x1004)
    applying AddMaskCodes gives
      (PILImage mode=RGB size=1002x1004, PILMask mode=L size=1002x1004)
    applying SegmentationAlbumentationsTransform gives
      [PILImage mode=RGB size=1002x1004, PILMask mode=L size=1002x1004]
    applying Resize gives
      [PILImage mode=RGB size=1002x1004, PILMask mode=L size=1002x1004]
    applying ToTensor gives
      [PILImage mode=RGB size=1002x1004, PILMask mode=L size=1002x1004]

Adding the next 3 samples

No before_batch transform to apply

Collating items in a batch
Error! It's not possible to collate your items in a batch
PILImage is not collatable

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-21-3028df996ec0> in <module>
      9                                 p_affine=0.75, p_lighting=0),Normalize.from_stats(*imagenet_stats)]
     10                   )
---> 11 manual.summary(path_images)
     12 dls = manual.dataloaders(path_images,bs=bs)
     13 dls.show_batch(vmin=0,vmax=1,figsize=(12, 9))

~/anaconda3/envs/seg/lib/python3.7/site-packages/fastai2/data/block.py in summary(self, source, bs, **kwargs)
    181         why = _find_fail_collate(s)
    182         print("Make sure all parts of your samples are tensors of the same size" if why is None else why)
--> 183         raise e
    184 
    185     if len([f for f in dls.train.after_batch.fs if f.name != 'noop'])!=0:

~/anaconda3/envs/seg/lib/python3.7/site-packages/fastai2/data/block.py in summary(self, source, bs, **kwargs)
    175     print("\nCollating items in a batch")
    176     try:
--> 177         b = dls.train.create_batch(s)
    178         b = retain_types(b, s[0] if is_listy(s) else s)
    179     except Exception as e:

~/anaconda3/envs/seg/lib/python3.7/site-packages/fastai2/data/load.py in create_batch(self, b)
    124     def retain(self, res, b):  return retain_types(res, b[0] if is_listy(b) else b)
    125     def create_item(self, s):  return next(self.it) if s is None else self.dataset[s]
--> 126     def create_batch(self, b): return (fa_collate,fa_convert)[self.prebatched](b)
    127     def do_batch(self, b): return self.retain(self.create_batch(self.before_batch(b)), b)
    128     def to(self, device): self.device = device

~/anaconda3/envs/seg/lib/python3.7/site-packages/fastai2/data/load.py in fa_collate(t)
     44     b = t[0]
     45     return (default_collate(t) if isinstance(b, _collate_types)
---> 46             else type(t[0])([fa_collate(s) for s in zip(*t)]) if isinstance(b, Sequence)
     47             else default_collate(t))
     48 

~/anaconda3/envs/seg/lib/python3.7/site-packages/fastai2/data/load.py in <listcomp>(.0)
     44     b = t[0]
     45     return (default_collate(t) if isinstance(b, _collate_types)
---> 46             else type(t[0])([fa_collate(s) for s in zip(*t)]) if isinstance(b, Sequence)
     47             else default_collate(t))
     48 

~/anaconda3/envs/seg/lib/python3.7/site-packages/fastai2/data/load.py in fa_collate(t)
     45     return (default_collate(t) if isinstance(b, _collate_types)
     46             else type(t[0])([fa_collate(s) for s in zip(*t)]) if isinstance(b, Sequence)
---> 47             else default_collate(t))
     48 
     49 # Cell

~/anaconda3/envs/seg/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py in default_collate(batch)
     79         return [default_collate(samples) for samples in transposed]
     80 
---> 81     raise TypeError(default_collate_err_msg_format.format(elem_type))

TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'fastai2.vision.core.PILImage'>

If I delete split_idx = 0 it is working!!!

20

This is a bug in summary, I’ll look into it. If you remove the summary line, you will get a warning that there is something wrong with your dataloaders (same bug, ignore it) and show_batch will work.
Will look at why there is this problem.

Edit: No the problem is actually with ItemTransform, what I said before only works for the training set, not the validation set.

21

Fixed now (you need an editable install with the latest fastcore for the fix).

1 Like
22

I don´t understand that.

Do I need to update my code??

Thank you very much for the fix!!! Nice work!! :smiley:

23

No your code should work fine (with latest for fastai2 and fastcore). I was initially confused with the source of the bug.