Different transforms for x and y during augmentation

Part 2 & Alumni (2018)
1

Hi, I’m new to fast.ai environnement :slight_smile:

Is it possible to apply a different augmentation to the image target ?
For exemple in lesson 14 :

tfms = tfms_from_model(arch, sz_lr, tfm_y=TfmType.PIXEL, aug_tfms=aug_tfms, sz_y=sz_hr)
datasets = ImageData.get_ds(MatchedFilesDataset, (trn_x,trn_y), (val_x,val_y), tfms, path=PATH_TRN)

We use sz_lr on x and sz_y on y, but is it possible to apply aug_tfms on x and aug_tfms + grayscale (for exemple) only on the target image y ?

Thank you.

1 Like
2

tfms is a tuple of two Transforms lists. The first one is applied to your x and the second one to your y, so between your two lines, you can add/remove anything you want to each elements of tfms.

1 Like
3

Thank you for the quick response !

Edit : I think I got it !
Edit 2 : I don’t actually :frowning:

4

What is the correct way to add a tranform to tfms ?

I converted it to a list but tfms_list[0] is of type Transforms, how can I add something like RandomDihedral() since it’s of type RandomDihedral ?

Thank you ?

5

try this:
aug_tfms=[RandomCrop(…), RandomFlip(), RandomDihedral(), … ]

you can think of ‘aug_tfms’ as a list of functions that are applied to image. Each function takes an image as input. For 'RandomCrop() you have to provide crop size also eg ‘RandomCrop(224)’.
Hope this helps a bit :slight_smile:

6

This list of functions is applied to x and y, I want to apply something different on y.

@sgugger Basically gave me the anwser but I’m new to python and I don’t know how to add something like RandomFlip to a list of type Transforms

I don’t understand either how the first list of tfms is applied to x and the second to y when my tfms is like that:

([<fastai.transforms.Scale object at 0x7f43fe499470>, <fastai.transforms.RandomFlip object at 0x7f43fe4995f8>, <fastai.transforms.RandomDihedral object at 0x7f43fe499828>, <fastai.transforms.RandomCrop object at 0x7f44054019b0>, <fastai.transforms.Normalize object at 0x7f43fe499f98>, <fastai.transforms.ChannelOrder object at 0x7f44054010f0>],
[<fastai.transforms.Scale object at 0x7f43fe499780>, <fastai.transforms.CenterCrop object at 0x7f43fe4992b0>, <fastai.transforms.Normalize object at 0x7f43fe499f98>, <fastai.transforms.ChannelOrder object at 0x7f43fe499f28>])

y doesn’t have RandomDihedral but y still get the same transform as x when i look at them.

7

My mistake, I replied too quickly and said something wrong: in your list of transforms, the first part is applied to your training set (that’s why it has the augmentation) and the second part is applied to your validation set.
Each of the transform is applied to x and y if you want to (it depends on the argument you give to tfm_y). If you want to apply something to your ys only, this isn’t supported by fastai so you have to do it before as a preprocessing.

My guess would be a loop over all your ys. You can open an image from its filename with the open_image function. Then you can apply a specific transform to it by just typing img = tfm(img) if tfm is your Transform. Hope that helps, and sorry I misled you before.

1 Like
8

Or you could just create your own transform e.g. TransformGrayY.

9

Don’t worry about it, since I have to give the names of my images to ImageData.get_ds, does it mean that I have to write the processed images on my disk ? I’d like to avoid that.

That’s what I tried first but I can’t figure out how to only apply it to y.
Let’s take class RandomZoom :

class RandomZoom(CoordTransform):
    def __init__(self, zoom_max, zoom_min=0, mode=cv2.BORDER_REFLECT, tfm_y=TfmType.NO):
       super().__init__(tfm_y)
       self.zoom_max, self.zoom_min = zoom_max, zoom_min

    def set_state(self):
        self.store.zoom = self.zoom_min+(self.zoom_max-self.zoom_min)*random.random()

    def do_transform(self, x, is_y):
return zoom_cv(x, self.store.zoom)

How could I modifiy it to only apply to y when given in aug_tfms.

Thank you for all your help ! :slight_smile:

10

I guess you could try something like this:

def custom_y_tfm(img_x, img_y):
aug_func = RandomZoom()
return img_x, aug_func(img_y)

aug_tfms = [custom_y_tfm]

is that what are you looking for? haven’t tried it myself though :slight_smile:

11

Yes but it only takes x in input, the issue I face is syntax. Because a transform put inside aug_tfms will affect X and Y, I don’t know how to say to my tranform only affect Y.

12

did you try my example? I see that transform is implemented generically so that x and y both go into transform and come out also. By default transform function is applied on x and y is returned untouched. Maybe I am misundersanding the code and there are more nuances to it.

13

Can you show me where you see that ? I can’t find it

It gives me this when I run batches = [next(iter(md.aug_dl)) for i in range(9)]

<ipython-input-67-c04419f97430> in custom_y_tfm(img_x, img_y)
      1 def custom_y_tfm(img_x, img_y):
      2   aug_func = RandomZoom(10)
----> 3   return img_x, aug_func(img_y)
      4 
      5 aug_tfms = [custom_y_tfm]

TypeError: __call__() missing 1 required positional argument: 'y'
14

line 160 in: ‘https://github.com/fastai/fastai/blob/master/fastai/dataset.py’

def get1item(self, idx):
    x,y = self.get_x(idx),self.get_y(idx)
    return self.get(self.transform, x, y)

line 172:

def get(self, tfm, x, y):
    return (x,y) if tfm is None else tfm(x,y)
15

Yes you have this bug because aug_func requires two arguments (x and y). So let’s give it to him!

def custom_y_tfm(img_x, img_y):
    aug_func = RandomZoom(10)
    aug_y, _ = aug_func(y,None)
    return img_x, aug_y

I didn’t try this code, and I’m not sure aug_func is going to return two arguments so maybe the ,_ is unnecessary but you’ll see that easily when you test.

2 Likes
16

probably it should be:
aug_y, _ = aug_func(img_y,None)

trying to pass ‘img_y’ to augmenter

17

Oh yes, sorry.

18

Thank you both it works perfectly !

One last question is it possible to modify the transformation in the same way for the validation data set ?

19

If you want the transformation to be applied to the validation set too, you have to add it in the second elements of your tfms.
Something like tfms[1].tfms.insert(0,your_custom_tfm) should work (of course you can replace the 0 by your desired position inside the list).

At the end of the day though, it’s probably easier to stop using tfms_from_model and just build your own custom lists.

1 Like
20

You can just do this I think:

def do_transform(self, x, is_y):
  return zoom_cv(x, self.store.zoom) if is_y else x
1 Like