Error using dls.show_batch while using albumentations

harish3110 · July 24, 2020, 3:56pm

Hi there, I’m working on the “SIIM-ISIC Melanoma Classification” competition on Kaggle and wanted to use a specific data augmentation strategy I came across using the albumentation library and implemented it as follows

p = 0.5

train_tfms = A.Compose([
        A.Cutout(p=p),
        A.RandomRotate90(p=0.5),
        A.Flip(p=p),
        A.OneOf([
            A.RandomBrightnessContrast(brightness_limit=0.2,
                                       contrast_limit=0.2,
                                       ),
            A.HueSaturationValue(
                hue_shift_limit=20,
                sat_shift_limit=50,
                val_shift_limit=50)
        ], p=p),
        A.OneOf([
            A.IAAAdditiveGaussianNoise(),
            A.GaussNoise(),
        ], p=p),
        A.OneOf([
            A.MotionBlur(p=0.2),
            A.MedianBlur(blur_limit=3, p=0.1),
            A.Blur(blur_limit=3, p=0.1),
        ], p=p),
        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=p),
        A.OneOf([
            A.OpticalDistortion(p=0.3),
            A.GridDistortion(p=0.1),
            A.IAAPiecewiseAffine(p=0.3),
        ], p=0.5),
        ])

class AlbumentationsTransform(ItemTransform):
    split_idx = 0
    def __init__(self, aug,**kwargs):
        super().__init__(**kwargs)
        self.aug = aug
    def encodes(self, imgs):
        aug = self.aug(image=np.array(img))
        return PILImage.create(aug["image"])
        

train_tfms = AlbumentationsTransform(train_tfms)

But I get and error TypeError: isinstance() arg 2 must be a type or tuple of types after building my dataloader and viewing a batch as follows

bs = 32

dblock = DataBlock(
    blocks = (ImageBlock, CategoryBlock),
    get_x = ColReader('image_name', pref=path/'train', suff='.jpg'),
    get_y = ColReader('benign_malignant'),
    splitter = splitter,
    item_tfms = [Resize(256), train_tfms], 
    batch_tfms= Normalize.from_stats(*imagenet_stats))

dls = dblock.dataloaders(df, bs=bs) 
dls.show_batch(max_n=3)

harish3110 · July 24, 2020, 3:57pm

The entire error trace is as follows

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-57-90634fcc3c9e> in <module>
----> 1 dls.show_batch()

~/anaconda3/lib/python3.7/site-packages/fastai2/data/core.py in show_batch(self, b, max_n, ctxs, show, unique, **kwargs)
     97         if b is None: b = self.one_batch()
     98         if not show: return self._pre_show_batch(b, max_n=max_n)
---> 99         show_batch(*self._pre_show_batch(b, max_n=max_n), ctxs=ctxs, max_n=max_n, **kwargs)
    100         if unique: self.get_idxs = old_get_idxs
    101 

~/anaconda3/lib/python3.7/site-packages/fastai2/data/core.py in _pre_show_batch(self, b, max_n)
     85     def _pre_show_batch(self, b, max_n=9):
     86         "Decode `b` to be ready for `show_batch`"
---> 87         b = self.decode(b)
     88         if hasattr(b, 'show'): return b,None,None
     89         its = self._decode_batch(b, max_n, full=False)

~/anaconda3/lib/python3.7/site-packages/fastai2/data/core.py in decode(self, b)
     75             if isinstance(f,Pipeline): f.split_idx=split_idx
     76 
---> 77     def decode(self, b): return self.before_batch.decode(to_cpu(self.after_batch.decode(self._retain_dl(b))))
     78     def decode_batch(self, b, max_n=9, full=True): return self._decode_batch(self.decode(b), max_n, full)
     79 

~/anaconda3/lib/python3.7/site-packages/fastai2/data/core.py in _retain_dl(self, b)
     55     def _retain_dl(self,b):
     56         if not getattr(self, '_types', None): self._one_pass()
---> 57         return retain_types(b, typs=self._types)
     58 
     59     @delegates(DataLoader.new)

~/anaconda3/lib/python3.7/site-packages/fastcore/dispatch.py in retain_types(new, old, typs)
    166 def retain_types(new, old=None, typs=None):
    167     "Cast each item of `new` to type of matching item in `old` if it's a superclass"
--> 168     if not is_listy(new): return retain_type(new, old, typs)
    169     if typs is not None:
    170         if isinstance(typs, dict):

~/anaconda3/lib/python3.7/site-packages/fastcore/dispatch.py in retain_type(new, old, typ)
    160         typ = old if isinstance(old,type) else type(old)
    161     # Do nothing the new type is already an instance of requested type (i.e. same type)
--> 162     if typ==NoneType or isinstance(new, typ): return new
    163     return retain_meta(old, cast(new, typ))
    164 

TypeError: isinstance() arg 2 must be a type or tuple of types

amritv · July 24, 2020, 4:18pm

Looking at dblock.summary(df) might help with additional clues to see what the output of DataBlock is, maybe something there is affecting show_batch

harish3110 · July 24, 2020, 4:29pm

@amritv I did check that and there’s nothing I could infer from it. It’s output is as follows:

Setting-up type transforms pipelines
Collecting items from          image_name  patient_id     sex  age_approx  \
0      ISIC_2637011  IP_7279968    male        45.0   
1      ISIC_0015719  IP_3075186  female        45.0   
2      ISIC_0052212  IP_2842074  female        50.0   
3      ISIC_0068279  IP_6890425  female        45.0   
4      ISIC_0074268  IP_8723313  female        55.0   
...             ...         ...     ...         ...   
33121  ISIC_9999134  IP_6526534    male        50.0   
33122  ISIC_9999320  IP_3650745    male        65.0   
33123  ISIC_9999515  IP_2026598    male        20.0   
33124  ISIC_9999666  IP_7702038    male        50.0   
33125  ISIC_9999806  IP_0046310    male        45.0   

      anatom_site_general_challenge diagnosis benign_malignant  target  \
0                         head/neck   unknown           benign       0   
1                   upper extremity   unknown           benign       0   
2                   lower extremity     nevus           benign       0   
3                         head/neck   unknown           benign       0   
4                   upper extremity   unknown           benign       0   
...                             ...       ...              ...     ...   
33121                         torso   unknown           benign       0   
33122                         torso   unknown           benign       0   
33123               lower extremity   unknown           benign       0   
33124               lower extremity   unknown           benign       0   
33125                         torso     nevus           benign       0   

       tfrecord  width  height  
0             0   6000    4000  
1             0   6000    4000  
2             6   1872    1053  
3             0   1872    1053  
4            11   6000    4000  
...         ...    ...     ...  
33121         2   2592    1936  
33122        11   6000    4000  
33123         3   1872    1053  
33124        11   1872    1053  
33125         4   1872    1053  

[33126 rows x 11 columns]
Found 33126 items
2 datasets of sizes 26134,6558
Setting up Pipeline: ColReader -> PILBase.create
Setting up Pipeline: ColReader -> Categorize

Building one sample
  Pipeline: ColReader -> PILBase.create
    starting from
      image_name                       ISIC_2637011
patient_id                         IP_7279968
sex                                      male
age_approx                                 45
anatom_site_general_challenge       head/neck
diagnosis                             unknown
benign_malignant                       benign
target                                      0
tfrecord                                    0
width                                    6000
height                                   4000
Name: 0, dtype: object
    applying ColReader gives
      data/jpeg-melanoma-256/train/ISIC_2637011.jpg
    applying PILBase.create gives
      PILImage mode=RGB size=256x256
  Pipeline: ColReader -> Categorize
    starting from
      image_name                       ISIC_2637011
patient_id                         IP_7279968
sex                                      male
age_approx                                 45
anatom_site_general_challenge       head/neck
diagnosis                             unknown
benign_malignant                       benign
target                                      0
tfrecord                                    0
width                                    6000
height                                   4000
Name: 0, dtype: object
    applying ColReader gives
      benign
    applying Categorize gives
      TensorCategory(0)

Final sample: (PILImage mode=RGB size=256x256, TensorCategory(0))


Setting up after_item: Pipeline: AlbumentationsTransform -> Resize -> ToTensor
Setting up before_batch: Pipeline: 
Setting up after_batch: Pipeline: IntToFloatTensor -> Normalize

Building one batch
Applying item_tfms to the first sample:
  Pipeline: AlbumentationsTransform -> Resize -> ToTensor
    starting from
      (PILImage mode=RGB size=256x256, TensorCategory(0))
    applying AlbumentationsTransform gives
      (PILImage mode=RGB size=256x256, TensorCategory(0))
    applying Resize gives
      (PILImage mode=RGB size=256x256, TensorCategory(0))
    applying ToTensor gives
      (TensorImage of size 3x256x256, TensorCategory(0))

Adding the next 3 samples

No before_batch transform to apply

Collating items in a batch

Applying batch_tfms to the batch built
  Pipeline: IntToFloatTensor -> Normalize
    starting from
      (TensorImage of size 4x3x256x256, TensorCategory([0, 0, 0, 0], device='cuda:0'))
    applying IntToFloatTensor gives
      (TensorImage of size 4x3x256x256, TensorCategory([0, 0, 0, 0], device='cuda:0'))
    applying Normalize gives
      (TensorImage of size 4x3x256x256, TensorCategory([0, 0, 0, 0], device='cuda:0'))

muellerzr · July 24, 2020, 4:40pm

I’m fairly certain this is because there isn’t a decode available. IE every transform has a decodes, as they all eventually inherit Transform from fastcore, which contains a decode.

github.com

fastai/fastcore/blob/master/fastcore/transform.py#L73


    if enc:
        self.encodes.add(enc)
        self.order = getattr(enc,'order',self.order)
        if len(type_hints(enc)) > 0: self.input_types = first(type_hints(enc).values())
        self._name = _get_name(enc)
    if dec: self.decodes.add(dec)


@property
def name(self): return getattr(self, '_name', _get_name(self))
def __call__(self, x, **kwargs): return self._call('encodes', x, **kwargs)
def decode  (self, x, **kwargs): return self._call('decodes', x, **kwargs)
def __repr__(self): return f'{self.name}: {self.encodes} {self.decodes}'


def setup(self, items=None, train_setup=False):
    train_setup = train_setup if self.train_setup is None else self.train_setup
    return self.setups(getattr(items, 'train', items) if train_setup else items)


def _call(self, fn, x, split_idx=None, **kwargs):
    if split_idx!=self.split_idx and self.split_idx is not None: return x
    return self._do_call(getattr(self, fn), x, **kwargs)

So while you transform works fine, you need to either A. make it inherit Transform, or B give it a decode. (As to exactly how I haven’t looked at that personally yet in detail)

harish3110 · July 25, 2020, 6:09am

@muellerzr Thanks that worked like a charm. Just inherited from Transform as follows

train_tfms = Transform(AlbumentationsTransform(train_tfms))

Also am I right in understanding that albumentation transforms can only be run on the CPU and not the GPU?

muellerzr · July 25, 2020, 10:26am

That is correct @harish3110