Item_tfms not being applied for custom ImageBlock class

Part 1 (2020) Non-beginner
1

I’m using a custom class called grizyImageBlock for five-channel imaging (g = green, r = red, and i, z, and y are silly names for infrared channels). However, I’ve been noticing that my item_tfms aren’t being applied at all (e.g., Resize() or CropPad()):

The class looks like this:

def open_npy(fn):
    im = torch.from_numpy(np.load(str(fn), allow_pickle=True))
    return np.nan_to_num(im)                   


class grizyTensorImage(TensorImage):
    _show_args = ArrayImageBase._show_args
    def show(self, ctx=None, vmin=None, vmax=None, **kwargs):
        return show_composite(self, ctx=ctx, vmin=vmin, vmax=vmax, **{**self._show_args, **kwargs})

    @classmethod
    def create(cls, fn,  **kwargs) ->None:
        if str(fn).endswith('.npy'): return cls(open_npy(fn=fn))
        
    def __repr__(self): return f'{self.__class__.__name__} size={"x".join([str(d) for d in self.shape])}'
    
grizyTensorImage.create = Transform(grizyTensorImage.create)

The goal is to do multivariate regression, so blocks=(grizyImageBlock, RegressionBlock) are being passed to the DataBlock constructor. When I call dblock.summary(df), I can confirm that item_tfms=CropPad(56) is not being applied:

Building one batch
Applying item_tfms to the first sample:
  Pipeline: CropPad -> ToTensor
    starting from
      (grizyTensorImage of size 5x224x224, tensor([-0.0775,  0.3370,  0.6503, -0.8647,  0.6967,  0.2905]))
    applying CropPad gives
      (grizyTensorImage of size 5x224x224, tensor([-0.0775,  0.3370,  0.6503, -0.8647,  0.6967,  0.2905]))
    applying ToTensor gives
      (grizyTensorImage of size 5x224x224, tensor([-0.0775,  0.3370,  0.6503, -0.8647,  0.6967,  0.2905]))

Has anybody encountered this problem?

EDIT: I also have the same issue for 3-channel images that are opened via another custom class, FITSImageBlock. This uses the astropy.io.fits file-handling system before converting to Pytorch Tensor objects. In this case as well, item_tfms are not being applied.

2

If my understanding is correct, CropPad runs before images are converted to tensors, and yours are already created as tensors. Looking at the source code for CropPad, it looks like it knows how to handle three different types:

def encodes(self, x:(Image.Image,TensorBBox,TensorPoint)):

Your grizyTensorImage is not a subclass of any of those types, so CropPad ignores it.

You could @patch the encodes function to provide an implementation for your type, but I’m not sure how to go about reusing the current code in the library without repeating it yourself.

2 Likes
3

Yeah, the underlying method for cropping images (Image.Image) uses the pillow library PIL.Image.crop so that won’t work. I think the easiest route would be to write your own function for cropping tensors and pass that as a transform.

1 Like