Resizing image segmentation masks introduce new classes because of pixel value interpolation!

Part 1 (2019)
1

I was working on a pixel-level segmentation project with multiple classes using fastai. So in my example, there are 4 classes, and the information is encoded as integers 0 to 3.

I recently discovered when I resize my image in openCV, it would interpolate pixel values. So for example, my original mask may only has 0s and 3s, but after resizing I’m getting a small amount of 1s and 2s. While for an image this would be fine, this is a segmentation mask, and a different integer means a different class entirely, and I’m wondering if this is messing up my model.

Has anyone run into this issue? FYI, this resizing was done outside fastai using my own scripts, and now I’m wondering if the fastai resizing operations also introduce this artifact???

2

Fastai applies transformations to imagesegment class differently to prevent those issues. You can see in the implementation that it changes interpolation mode to be nearest:

1 Like
3

I think I’m having this issue right now, despite my encodes() function says that the unique values in that specific mask are only [0, 2]. The major suspect at this point is the resize function.

image

4

Can you post your Datablock/Dataloader code?

5

Obviously it was my fault (as usual). I had to do a little trick to be able to load RGB segmentation masks and convert them on the fly to grayscale. I wanted this feature because I wanted to be able to look at masks on disk “with the naked eye” and if you keep them in grayscale… well, you just see black. Anyway, I was doing this:

image

but it was not sufficient, Resize() in fastai/vision/augment.py was thinking those PILMasks were not true fastai.vision.core.PILMasks, so between the tuple resamples=(Image.BILINEAR, Image.NEAREST) it was applying Image.BILINEAR to my “false PILMasks”. I just added a cast to fastai.vision.core.PILMask at the end of my encodes() function and now it seems that Resize() behaves correctly.

1 Like
6

Don’t know if it will help but there is some sort of custom resizer in the segmentation example:

class CustomResizer(Transform):
    order=1
    "CutomResizer that resizes images with BILINEAR and masks with NEAREST"
    def __init__(self, size, resample=Image.BILINEAR):
        if not is_listy(size): size=(size,size)
        self.size,self.resample = (size[1],size[0]),resample

    def encodes(self, o:PILImage): return o.resize(size=self.size, resample=self.resample)
    def encodes(self, o:PILMask):  return o.resize(size=self.size, resample=Image.NEAREST)

I got this from here: Tutorial - Custom transforms | fastai