In image segmentation when we are decreasing original image size for training, lets say by 2 for both dimension, what transformation we do for the ground truth labels for the pixels?
It seems that in this case each 4 pixels of the original image become a single pixel of the transformed image, what are we doing with labels of these pixels?
The targets is reduced in the same way, using nearest interpolation to determine the label of the pixel.
Thanks, but how can you do interpolation for the labels in this case?
A segmentation label should have a discrete value. And if you create 1 pixels from 4, you may have cases when all 4 of them had different segment labels or 2 has one and 2 other has second one.
How the algorithm will choose in this case which label to keep for a reduced image?
Any update on this @sgugger or @vladgets? It seems to limit the types of transforms that can be applied to Edited my reply because ImageSegments for segmentation problems.open_mask handles this correctly, image segment data should be stored internally as floats and the resampling methods can be found here
# image data (0-1 as float) can be transformed
img = Image(torch.randn(3, 224, 224))
img.zoom(scale=0.8)
img.data
# prints pixel data normally
# segmentation label data (0-n as long) cannot be transformed
label = ImageSegment(torch.randint(0, 10, (3, 224, 224)))
label.zoom(scale=0.8)
label.data
# RuntimeError: grid_sampler(): expected input and grid to have same dtype, but input has long long and grid has float
# segmentation label data should still be stored internally as float
# and then everything will work out nicely
label_proper = ImageSegment(torch.randint(0, 10, (3, 224, 224)).float())
label_proper.zoom(scale=0.8)
label_proper.data
# all good
I have published a detailed guide on object extraction (using instance/image segmentation) here. It can work with any size of images. The guide includes a prebuilt docker image and code needed to extract the objects from an image.