I fixed the issue by passing div=True and 2 classes in the SegmentationDataset. For details please refer to this thread: https://forums.fast.ai/t/unet-binary-segmentation/29833
I was sending this param earlier also but missed passing 2 classes to the SegmentationDataset. We need to pass 2 classes when we have just one object in the mask. My fixed code looks like this:
data = (src.datasets(SegmentationDataset, classes=['background', 'cat'], div=True)
.transform(get_transforms(), size=size, tfm_y=True)
.databunch(bs=batch_size)
.normalize(imagenet_stats))