I got the exact same error-message when trying to run object detection on the coco-dataset. I have been using a fast.ai v1 implementation by @Bronzi88, mentioned in this thread: Object detection in fast.ai v1
I tried the examples/CocoTiny_Retina_Net.ipynb from the repo (https://github.com/ChristianMarzahl/ObjectDetection), and got the error message. I think you have identified the correct issue - great debugging btw! Using squishing instead of cropping when resizing seems to be the issue. This is also discussed by Jeremy in lesson 8 (v2) at https://youtu.be/Z0ssNAbe81M?t=5454
A fix that worked for me was using the following code for the databunch:
data = (ObjectItemList.from_folder(
.split_by_rand_pct()
.label_from_func(get_y_func)
.transform(tfm_y=True, size=size, resize_method=ResizeMethod.SQUISH)
.databunch(bs=64, collate_fn=bb_pad_collate))
Finally it seems certain tranforms like too much rotation can cause the same issue.
Regards
Hallvar