Batch transform not being applied to bounding box

fastai
1

I’m having some issues with applying transformations to items that have a bounding box.
In this image no transformations were applied and the bounding boxes are applied correctly

Screen Shot 2021-07-28 at 10.16.46 am
Screen Shot 2021-07-28 at 10.16.46 am1814×1106 340 KB

When I apply this transformations

batch_tfms = [FlipItem(),
              RandomResizedCrop(128, min_scale=0.35),
              IntToFloatTensor(),
              Normalize.from_stats(*imagenet_stats)]

The bounding boxes don’t seem to be transformed with the image making them appear out of place

Screen Shot 2021-07-28 at 10.22.50 am
Screen Shot 2021-07-28 at 10.22.50 am1802×710 219 KB

Also couldn’t figure out how to use show_batch with the same images but after having a look at a few samples this seems like a legit issue.
Any help would be appreciated.
Thanks

2

How are you creating dls, and how are you applying batch_tfms?

3

Hi @idraja
here’s a snippet of the setup

dog_images = glob.glob(f'{path}/images/Images/*/*')
getters = [lambda x: f"{get_image_path(img2bbox[x][1][0])}/{x}",# get file path 
           lambda x: img2bbox[x][0], # get BB 
           lambda x: img2bbox[x][1]] # get class name
def get_train_imgs(noop): return append_extention(imgs)
db = DataBlock(blocks=(ImageBlock, BBoxBlock, BBoxLblBlock),
               splitter=RandomSplitter(),
               get_items=get_train_imgs,
               getters=getters,
               item_tfms=item_tfms,
               batch_tfms=batch_tfms,
               n_inp=1)
dls = db.dataloaders(dog_images)
4

Can you show the code for displaying the batch where the bounding boxes are misplaced?

5

Here’s the full notebook, this is the version not working if I remove the batch_tfms it works fine
dog_breeds_fastai_v4.ipynb - Colaboratory.pdf (1.1 MB)

6

The issue comes from RandomResizedCrop without it the bounding boxes are in the correct place even when using other transforms.
I can’t find any documentation around what transforms can be used with bounding boxes

7

I think what you need to do is move FlipItem and RandomResizedCrop to item_tfms.

Also fyi RandomResizedCrop might cut out some of your bounding boxes entirely. I tried this approach out with the COCO_TINY dataset and it seems to be working, though that dataset has some weird annotations so it’s hard to tell. Share back what you find by moving those two transforms to item_tfms as I’m curious to see if it works!

1 Like
8

I can confirm that moving FlipItem and RandomResizedCrop to item transformations works fine.
It’s a shame that can’t be done on a batch level but from what I can see it makes sense since the BBs are applied per item transformations that change the item position can’t be done on a batch level

Thanks for your time @idraja

1 Like