What should getter for BBoxBlock return?

Part 1 (2020)
1

hi,

I would like to create a model that takes an image (ImageBlock) as input and a bounding box (BBoxBlock) as output, so:

datablock = DataBlock(
    blocks = (ImageBlock, BBoxBlock), 
    get_items = get_image_paths, 
    get_y = get_y, 
    ... )

but I don’t know what get_y( ) should return?

I wasn’t able to browse the source code for BBoxBlock, so tried best-guessing a number of different functions for get_y( ), like:

def get_bounding_box1x4(x) :
  result = tensor([10.0, 10.0, 20.0, 20.0])
  print("get_bounding_box1x4(x)", str(x),"result", result, "shape", result.shape)
  return result

def get_bounding_box1x8(x) :
  result = tensor([10.0, 20.0, 20.0, 20.00, 10.0, 30.0, 20.0, 30.0 ])
  print("get_bounding_box1x8(x)", str(x),"result", result, "shape", result.shape)
  return result

def get_bounding_box1x4x2(x) :
  result = tensor([[100.0, 200.0], [200.0, 200.00], [100.0, 300.0], [200.0, 300.0] ])
  print("get_bounding_box1x4(x)", str(x),"result", result, "shape", result.shape)
  return result

def get_bounding_box4x2(x) :
  result = tensor([100.0, 200.0], [200.0, 200.00], [100.0, 300.0], [200.0, 300.0])
  print("get_bounding_box4x2(x)", str(x),"result", result, "shape", result.shape)
  return result

def get_bounding_box2x2(x) :
  result = tensor([100.0, 200.0], [200.0, 200.00])
  print("get_bounding_box2x2(x)", str(x),"result", result, "shape", result.shape)
  return result

def get_TensorBBox(x) :
  xmin = 100.0
  ymin = 100.0
  xmax = 200.0
  ymax = 200.0
  result = TensorBBox.create([xmin, ymin, xmax, ymax])
  print("get_TensorBBox(x)", str(x),"result", result, "shape", result.shape)
  return result

but none of them seem to make sense, because creating dataloaders always fails:

dls = datablock.dataloaders(Path(project_folder),path=Path(project_folder),bs=4)

with output like:

get_image_paths(path) /content/data/FastAIDeDup03
get_TensorBBox(x) BBOX(83,143,52,169).jpg
result TensorBBox([[100., 100., 200., 200.]]) shape torch.Size([1, 4])
get_TensorBBox(x) BBOX(83,143,52,169).jpg
result TensorBBox([[100., 100., 200., 200.]]) shape torch.Size([1, 4])
get_TensorBBox(x) BBOX(83,143,52,169).jpg
result TensorBBox([[100., 100., 200., 200.]]) shape torch.Size([1, 4])
Could not do one pass in your dataloader, there is something wrong in it

What should get_y( ) return?

1 Like
2

There are examples both in this notebook and nb50:

However BBoxBlock should be used with BBoxLblBlock, also when debugging you should do datablock.summary(path) as it will tell you exactly where the issue lies. (I have not tried doing one and not the other, hence why I presume that. datablock.summary will be able to tell us much more information though)

2 Likes
3

Thank you VERY much.

Just to share my progress: this taught me four things which might be useful to other newbies:

  1. when using BBoxBlock, one should also use BBoxLblBlock
  2. the getter function for BBoxBlock should return a list of bounding boxes, and each bounding box is simply a list of 4 coordinates [x1,y1,x2,y2]
  3. the getter function for BBoxLblBlock should return a list of strings
  4. these two getter functions need to be combined in a list of getters

getters for BBoxBlock AND BBoxLblBlock:

def get_bounding_box(x) :
  result = [[100.0,100.0,200.0,200.0]] # dummy values obviously
  print("get_bounding_box(x)", x, "result", result)
  return result

def get_label(x) :
  result = ["bla"] # dummy values obviously
  print("get_label(x)", x, "result", result)
  return result

# combined 'getter'
get_y = [ get_bounding_box, get_label ]

datablock

datablock = DataBlock(
    blocks = (ImageBlock, BBoxBlock, BBoxLblBlock), 
    n_inp=1, # now needed, because we have > 2 entries in 'blocks' 
    get_items = get_image_paths, 
    get_y = get_y 
)
7 Likes
4

Thanks a lot, this is really helpful! Actually, this should be in the documentation, as its hard to find such information.

5

A PR would be most welcome :slight_smile:

6

can anyone please help me out?
if i have to use my own function instead of get_image_paths.
how can i do that?

7

If you are using your own function, you can setup your datablock and loader as below:

db_block = DataBlock( blocks = (ImageBlock, CategoryBlock), 
                     get_items =  get_df_lists, 
                     get_x = get_x_imgCrop,
                     get_y = get_y_label,
                     splitter = RandomSplitter(valid_pct = 0.2, seed = 43),
                     item_tfms = item_tfms,
                     batch_tfms = batch_tfms)

Here I define a function get_df_lists. It needs to return a list of objects. A single object from this list will be the input for the getters: get_x, and get_y. For example, get_df_lists returns a list object with each row as the rows of a dataframe. My get_y_label is defined as:

def get_imgCrop_test (o): ## o is expected to be a list with several items. The first item is the filename
       return PILImage.create(o[0])

def get_label_test (o): ## Parent of file is the class
    class_label = int(str(Path(o[0]).parent))
    return class_label

You would need to define something similar for your bounding box information

8

ok! great
where can I get a tutorial on fastai object detection with an updated version?

9

Hi @LambertWM @muellerzr ,

I am also trying to do an object detection with external data.
When I try to create dataloaders with below command:
dls = siim.dataloaders(path/'train')

It fails and throws below error, kindly help if possible.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-228fd72431ff> in <module>
----> 1 dls = siim.dataloaders(path/'train')

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/data/block.py in dataloaders(self, source, path, verbose, **kwargs)
    111 
    112     def dataloaders(self, source, path='.', verbose=False, **kwargs):
--> 113         dsets = self.datasets(source, verbose=verbose)
    114         kwargs = {**self.dls_kwargs, **kwargs, 'verbose': verbose}
    115         return dsets.dataloaders(path=path, after_item=self.item_tfms, after_batch=self.batch_tfms, **kwargs)

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/data/block.py in datasets(self, source, verbose)
    108         splits = (self.splitter or RandomSplitter())(items)
    109         pv(f"{len(splits)} datasets of sizes {','.join([str(len(s)) for s in splits])}", verbose)
--> 110         return Datasets(items, tfms=self._combine_type_tfms(), splits=splits, dl_type=self.dl_type, n_inp=self.n_inp, verbose=verbose)
    111 
    112     def dataloaders(self, source, path='.', verbose=False, **kwargs):

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/data/core.py in __init__(self, items, tfms, tls, n_inp, dl_type, **kwargs)
    308     def __init__(self, items=None, tfms=None, tls=None, n_inp=None, dl_type=None, **kwargs):
    309         super().__init__(dl_type=dl_type)
--> 310         self.tls = L(tls if tls else [TfmdLists(items, t, **kwargs) for t in L(ifnone(tfms,[None]))])
    311         self.n_inp = ifnone(n_inp, max(1, len(self.tls)-1))
    312 

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/data/core.py in <listcomp>(.0)
    308     def __init__(self, items=None, tfms=None, tls=None, n_inp=None, dl_type=None, **kwargs):
    309         super().__init__(dl_type=dl_type)
--> 310         self.tls = L(tls if tls else [TfmdLists(items, t, **kwargs) for t in L(ifnone(tfms,[None]))])
    311         self.n_inp = ifnone(n_inp, max(1, len(self.tls)-1))
    312 

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastcore/foundation.py in __call__(cls, x, *args, **kwargs)
     95     def __call__(cls, x=None, *args, **kwargs):
     96         if not args and not kwargs and x is not None and isinstance(x,cls): return x
---> 97         return super().__call__(x, *args, **kwargs)
     98 
     99 # Cell

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/data/core.py in __init__(self, items, tfms, use_list, do_setup, split_idx, train_setup, splits, types, verbose, dl_type)
    234         if do_setup:
    235             pv(f"Setting up {self.tfms}", verbose)
--> 236             self.setup(train_setup=train_setup)
    237 
    238     def _new(self, items, split_idx=None, **kwargs):

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/data/core.py in setup(self, train_setup)
    256             for f in self.tfms.fs:
    257                 self.types.append(getattr(f, 'input_types', type(x)))
--> 258                 x = f(x)
    259             self.types.append(type(x))
    260         types = L(t if is_listy(t) else [t] for t in self.types).concat().unique()

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastcore/transform.py in __call__(self, x, **kwargs)
     71     @property
     72     def name(self): return getattr(self, '_name', _get_name(self))
---> 73     def __call__(self, x, **kwargs): return self._call('encodes', x, **kwargs)
     74     def decode  (self, x, **kwargs): return self._call('decodes', x, **kwargs)
     75     def __repr__(self): return f'{self.name}:\nencodes: {self.encodes}decodes: {self.decodes}'

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastcore/transform.py in _call(self, fn, x, split_idx, **kwargs)
     81     def _call(self, fn, x, split_idx=None, **kwargs):
     82         if split_idx!=self.split_idx and self.split_idx is not None: return x
---> 83         return self._do_call(getattr(self, fn), x, **kwargs)
     84 
     85     def _do_call(self, f, x, **kwargs):

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastcore/transform.py in _do_call(self, f, x, **kwargs)
     87             if f is None: return x
     88             ret = f.returns(x) if hasattr(f,'returns') else None
---> 89             return retain_type(f(x, **kwargs), x, ret)
     90         res = tuple(self._do_call(f, x_, **kwargs) for x_ in x)
     91         return retain_type(res, x)

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastcore/dispatch.py in __call__(self, *args, **kwargs)
    116         elif self.inst is not None: f = MethodType(f, self.inst)
    117         elif self.owner is not None: f = MethodType(f, self.owner)
--> 118         return f(*args, **kwargs)
    119 
    120     def __get__(self, inst, owner):

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/vision/core.py in create(cls, x, img_size)
    197     "Basic type for a tensor of bounding boxes in an image"
    198     @classmethod
--> 199     def create(cls, x, img_size=None)->None: return cls(tensor(x).view(-1, 4).float(), img_size=img_size)
    200 
    201     def show(self, ctx=None, **kwargs):

~/miniconda3/envs/fast/lib/python3.8/site-packages/fastai/torch_core.py in tensor(x, *rest, **kwargs)
    125     # if isinstance(x, (tuple,list)) and len(x)==0: return tensor(0)
    126     res = (x if isinstance(x, Tensor)
--> 127            else torch.tensor(x, **kwargs) if isinstance(x, (tuple,list))
    128            else _array2tensor(x) if isinstance(x, ndarray)
    129            else as_tensor(x.values, **kwargs) if isinstance(x, (pd.Series, pd.DataFrame))

ValueError: too many dimensions 'str'
10

Can you share the code that you’re using to create siim?

11

Hi, this is the code, I mostly referred to Zack’s notebook

And the train.json is of the format of Coco data.

imgs, lbl_bbox = get_annotations(path/'train.json')

img2bbox = dict(zip(imgs, lbl_bbox))

getters = [lambda o: path/'train'/o, lambda o: img2bbox[o][0], lambda o: img2bbox[o][1]]

def get_train_imgs(noop):  return imgs

siim = DataBlock(blocks=(ImageBlock, BBoxBlock, BBoxLblBlock),
                 splitter=RandomSplitter(),
                 get_items=get_train_imgs,
                 getters=getters,
                 item_tfms=item_tfms,
                 batch_tfms=batch_tfms,
                 n_inp=1)
                 
dls = siim.dataloaders(path/'train')
1 Like
12

Hi @KevinB , any hint what I might be doing wrong here?

13

Can you share your item_tfms and batch_tfms code as well?

I haven’t been able to recreate your issue so far

14

I do run into this issue when I try running your code but it works with torch 1.7.1 and the error message is different so I think you’re seeing something else.

15

I got the problem, there was a issue with my train.json. And it is now fixed, thanks for asking right questions :slight_smile:

1 Like