How to create multiple inputs of different types for datablock API fastai v2

My current project: using image and tags to predict highscore tags (more important tags)

Input required: 1 image, 50 tags for each image (padding if necessary), mask (whether tag is padded or not)
Output required: 1 vector of 50 binary classification (whether that tag is highscore or not)

Based on my understanding of DataBlock API, input requires ImageBlock, TagBlock, MaskBlock with n_inp=3. item_tfms and batch_tfms is only applicable for ImageBlock.

I have 2 following questions

  1. How do I apply item_tfms and batch_tfms to only ImageBlock?
  2. How do I show_batch that show ImageBlock only?

Does anyone have any idea how to do it? Thank you in advance


dblock = DataBlock(blocks=(ImageBlock, ImageBlock, CategoryBlock),

but you can’t use get_x /get_y . You have to pass a list with getters instead (which should have three things, the get for the first image, for the second image and then for your target).

getters = [
           ColReader('image_name', pref=path),
           ColReader('image_name', pref=path),

dblock = DataBlock(blocks=(ImageBlock, ImageBlock, CategoryBlock),
                    item_tfms=[Resize(224)], n_inp=2)

dls = dblock.dataloaders(df, bs=24)

You actually can pass in two things to get_y. You need to set it up as:

get_x = ColReader('image_name', pref=path)
get_y = [Pipeline(ColReader('image_name', pref=path)), Pipeline(ColReader('class'))]

And that should do the trick :smiley: