How to create multiple inputs of different types for datablock API fastai v2

[dinhtungtp (Nguyen Dinh Tung)]

My current project: using image and tags to predict highscore tags (more important tags)

Input required: 1 image, 50 tags for each image (padding if necessary), mask (whether tag is padded or not)
Output required: 1 vector of 50 binary classification (whether that tag is highscore or not)

Based on my understanding of DataBlock API, input requires ImageBlock, TagBlock, MaskBlock with n_inp=3. item_tfms and batch_tfms is only applicable for ImageBlock.

I have 2 following questions

1. How do I apply item_tfms and batch_tfms to only ImageBlock?
2. How do I show_batch that show ImageBlock only?

Does anyone have any idea how to do it? Thank you in advance

[dclipca (Dan Clipca)]

This

dblock = DataBlock(blocks=(ImageBlock, ImageBlock, CategoryBlock),
                    get_x=get_x, 
                    get_y=get_y,
                    item_tfms=[Resize(224)],
                    n_inp=2)

but you can’t use get_x /get_y . You have to pass a list with getters instead (which should have three things, the get for the first image, for the second image and then for your target).

getters = [
           ColReader('image_name', pref=path),
           ColReader('image_name', pref=path),
           ColReader('class'),
]

dblock = DataBlock(blocks=(ImageBlock, ImageBlock, CategoryBlock),
                    getters=getters, 
                    item_tfms=[Resize(224)], n_inp=2)

dls = dblock.dataloaders(df, bs=24)

Create DataBlock with Two Input Images

You need a model that accepts two inputs, It can be built with a pretrained model, and pass those two inputs to it.

[muellerzr (Zachary Mueller)]

@dclipca

but you can’t use get_x /get_y . You have to pass a list with getters instead (which should have three things, the get for the first image, for the second image and then for your target).

You actually can pass in two things to get_y. You need to set it up as:

get_x = ColReader('image_name', pref=path)
get_y = [Pipeline(ColReader('image_name', pref=path)), Pipeline(ColReader('class'))]

And that should do the trick