What to do when data has 2 inputs, instead of 1?

I am implementing Semantic Image Synthesis with Spatially-Adaptive Normalization. Generally, when we have to fit models, we create the Learner object or create_cnn, but for this paper, it is a bit complicated.

The model arch is as follow, pass some noise as input, and add segmentation map to the result of conv layers in between the model (the output of the model is colored image). The problem is when we create the learner objects we have to pass data object also. Now we can create some data object, but I think there is no way to tell the Learner object on how to use those inputs for this task (Also, in general, I have used fit_one_cycle in cases where there is a single input and output, but in this case I have 2 inputs[noise and seg map] and 1 output (colored seg map)

So to solve this problem, I can think of the usual soln of batch training by for x in data.train_ds and then train it. But I would not be able to use cyclicLR in this way (or the benefits that come with fit_one_cycle like AdamW, cyclic mom) method.

Any suggestions on how to proceed here.

Also, I have struggling to understand, how we can use cyclic momentum with Adam.

I have been going through the source code of fastai, and can see that when we call fit_one_cycle it is nothing but some callback inits and other code and then we call fit, which is something that is easy to implement. So, I think we have to create a new fit method specific to our need, which is not difficult I think after seeing the code.

But if someone, can comment is it the right method to deal with this 2 input situation or there are some other hacks.

If you look at the source code within basic_train.py you’ll see there’s a global scope method called fit. fit iterates through the batches that the data loader passes it

for xb,yb in progress_bar(dl, parent=pbar, leave=(pbar is not None)):
            if cb_handler: xb, yb = cb_handler.on_batch_begin(xb, yb, train=False)
            val_loss = loss_batch(model, xb, yb, loss_func, cb_handler=cb_handler)
            val_losses.append(val_loss)
            if not is_listy(yb): yb = [yb]
            nums.append(yb[0].shape[0])
            if cb_handler and cb_handler.on_batch_end(val_losses[-1]): break
            if n_batch and (len(nums)>=n_batch): break

To handle the forward and backward passes on a batch it calls out to loss_batch (also in the global scope of this file).

It passes loss_batch the model and the x and y for the batch (xb and yb). If the xb and yb aren’t already list-like it turns them into trivial lists by wrapping them with [ ]. Then for the forward pass the model(*xb) is called (i.e. the forward function of model is called and the x in the batch is unpacked.

My understanding is that to get two different pieces of data into the beginning of your network you need to define your own ItemBase class with the suitable structure for your inputs e.g. ImageNoiseItem and then on your network’s forward function appropriately unpack and process the data.

2 Likes

Yes exactly. As long as your custom ItemList returns the list of things you want and you model accepts two inputs, this will work.
Look at the tabular section: a tabular model takes two tensors and TabularList in its get method returns two items as a list.

4 Likes

Turns out I overthinked a little on this. I just need a custom itemlist that returns x as tuple and in the model forward pass I can unwrap that tuple.

Also, can someone explain momentum in Adam

1 Like

When I ran into those kind of hurdles, I decided I should learn enough Pytorch to not depend on fastai.

This is how I would proceed for a 2 input model:

  1. Build your model as a subclass of torch.nn.Module

Your forward function will have 2 inputs: def forward(self, x1, x2)

  1. Build your dataset as a subclass of torch.utils.data.Dataset

Have your getitem function return [x1, x2] , y
Where x1 and x2 are your 2 inputs in a list, and y is your ground truth/label.

  1. Create your train and validation (potentially test as well) datasets using the above subclass.
    e.g.

train_ds = myDataset (args)

  1. Create your model object:

net = myNet(args)

  1. Define your device and criterion:
    e.g.:

use_cuda = torch.cuda.is_available()
device = torch.device(“cuda:0” if use_cuda else “cpu”)
criterion = nn.L1Loss()

  1. Now at this point you can build your own training loop (using a dataloader) or use fastai.

To use fastai, we can do for example:

from fastai.basics import *
databunch = DataBunch.create(trn_ds,val_ds, device=device, bs =32)
learn = Learner(databunch,net,callback_fns=[ShowGraph], loss_func = criterion)
lr = 1e-2
learn.fit(2,lr)

Obviously I skipped a lot of the details. Mainly you have to figure out how to write your model and dataset classes.

I find that it is a useful skill to have because most people don’t use fastai; so if you know enough Pytorch you can reuse other people’s Pytorch code and still use the nice fastai training interface. It also gives you all the flexibility of Pytorch if you are creating unusual models and datasets.

Let me know if you have any questions!

6 Likes

Yeah I agree we should not always depend on fastai. So I also, started working on some notebooks, where I implement important fastai functionality in pytorch so you don’t have to import fastai. I really like learning from fastai code.

I started on it a few days back and only lr_find, plot_lr and fit_one_cycle is in progress.

If someone wants to check, the link. I still have to comment the notebook though.

Hello, I’m new to coding, I have a similar problem, and I was wondering if anyone would be able to help me out. I’m trying to create a Custom ItemList that returns an image (an xray of a hand) along with either a 1 or a 0 as inputs (1 and 0 are for gender; 1 means male and 0 means female). The data set I’m using is from a pediatric bone age prediction challenge from 2017 (which has since ended). I’m trying to predict the age of the patient, based on the image and the gender.

Data can be found here.
More information can be found here.

This is what my dataframe looks like:

df.head()

id boneage IsMale
0 1377 180 0.0
1 1378 12 0.0
2 1379 94 0.0
3 1380 120 1.0
4 1381 82 0.0

I’m trying to follow this tutorial on creating a Custom ItemList however my custom ItemList differs from the example in 2 ways: I’d like to have an image and a number as an input (instead of 2 images), and I’d like to use .from_csv instead of .from_folder.

This is my attempt at a custom ItemList:

class ImageTuple(ItemBase):
    def __init__(self, img1, gender):
        self.img1,self.gender = img1,gender
        self.obj,self.data = (img1,gender),[-1+2*img1.data,gender.data]

class CustomItemList(ItemList):
    def __init__(self, items, itemsB=None, **kwargs):
        super().__init__(items, **kwargs)
        self.itemsB = itemsB
        self.copy_new.append('itemsB')
    def get(self, i):
        img1 = super().get(i)
        itemsB = df['IsMale']
        gender = self.itemsB[i]
        return ItemTuple(img1, gender)

When I attempt to create a databunch with this ItemList, I get an error:

<ipython-input-13-3410e54c85a8> in __init__(self, items, itemsB, **kwargs)
      1 class ImageTupleList(ItemList):
      2     def __init__(self, items, itemsB=None, **kwargs):
----> 3         super().__init__(items, **kwargs)
      4         self.itemsB = itemsB
      5         self.copy_new.append('itemsB')

TypeError: __init__() got an unexpected keyword argument 'folder'

Or when I try to use the ImageList class instead of ItemList I get:

<ipython-input-10-d9d9dcb87708> in get(self, i)
     12         img1 = super().get(i)
     13         itemsB = df['IsMale']
---> 14         gender = self.itemsB[i]
     15         return ImageTuple(img1, gender)
     16 #     def from_folders(cls, path, folderA, folderB, **kwargs):

TypeError: 'NoneType' object is not subscriptable

On github, I found a custom dataset loader for pytorch for this dataset available here under class CelebaDataset(Dataset) ,however, I’m not sure how to plug this into the fastai library.

There may be a simple solution for someone who is more experienced but this has become a brick wall for a noob like myself, so if someone knows the answer and can show me how to do this that would be greatly appreciated!

Thanks!