[Solved] Forward pass error on multiple input CNN

fastai
1

Still working on V1 Fastai. I have a DataBunch that generates an X tuple with 2 inputs that I’m attempting to feed into a CNN with a double input base. I’m getting an error that seems to be coming from the way FastAi is unwrapping the data and feeding it to the final model. I have tried unpacking it a few ways but can’t quite figure out how to pass the data streams down to the model base.

Gist of 2 input model
Gist of DataBunch Preparation

TypeError: forward() takes 2 positional arguments but 3 were given

data
ImageDataBunch;

Train: LabelList (29901 items)
x: MultiChannelImageTupleList
(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512))
y: CategoryList
513,840,1020,254,144
Path: recursion_data\train;

Valid: LabelList (6614 items)
x: MultiChannelImageTupleList
(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512)),(Image (6, 512, 512), Image (6, 512, 512))
y: CategoryList
239,914,450,657,986
Path: recursion_data\train;

Test: None
2

The inputs are destructured when fed to your model, so your forward method should be def forward(self, x1, x2):

See the tabular model for an example (tabular data is a list of two things, cats and conts, for the input)

1 Like
3

Hey thanks! I tried that, and got the following errors:

class DoubleModel(nn.Module):
    def __init__(self, mod1, mod2): 
        super().__init__()
        self.mod1 = mod1
        self.mod2 = mod2

    def forward(self, x):
        x_mod1 = self.mod1(x[0])
        x_mod2 = self.mod2(x[1])
        x = nn.Sequential(torch.cat([x_mod1, x_mod2], dim=1))
        return x

class FullModel(nn.Module):
    def __init__(self, base):
        super().__init__()
        self.base = DoubleModel(base, base)        
                
        nf = (num_features_model(*self.base.children())) * 2
        head = create_head(nf=nf, nc=data.c, ps=0.5,lin_ftrs=None, concat_pool=True, bn_final=False)
        self.head = head
        
    def forward(self, x1, x2):
        base = self.base((x1, x2))
        return nn.Sequential(base, head)

TypeError: torch.cuda.FloatTensor is not a Module subclass

I think it has to do with how FullModel is calling DoubleModel and feeding the tensors through. See code below. I also tried

class DoubleModel(nn.Module):
    def __init__(self, mod1, mod2): 
        super().__init__()
        self.mod1 = mod1
        self.mod2 = mod2

    def forward(self, x1, x2):
        x_mod1 = self.mod1(x1)
        x_mod2 = self.mod2(x2)
        y = nn.Sequential(torch.cat([x_mod1, x_mod2], dim=1))
        return y

class FullModel(nn.Module):
    def __init__(self, base):
        super().__init__()
        self.base = DoubleModel(base, base)        
                
        nf = (num_features_model(*self.base.children())) * 2
        head = create_head(nf=nf, nc=data.c, ps=0.5,lin_ftrs=None, concat_pool=True, bn_final=False)
        self.head = head
        
    def forward(self, x1, x2):
        x = self.base(x1, x2)
        return self.head(x)

TypeError: torch.cuda.FloatTensor is not a Module subclass

The obvious solution for the data flow would be to move the two-arm definition into the same class, but I structured it this way because I realized I needed it structured as a submodule for learn.freeze() work properly. When I tried to define the layers to freeze when the model was generated as a single class I was getting a “repeat parameter” error I think because I’m duplicating the same base model.

4

Solved this! I was wrapping the torch.cat in nn.Sequential. This plus the input modification fixed the issue. I’ve had a lot of trouble sorting out this 2-armed model, so I’m updating the Gist in case anybody comes across this in the future (though who knows if it will work in V2)

1 Like