Feature Fusion

Hi folks,
I have two different image data-set but related to a same class. I am going to use pre-trained net like alexnet for both to detect features and then, concatenate those features into a classifier. I would appreciate if you can help me within fastai platform.

In pytorch, we use this for dataloaders, while in fastai, I do not have any idea:

class DoubleSource(torch.utils.data.Dataset):
  def __init__(self, img_i , img_ii, transform=None):
    self.img_i = img_i
    self.img_ii = img_ii
    if transform is None:
      self.transform = transforms.ToTensor()
      self.transform = transform
  def __getitem__(self, idx):
    names_i, labels = self.img_i[idx]
    names_ii, labels = self.img_ii[idx]
    input_i = Image.open(names_i).convert('RGB')
    input_ii = Image.open(names_ii).convert('RGB')
    input_i = self.transform(input_i)
    input_ii = self.transform(input_ii)
    return input_i, input_ii, labels, names_i, names_ii
  def __len__(self):
    return len(self.img_i)

In pytorch, we use this for pre-trained model, while in fastai, I do not have any idea:

class initialize_model(nn.Module):
  def __init__(self, num_classes=10, feature_extract=True, use_pretrained=True):
    super(initialize_model, self).__init__()

    model_ft = models.alexnet(pretrained=use_pretrained)
    set_parameter_requires_grad(model_ft, feature_extract)

    self.features_i = model_ft.features
    self.features_ii = model_ft.features

    self.classifier = model_ft.classifier
    self.classifier[1] = nn.Linear(2 * model_ft.classifier[1].in_features, model_ft.classifier[1].out_features)
    self.classifier[6] = nn.Linear(model_ft.classifier[6].in_features, num_classes)
  def forward(self, input_i, input_ii):
    output_i = self.features_i(input_i)
    output_i = output_i.view(output_i.size(0), -1)
    output_ii = self.features_ii(input_ii)
    output_ii = output_ii.view(output_ii.size(0), -1)
    output = torch.cat((output_i, output_ii), dim=1)
    output = self.classifier(output)
    return output

Thanks in advance,

I do not have experience with this, but considering fastai’s data block api uses pytorch dataloader, I think you can do exactly what you would do in pytorch to concatenate your pre-trained models to pass on to data block.

thanks. I will try and report if I would success