Hm, does the issue arise in line 65?
x = torch.cat([x_tab, x_img], dim=1)
If so, I can imagine two possible “gotchas”. First, the problem is formulated as a regression task, but this can cause some weird behavior if you are trying to do classification. For the latter task you’ll want to remove label_cls=FloatList in the TabularList.from_df and ImageList.from_df constructors. Second, I didn’t allow lots of flexibility when joining the tabular and image models – you can see that the layers are hardcoded-in for resnet34 and for a tabular model with only a single hidden layer.
I’m not sure how helpful this is, so sorry if that doesn’t mitigate your issue. I’ve been using fastai2 for the past six months (highly recommended!) and this kind of task is easier to do. For fastai version 1, I bet that the image_tabular package written by @ytian (see above) is also much better than my simple Gist. 