Z - you cover the standard fastai tabular as well as xgboost and random forests. Do you ever cover exporting trained embeddings to the tree methods? Could always export the embeddings and recreate the full data layer outside fastai, but seems like there should be a way to hijack the fastai feed into a prepped array for xgboost training.
My main problem is that in the example for the additional data they just pass the raw tensor. I would like, instead, to pass the embeddings of the tabular learner before they go to the fc layers. I would then concatenate and then do the fc. Any ideas on how to adapt this code for fastai?
self.cnn = models.inception_v3(pretrained=False, aux_logits=False)
self.cnn.fc = nn.Linear(
self.fc1 = nn.Linear(20 + 10, 60)
self.fc2 = nn.Linear(60, 5)
def forward(self, image, data):
x1 = self.cnn(image)
x2 = data
x = torch.cat((x1, x2), dim=1)
x = F.relu(self.fc1(x))
x = self.fc2(x)
Sure thing! So in regards to what this will presume, you know of the MixedDLhere
So now let’s go through our steps.
We’ll build some Tabular DL’s and vision DL’s we wish to make for our MixedDL.
When we get to the Tabular portion, we will want to calculate the embedding matrix size. We do this with get_emb_sz(to) (with the to object being dl.train on the Tabular DL)
We’ll make a Tabular Embedding only model, as this is all we want. this code looks like so:
"Basic model for tabular data."
def __init__(self, emb_szs, embed_p=0.):
ps = ifnone(ps, *len(layers))
self.embeds = nn.ModuleList([Embedding(ni, nf) for ni,nf in emb_szs])
self.emb_drop = nn.Dropout(embed_p)
def forward(self, x_cat, x_cont=None):
if self.n_emb != 0:
x = [e(x_cat[:,i]) for i,e in enumerate(self.embeds)]
x = torch.cat(x, 1)
x = self.emb_drop(x)
All this model does is take our input (which must be a tabular cat+cont if we’re following that example) (if there is no continuous it passes in an empty tensor)
So now we can build our model by passing in the emb_sz
Now we need our vision model. Both of these models can be thought of as “bodies”, and we’ll make a head for them all. So for our Vision model, we’ll call create_body(resnet50) and this is the body of our model
Now we get to the meat and potatoes. We have two bodies at this point, we need to make it into a cohesive model. First thing we want to do is concatenate their outputs before passing it to some head. But how do we calculate this? We’ll take both our models and call num_features_model(model). For instance a resnet50 will have 2048. We’ll pretend our other model has an output of 224. As a result, post concatenation we can presume the size would be 2048+224
Now we can call create_head(2048+224, num_classes) to create our head. Finally, we need to define a model. This model should accept both of our bodies as an input, calculate a head, and then in the forward function take care of everything:
def __init__(self, tab_body, vis_body, c):
self.tab, self.vis = tab_body, vis_body
nf = num_features_model(self.tab) + num_features_model(self.vis)
self.head = create_head(nf*2, c)
def forward(self, *x):
cat, cont, vis = x
tab_out = self.tab(cat, cont)
vis_out = self.vis(vis)
y = torch.cat((tab_out,vis_out), dim=1)
y = self.head(y)
And now we have a model that can train based on our inputs!
Now of course if you wanted to use transfer learning and differential learning rates on that resnet, your splitter should split based on the layer names (self.vis vs everything else)