How to save TabularPandas data to a regular Pandas Dataframe

Part 1 (2020)
6

The issue here is fastai’s model has two inputs: the categorical and continuous variables. Let’s rewrite that so it’s a bit more what it’s liking (as it currently things everything is a categorical variable):

(note there is no fastai magic here except for preprocessing):

row = prod.iloc[0]
cat_idxs = [prod.columns.get_loc(nm) for nm in cat_names] # SUPER IMPORTANT
cont_idxs = [prod.columns.get_loc(nm) for nm in cont_names]
cat = row[cat_idxs]
cont = row[cont_idxs]

learn.model.eval()

cat = tensor(cat, dtype=torch.long).unsqueeze(0)
cont = tensor(cont, dtype=torch.double).unsqueeze(0)

learn.model(cat, cont)

Now let’s walk through the why:

We need to get cat_idxs and cont_idxs because fastai’s tabular model has two inputs, a cat tensor and a cont tensor. On top of this, with how the preprocessing works fastai adds in a _na column during FillMissing, so if we check our cat_names you’ll notice it has a _na added to it (potentially many since you’re doing rossmann)

Next after we extract them from a row, we need to convert them to datatypes the fastai model expects: Long for the categorical, and Double for the continues. On top of this since it works by batch, we need to unsqueeze our tensors one dimension (aka just make something like [0,1,2] into [[0,1,2]]

After we do this we can pass it to learn.model.

Now personally I’d take more of this approach (which is how we mimic an inference situation):

dl = learn.dls.test_dl(prod.iloc[:1]) # We do to one so that fastai can properly process it
cat, cont, _ = dl.one_batch()
learn.eval()
learn.model(cat,cont)

And just let fastai handle the preprocessing. (especially since you’re doing learn.model here). But the first way shows how to preprocess and use that data instead.

So basically minimally in production you will need:

  • Your list of post processed cat_names to get their indicies (or be smart and plan ahead, and just store them away)
  • The list of cont_names and their indicies
  • Convert the data to the proper types
  • Unsqueeze the dimension to “batchify” them
  • Pass it to the model
1 Like