Transfer learning for tabular data - how to replace the head?

fastai
1

I’m trying to build a multi-label classifier with TabularModel and TabularLearner. There are 206 targets I need to predict. However, 402 additional (auxiliary) targets are also provided (which I don’t need to predict).

The idea:

  1. Train a model for the 402 auxiliary targets first.
  2. Use the above as a pre-trained model & apply transfer learning to then train the final model (206 targets).

The input features for both models are exactly the same.

Strategy:

  1. Create new architecture with same body & new head.
  2. Load the ‘body weights’ of pre-trained model

How do I do this? In particular, how do I load just the body weights?

2

Hi @sambit.
I would use the same strategy as cnn_learner:

  1. Load the full pretrained model
  2. Remove the head. In your case, you may want to keep embedding layers only.
  3. Initialize the new layers
  4. Freeze the model (you’ll need to pass a custom cut function)
  5. Train the head
  6. Train all the model
1 Like
3

Searching the forums would have resulted in this post, with all the answers:

TL:DR, transferring would really only be valuable at the embedding level, considering it the equivalent of our pretrained ResNets

1 Like