Wonder if there is any interesting work done around Transfer learning and tabular data?
I was thinking why not use same ideas in ULMFit where you would:
train an encoder-decoder on a large tabular dataset with goal to reconstruct the dataset
then on similar dataset (same columns/types) but will few rows and an additional label column, you could use the encoder as backbone to encode the features and add a head to learn the label column
For instance, part 1 could be trained on entire historical data but part 2 only on a recent subset with labels (either classification or regression).
This is just an idea and I’m not sure if it’s good or not.