Why do we use pretrained models? We want to utilise the patterns that the models has learnt from a lot of data. Why majorly image and text? The patterns that the model learns are common and can be easily transferred. A cat is always going to look like a cat. However, the semantic meaning of the columns and the relations between them (linear/non-linear) are patterns that can hardly be reused outside the context of the problem. If the context and the columns relations are the same (same data, but a different task) then we can use pretrained tabular data to fine tune.