My understanding, which is discussed in this forum post, is that the reason there aren’t pretrained models for tabular data is because the inputs (columns) vary depending on the use case. However, that post is 3 years old and there may be advances in that area.
Perhaps one angle you could explore is if there is any similarity between the embeddings of different tabular models, and if that can lead to transferability between pretrained and new models. For example in chapter 9 they show how the embeddings from a tabular model trained on sales data also learned something about the geographic location of the stores:
