I can confirn [this approach]( https://github.com/anhquan0412/fastai-tabular-text-demo/blob/master/mercari-tabular-text-version-2-all.ipynb ) works better.
It backrops over a entire tabular model before concatenating the output of the text and tabular model.
Where does this implementation come from? Did you do this as well?