I have been wondering about that for a while (in the context of NLP and tabular data). I found two resources recently that show how to do that in fastai.
-
This fastai post points you to this great Jupyter notebook example - ConcatModel is what would be relevant
-
In the Kaggle Quickdraw competition Radek made some great example notebooks. One of them shows how to combine 4 model architectures together (look at the MixedInputModel).