Image with metadata classifier

I am building an Image classifier with Columnar metadata.
I plan to build a model which takes input from any pretrained - Convnet and Embeddings for the columnar data and use them together in the last FC layers to finally predict the class.
I have an idea how to do that in keras. Can anyone help/guide me to how to build the same in fastai/pytorch?

Thanks in advance.

Ravi Jain