Combine nlp + cnn models

Hello everyone,

I’m trying to create a deep learning model where I can use both image and text information for a multi-label classification. I have adapted my own code from the following example: https://gist.github.com/jwuphysics/f19162d38e1b4fed0030b96411186c3a

I am now ready to have a Learner object, but lr_find() does not work. As far as I understand it, there seems to be a problem with the input data that is passed to the nn sub modules using the forward() function. I have provided all my code under the following gist:
https://gist.github.com/Codingpreneur/16b9103c8c700e007134c55cedae4aa2

It would be great if the fastai experts could give me a clue as to what I’m doing wrong. Once I’ve got it right, I’d also post a blog post about combining CNN + NLP in fastai.

Thanks in advance!

1 Like

@aklbg have you seen this paper? May be of help. https://arxiv.org/abs/1705.08631