Combine nlp + cnn models

Hello everyone,

I’m trying to create a deep learning model where I can use both image and text information for a multi-label classification. I have adapted my own code from the following example:

I am now ready to have a Learner object, but lr_find() does not work. As far as I understand it, there seems to be a problem with the input data that is passed to the nn sub modules using the forward() function. I have provided all my code under the following gist:

It would be great if the fastai experts could give me a clue as to what I’m doing wrong. Once I’ve got it right, I’d also post a blog post about combining CNN + NLP in fastai.

Thanks in advance!

1 Like

@aklbg have you seen this paper? May be of help.