I’m taking a kaggle competition with tabular, texts and images as input data. I’ve tried to use keras but got bad results so I shifted to fast.ai(using part of the data) and it got much better.
But the problem is that fast.ai don’t have a high-level wrapper for data with multiple kinds of inputs. It’s possible to pre-process the image data using some existing model and then save the features and treat them as numerical data. However I would like to first use ~1 dense layer for each kind of data before concatenating them (as the same kind of data are more relevant to each other and different kinds of data are suitable for different dropout rate etc.), so what should I do?
If I just need to learn to use pytorch to do this it should be much easier, as there are lots of tutorial and codes available online. However I’ve googled fast.ai tutorials and codes, but could only find a few less useful ones, and didn’t find fast.ai code that deals with multiple kinds of inputs.
So if anyone knows about such code using fast.ai to train a nn that deals with multiple kinds of inputs, could you please throw me a link? I think I would be able to write proper code if there’s such a reference.
you will also need a model that can accommodate all this data together. You may start from here: https://docs.fast.ai/tutorial.itemlist.html
and here: https://gist.github.com/EtienneT/c07994bc96e9fad7a30a89cb9f20bc6b
If you like, we could work together, I have the same problem.
I don’t know how to build such a model…
It should take a pair (image, data) and feed this to output a numerical value.
Feed the image trough a convnet (resent18 for instance) and the data through a TabularModel, and then join everything after the flatten layers. Stack some denselayers and output your desired output.
Thanks for your links, Thomas. I’m certainly willing to collaborate but since I’m new to fast.ai I’m not sure how much I can help. I’ll get familar with these functionalities first and I would be glad if you could give any other advice on what we should do.
Thanks, those resources are great. I’ve modified some code from that nlp & tabular model, the concated model was built but no matter what I try(including trying to over-fit) it always output the same value(~median) for every input, what should I do?
I had looked at the concated dataset, the features and y corresponds correctly.
All of the models(all of 3 are tabular(cat, cont, extracted image) currently) could learn and work just fine. I’ve also tried to make the dropout for categorical and continous to be 1(thus making the concatenated model’s architect exactly the same as the image model) but it still always have the same output.
He used create_func to get the dataset, but I would like to use a dataframe to create the dataset, and create_func seems isn’t used anymore? Could you please share some code(about how you create the dataset)?
I successfully trained a model with Image + Tabular. Still trying to integrate text now.
You can also see my model at the bottom of this post and how I loaded the data.
I am currently working on integrating text data. I successfully implemented the data collate function for the variable length text input in the batches. The image, tabular AND text data is correctly passed to the forward function of my Module. But I am running into problems for loading a pre-trained text-encoder I can use in my custom Module… Here is the thread about that if you are interested: Pre-trained text encoder.
I’m interested in doing the same but with text and images. I’m too trying to adapt the nlp + tabular notebook but am stuck with the custom collate and forward functions. Is it possible for you to advice on how you adapted them?
Actually I’m busy doing other things these days and I had only concated image data as output tabular features from pretrained cnn, which I didn’t use collate functions. I’m still trying to use language model’s pre-trained weight for kaggle. Of course I’ll tell you if I have fixed all.