Let’s say I’m starting from images so using Vision, then extract their EXIF so now I have Images and many fields with various values thus making it Tabular(?), and on some of these fields I want to use NLP thus making it also Textual.
In some cases, the input data itself will have binary files as well as a spreadsheet with extra information.
Some people already referred to similar issues in fastai v1:
So, what is the way in fastai v2 to load the data from different kinds and combine it?
DataFrame rows? Tuples? Designated objects? Multiple Datasets?
Can you please provide an example/tutorial? (possibly referring to the challenges mentioned in the above forum posts).
Hi, loading different kinds of data from a variety of sources is a tedious work but it is not too complicated. Doing that efficiently, it is another story.
By the way, eventually, you will end up with tensors and get to the crucial part: how and when do you have to combine them? It is not a easy task. Pay attention to the size of your input, and how the different components of your model are connected one another. The simplest naive approach is to “fuse”(e.g. concat) everything at the end just before your model’s “head”. Sometimes works, give it a try.
You can reference an article I wrote on creating a MixedItemList for text + tabular here:
I’m working on a follow-up article that demonstrates how one may hack the existing fastai learners/models so as to consume such a dataset, with hopes of doing this all over again in December with the v2 bits. Btw, any advice on how to do this correctly are much appreciated
I’ve done some work a few months ago but got consumed with other things in between then and now :). I don’t have much time right now to work on this personally but if you’d like to collaborate on this, I’m down! I’ll let you know when I get a bit more free time but feel free to keep me updated on your work in the meantime.
