Using image data with tabular data as input

etremblay · January 27, 2019, 2:51pm

I am fairly new to deep learning and fastai, but did 2018 part 1.

Let’s say I have images and for each image I also have additional information like categorical variables, real values etc. I would be interested at predicting a binary output for each image (also using the structured data associated for each of those images).

Then once everything is trained look at the embeddings of my categorical variables and maybe use that for other tasks.

Any easy way to do that in fastai (using images + structured data combined)?

Thank you!

Even · January 27, 2019, 5:27pm

You have to write a custom data block, but it’s very doable. Wayde Gilliam (not sure his handle here) has a great example of doing so for tabular+text:

github.com

ohmeow/medium-finding-data-block-nirvana/blob/master/yelp-00-custom-itemlist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Data Block Nirvana (a journey through the fastai data block API)\n",
    "\n",
    "This notebook illustrates how to create a custom `ItemList` for use in the fastai data block API.  It is heavily annotated to further aid in also understanding how all the different bits in the API interact as well as what is happening at each step and why.\n",
    "\n",
    "Please consult the [fastai docs](https://docs.fast.ai/) for installing required packages and setting up your environment to run the code below.\n",
    "\n",
    "The accompanying Medium article highlighing the data block API mechanics based on my work here can be found [here](https://medium.com/@wgilliam/finding-data-block-nirvana-a-journey-through-the-fastai-data-block-api-c38210537fe4)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Yelp Dataset\n",

This file has been truncated. show original

along with a corresponding blog post:

That should help get you started.

Hope that helps.

etremblay · January 27, 2019, 7:12pm

Thank you, will definitely read that!