Thought Experiment: Food Recognition

Deep Learning
1

Hi All!

This hive mind of fast.ai is amazing :slight_smile: Reading through all the help and knowledge here has really been mindblowing in the year that I stopped working to a) immerse myself in deep learning & b) deal with the effects of chemotherapy treatment.

As part of an effort to figure out how to better track my food intake for my health, I recently came across this app: http://www.caloriemama.ai/#CalorieMama

The premise is that you snap a picture of your food, and it tells you what it is. It s pretty robust and allows you to create ur own bounding boxes and define foods within a picture that it did not recognize.

My question is, should I want to create my own deep learning model to do something similar, where would I start?

I imagine any packed food is easier to start with, since those images are well defined and a sample data set is easy to put together.

What about for plates of food for example?

I m curious how difficult it was for this company to put a deep learning model together to recognize the millions of different food types and combinations.

1 Like
2

Here is a recent competition on kaggle as part of CVPR to identify 200 different food dishes. https://www.kaggle.com/c/ifood2018/
Basically the fastai multiclassification approach will work just fine, given enough images, enough for me to get 2nd place.
I did read one paper that approached the problem with a ‘wide slice’ resnet, since images of plates of food often have a context based on top to bottom.
To get to calories, you can build a library against those food dishes - pad Thai, chocolate cake, pizza, perhaps prompting the user for portion information.
It wouldn’t be too difficult to build such a model, just a few days for that comp. as always, collecting the annotated data is the grind and the value.