Thought Experiment: Food Recognition

Hi All!

This hive mind of is amazing :slight_smile: Reading through all the help and knowledge here has really been mindblowing in the year that I stopped working to a) immerse myself in deep learning & b) deal with the effects of chemotherapy treatment.

As part of an effort to figure out how to better track my food intake for my health, I recently came across this app:

The premise is that you snap a picture of your food, and it tells you what it is. It s pretty robust and allows you to create ur own bounding boxes and define foods within a picture that it did not recognize.

My question is, should I want to create my own deep learning model to do something similar, where would I start?

I imagine any packed food is easier to start with, since those images are well defined and a sample data set is easy to put together.

What about for plates of food for example?

I m curious how difficult it was for this company to put a deep learning model together to recognize the millions of different food types and combinations.

1 Like

Here is a recent competition on kaggle as part of CVPR to identify 200 different food dishes.
Basically the fastai multiclassification approach will work just fine, given enough images, enough for me to get 2nd place.
I did read one paper that approached the problem with a ‘wide slice’ resnet, since images of plates of food often have a context based on top to bottom.
To get to calories, you can build a library against those food dishes - pad Thai, chocolate cake, pizza, perhaps prompting the user for portion information.
It wouldn’t be too difficult to build such a model, just a few days for that comp. as always, collecting the annotated data is the grind and the value.