State Farm, why doesn't the model work with my own test data?


I’ve been working on the State Farm problem from lesson 3. I have watched part of lesson 4, and I’ve built a model using Vgg conv layers and the dense layers - as Jeremy shows in lesson 4.

I’m getting similar results to Jeremy, (~90% acc, 81% val acc). That’s cool, and it was at this point I figured the model should have generalised enough so I can replace the test set with pictures I took of me sitting in my own car in various states of distraction. I thought with a validation accuracy of 81% the model would stand a chance of getting some of the pictures correct.

Sadly it seems to get them all wrong.

At first my predictions were all over the place, then I had a moment of realisation. I am from the UK, and we drive on the left, so all my test set pictures were flipped in the Y axis. After flipping the data, my photos were now in the correct orientation (I also scaled them to 640x480, I know the code scales them to 224x224, but I just wanted to make things the same).

The predictions were now different, but still all wrong. The model seemed to like classifying me as mainly c8 (doing hair and makeup).

I thought perhaps I had something wrong with my model, so I tried running Jeremy’s model, and it yielded the same results (good I guess, at least I didn’t have a bug).

Has anyone tried using their own data? Did it work?

I’m happy to share my test data / results / model code.

I’d also be interested to hear of any ideas as to why my own data doesn’t work.