Great Results in Notebook, Not Kaggle

Trying to do the Dogs vs. Cats Redux completely on my own and I am getting 1.9 loss when I submit my results to Kaggle. Looking for some ideas on what I’m doing wrong. Here is my repository with the code:

If I am missing anything here, let me know. All I did was take the VGG16 application built into Keras and popped off the top layer and added a fully connected layer as my output that has a sigmoid activation. The loss seems really good and in line with everyone else, but when I submit, I am just not getting good results. I trimmed the edges so the high was 98% and the low was 2% which did help me because before I have an 8 or 9 loss. So I’m going in the right direction, but I’m just a ways off where I want to wrap this one up.

You are overfitting quite much. When you are close to 100% accuracy, it is often useful to compare error rates: 98.27 test accuracy vs 97.28 validation accuracy doesn’t seem that big difference, but if you compare error rates, it is actually 1.73% test error vs 2.72% validation error. It’s 50% difference.

Likely there is a class of images that are causing some problems in your train/test/valid split. In Cats vs. Dogs, there are a few images that are icons or logos, there are images in which the animal is behind bars (in a cage). These typically caused problems, if the train/test/validation split happened to have a bit of skewed selection, when I tried to get the best possible result in Cats Vs Dogs.

It might help if you visualize and analyze images that are misclassified in your validation set.

Also, try even lower clipping. I’ve typically used validation accuracy as an upper bound, so in your case the clipping range could be [3%,97%].

I finally figured out what my issue was and it is one that I’ve had before. I was using flow_from_directory to pull my test images into batches and I didn’t say shuffle=False. Then I tried to still connect it when I was generating my file using tst_batch.filenames[i]. Since the default for flow_from_directory is shuffle=True, the images were all shuffled up, but the filenames weren’t because the filenames don’t shuffle when the images shuffle. After changing that and adding shuffle=False, I get 0.10950 which is much better. I still have a lot I can do to improve this, but I’m at least in the ballpark at that point.