After getting a validation accuracy (validation set had 2000 images, only 4 misclassifications) of 99.85% on cats and dogs redux, I was pumped to get cracking on StateFarm. I tried training my own model from scratch using @jeremy's notebook (available here) and then wanted to use the pre-computed VGG weights for convolution layers.
My bare-boned model from scratch gives OK-ish results (it overfits badly) but I'm not too concerned about that at the moment coz I'd rather use the pre-computed weights. Here's where the problem begins..
When I use the pre-computed VGG weights (convolution), my accuracy stays pretty low and never goes above 14%. After reading this thread, I realized that my directory organization structure may be wrong; so I checked it, it was ok. I also noticed that a lot of people played with dropout, learning rates etc. I did exactly the same. I tried a bunch of Dropouts and a bunch of learning rates, both, high and low; I went as low as 1e-7 but it didn't seem to help.
I feel a little stumped as I can't seem to figure out what's going wrong with my model.
My notebook may be found here.
My notebook for directory organization may be found here.
Would anyone happen to know what I'm doing wrong?