I have been working on using the Keras VGG16 model and I am running into some weird issues. I am having decent results from my acc and val_acc, but then when I go to actually predict my test, I am getting results that are no better than random. I’ve tried this with the dogs and cats competition and the Deep Learning Challenge # 1 and I am getting similar results. When I do a prediction on the validation set, it looks really good, but then I try to predict something from test and my model acts like it’s never seen anything like it before. Has anyone seen this and am I missing some easy gotcha?
Here is what I am using:
%matplotlib inline
import keras
from keras import backend as K
from keras import applications
from keras.models import Sequential, Model
from keras.applications import VGG16
from keras.layers.core import Dense, Flatten, Dropout
from keras.preprocessing import image
import pandas as pd
import numpy as np
import matplotlib
from matplotlib import pyplot as plt
from glob import glob
import os
import sys
from keras.applications.vgg16 import preprocess_input
batch_size = 64
sys.tracebacklimit = 2
idg = image.ImageDataGenerator()#preprocessing_function=preprocess_input)
trn_batches = idg.flow_from_directory("label_train_img", target_size=(224,224))
val_batches = idg.flow_from_directory("label_valid_img", target_size=(224,224))
trn_batches = idg.flow_from_directory("label_train_img", target_size=(224,224),batch_size=trn_batches.n)
val_batches = idg.flow_from_directory("label_valid_img", target_size=(224,224),batch_size=val_batches.n)
trn_batch, trn_labels = trn_batches.next()
val_batch, val_labels = val_batches.next()
vgg16 = VGG16()
vgg16.layers.pop()
for layer in vgg16.layers: layer.trainable=False
m = Dropout(0.5)(vgg16.layers[-1].output)
m = Dense(25, activation='softmax')(m)
vgg16 = Model(vgg16.input, m)
vgg16.compile(optimizer="adam", loss="categorical_crossentropy", metrics=['accuracy'])
for i in range(trn_batch.shape[0]):
trn_batch[i] = preprocess_input(trn_batch[i])
for i in range(val_batch.shape[0]):
val_batch[i] = preprocess_input(val_batch[i])
vgg16.fit(trn_batch, trn_labels, epochs=10,validation_data=(val_batch, val_labels))
test_batches = idg.flow_from_directory("label_test_img", target_size=(224,224))
test_batches = idg.flow_from_directory("label_test_img", target_size=(224,224),batch_size=test_batches.n)
test_batch, test_labels = test_batches.next()
test_filenames=test_batches.filenames
legend = trn_batches.class_indices
legend = {y:x for x,y in legend.items()}
submission = pd.DataFrame(columns=['image_id','label'])
for i in range(0,test_batches.n):
im = test_batch[i]
im = preprocess_input(im)
im_name = test_filenames[i].split("/")[1].split(".")[0]
prediction = legend[np.argmax(vgg16.predict(np.array(im,ndmin=4)))]
submission = submission.append({'image_id':im_name, 'label':prediction},ignore_index=True)
submission.to_csv("Submission_ImageClass.csv")
I am just at a loss because everthing I’m doing seems to be giving me really good results, but then I try with the test dataset and I am getting random results.
Accuracy Readout:
Train on 2591 samples, validate on 620 samples
Epoch 1/10
2591/2591 [==============================] - 11s - loss: 2.8741 - acc: 0.3350 - val_loss: 1.9315 - val_acc: 0.4661
Epoch 2/10
2591/2591 [==============================] - 11s - loss: 1.4932 - acc: 0.5901 - val_loss: 1.5955 - val_acc: 0.5468
Epoch 3/10
2591/2591 [==============================] - 11s - loss: 1.1251 - acc: 0.6708 - val_loss: 1.4350 - val_acc: 0.5935
Epoch 4/10
2591/2591 [==============================] - 11s - loss: 0.8492 - acc: 0.7240 - val_loss: 1.5399 - val_acc: 0.5806
Epoch 5/10
2591/2591 [==============================] - 11s - loss: 0.6500 - acc: 0.7862 - val_loss: 1.4338 - val_acc: 0.6097
Epoch 6/10
2591/2591 [==============================] - 11s - loss: 0.5653 - acc: 0.8163 - val_loss: 1.4801 - val_acc: 0.6016
Epoch 7/10
2591/2591 [==============================] - 11s - loss: 0.4419 - acc: 0.8530 - val_loss: 1.5429 - val_acc: 0.6081
Epoch 8/10
2591/2591 [==============================] - 11s - loss: 0.4050 - acc: 0.8680 - val_loss: 1.4104 - val_acc: 0.6371
Epoch 9/10
2591/2591 [==============================] - 11s - loss: 0.3518 - acc: 0.8842 - val_loss: 1.5256 - val_acc: 0.6226
Epoch 10/10
2591/2591 [==============================] - 11s - loss: 0.3148 - acc: 0.8927 - val_loss: 1.4675 - val_acc: 0.6452
Results on Test Data: