Poor performance on Dogs vs Cats


(arnaud) #1

Good evening/morning wherever you are :slight_smile:

I’m trying to submit my first predictions on Dogs and Cats competition on Kaggle and for that I followed Lesson 2, essentially the parts that uses Vgg16 and manually fine-tunes it (dropping the last layer, and retraining a new dense final layer).

The only thing I change from the lesson2 notebook is actually using Adam() optimizer.

Fitting the model, I get for the first run, this :

Epoch 1/1
23000/23000 [==============================] - 1319s - loss: 0.2024 - acc: 0.9634 - val_loss: 0.1202 - val_acc: 0.9800

Then that :

Epoch 1/1
23000/23000 [==============================] - 1319s - loss: 0.2315 - acc: 0.9652 - val_loss: 0.1205 - val_acc: 0.9810

First, I find it strange that my loss is actually going up when learning more from the second epoch… But overall losses are reasonable (0.2ish is a good starting point I guess).
And when I submit my predictions (using predict_generator() which I’m afraid makes troubles but I have memory issues so thought it was my only solution), my Kaggle loss is very poor : 1.51952.

Is there something obvious I’m missing here in the pipeline? Hopefully you can give me some hints.

Good evening, thanks for your help,
A


(David Gutman) #2

Two possibilities:

  1. You’re shuffling. Make sure shuffle = False in flow_from_directory for your prediction generator.

You should also make sure the file names match up.

If your generator is named test, then the ids should be in order in test.filenames. Then the corresponding labels would be labels = model.predict_generator(test, test.n)[:,1].

  1. You’re not clipping. labels = np.clip(labels, 0.01,0.99)

(arnaud) #3

Thank you very much for this quick answer.

I bet solution (1) will save me - I didn’t precise shuffle=False. Otherwise, filenames did match and labels were clipped already.

Will update asap :smiley:


(David Gutman) #4

Do a sanity check before submitting - check images to make sure your predictions make sense, most should be right! You can also look at the images you’re most unsure of, e.g. abs(labels - 0.5).argsort()[:10] should be the indices of the ten most uncertain images (assuming labels is a vector of soft max output for dogs).

Good luck!


(Stephen Lizcano) #5

Why does shuffling need to be set to false here?


(David Gutman) #6

You want to make sure your predictions line up with your filelist.


(arnaud) #7

Thanks again for your help. Didn’t realize the competition ended yesterday, but making sure there was shuffle=False of course gave more realistic predictions ! Loss of 0.10613 with that model, fair enough for 10 lines :stuck_out_tongue:


#8

Hi, I’m facing a similar problem, of poor performance on Kaggle for Lesson 1.

Summary of my issue:

  1. I downloaded the “dogs-vs-cats-redux-kernels-edition” data and rebuilt the VGG16 model by finetune and fit commands for the new data/classes. Next, I run predict method on the test images (see code below).
  2. When I upload the test prediction results, the “Public Score” is 0.66868. The Public Leaderboard has a best score of 0.03. This suggests that my results are pretty bad.
  3. When the model was finetuning and fitting, the val_acc was 0.98 and val_loss was 0.1, suggesting that the model was pretty good with validation data.
  4. Why is my ‘test’ “Public Score” so bad? I’ve spot checked about 25 images, of them 24 are are correctly predicted except for 1 image.

As per this thread, shuffle should be false. I use utils.get_data to load test images which has shuffle=false already.
Are there any other ideas?

My code for model fitting and prediction
model = Vgg16()
batches = model.get_batches(path+“train”, batch_size=batch_size)
val_batches = model.get_batches(path+“valid”, batch_size=batch_size*2)
model.finetune(batches)
model.fit(batches, val_batches, nb_epoch=3)

# get test images
test_data = utils.get_data(path+“test”)
# run prediction on test images
[preds, idxs, classes] = model.predict(test_data)

# change the predicted ‘classes’ output to kaggle submission format
classes = [c.replace(‘dogs’, ‘1’) for c in classes]
classes = [c.replace(‘cats’, ‘0’) for c in classes]

# get the filenames from test folder
_, _, filenames = next(os.walk(path+“test/unknown”))
filenames = [f.replace(’.jpg’, ‘’) for f in filenames]

# join the test image filenames and their predictions
output = np.column_stack((filenames, classes))


#9

I further investigated the above issue and figured the reason for my poor kaggle score.

The rules for submission states that “For each image in the test set, you should predict a probability that the image is a dog (1 = dog, 0 = cat).”

In my submission to kaggle, I was not uploading the probably of a dog. But rather just 1 or 0 if the image is a dog or cat respectively. Once I changed the code, to upload probability of a dog, my “Public Score” improved to 0.17 :slight_smile:

Still need to work on it, to improve the Score, but at least getting better.


(Ben) #10

Hi, can you share how do you transform 2 probabilities into one number?
are you using softmax? Thanks!


(Kevin L.) #11

What is the output of your network currently?


(Ben) #12

thanks for your prompt reply, I have solved the problem.
I use the formula 0 * prob(cat) + 1 * prob(dog) to generate one output instead of two separate probabilities.