Lesson1: Poor Kaggle ranking

I’ve read through all of the other posts here regarding poor Kaggle scores but have not been able to determine why my submission is scored so poorly. My “best” submission to date is: 0.19143 (rank 784).

I have hand-verified every step including looking through the first 75 images in the test set to ensure that the results recorded in my csv are correct (only a few were predicted incorrectly). I’ve watched Jeremy’s solution (the beginning of the Lesson 2 video) several times. I believe I am following his steps exactly.

Here are the steps I’m taking. Do you see any obvious mistakes? Thank you in advance.

batches     = vgg.get_batches(path+'train', batch_size=batch_size)#, shuffle=False)
val_batches = vgg.get_batches(path+'validation', batch_size=batch_size*2)#, shuffle=False)
vgg.finetune(batches)
vgg.fit(batches, val_batches, nb_epoch=1)

Found 23000 images belonging to 2 classes.
Found 2000 images belonging to 2 classes.
Epoch 1/1
23000/23000 [==============================] - 197s - loss: 0.1209 - acc: 0.9690 - val_loss: 0.0651 - val_acc: 0.9845

batches, predictions = vgg.test(path+'test', batch_size = batch_size * 2)
predictions[:5]
array([[ 1.    ,  0.    ],
       [ 0.9874,  0.0126],
       [ 1.    ,  0.    ],
       [ 0.    ,  1.    ],
       [ 1.    ,  0.    ]], dtype=float32)

filenames = batches.filenames # full path to all test images
filenames[:5]
['unknown/84.jpg',
 'unknown/12453.jpg',
 'unknown/3841.jpg',
 'unknown/8713.jpg',
 'unknown/478.jpg']
ids = # convert filenames into ids
ids[:5]
[84, 12453, 3841, 8713, 478]

isdog = predictions[:, 1]
isdog[:5]
array([ 0.    ,  0.0126,  0.    ,  1.    ,  0.    ], dtype=float32)

subm = np.stack([ids, isdog], axis=1)
subm[:5]
array([[    84.    ,      0.    ],
       [ 12453.    ,      0.0126],
       [  3841.    ,      0.    ],
       [  8713.    ,      1.    ],
       [   478.    ,      0.    ]])

np.savetxt('submission.csv', subm, fmt='%d,%.5f', header='id,label', comments='')

@RobertJ, did you clip your predictions as Jeremy describes here in Lesson 2? Kaggle uses log-loss for the evaluation metric, and as explained in the video, extreme probability values (near 0 or 1) don’t play well with log-loss. Try clipping your prediction values and see if that helps your Kaggle result.

Out of curiosity, what was your training/validation accuracy? I’m having a different problem as I can’t seem to get training/validation accuracy past 92%.

1 Like

@barely_qualified I haven’t performed any clipping (I guess I haven’t watched far enough into lesson 2).

If you look at the top of this post you can see all of the output from my vgg.fit() function:

loss: 0.1209 - acc: 0.9690 - val_loss: 0.0651 - val_acc: 0.9845

I can share my code with you if that would be helpful.

Looks like you are using probabilities for prediction results (0, 1) and submitted those to Kaggle. For this particular competition Kaggle needs log loss.

Steps to follow:-

  1. Calculate log loss for the prediction and upload those to Kaggle.
  2. Your model training output is only one epoch. Try using more epochs. 20 may be ?

Clipping the predictions to 0.5 - 0.95 (inclusive) reduced my score to 0.09695 (rank 394).

isdog = np.clip(predictions[:, 1], 0.05, 0.95)

1 Like

You should not shuffle your validation set.

@torkku Why? Jeremy shuffles his validation set.

My current Kaggle score is: 0.07197

I just tested not shuffling the validation set and as I expected it made absolutely no difference to the Kaggle score.

@RobertJ Yep. Sorry, thought you were at a later phase where we are using pre-calculated weights, then you cannot shuffle your data. I’ve been gone for a while and this exact problem has been a problem for many people earlier.

Now I read your post more carefully. Your validation accuracy actually is very good: val_acc: 0.9845

So it’s probably something to do with your submission to Kaggle. Forgetting clipping should not drop your score to 0.07. :slight_smile:

Hm.

I think your submission columns are probably reversed.

Probably:

isdog = predictions[:, 1]

Should actually be:

iscat = predictions[:, 1]

That would make your Kaggle score 1-0.07197 which is pretty close to 0.93.

Can you check this out?

EDIT: Changed the term “data-augmentation” to “pre-calculated weights”.

1 Like

@torkku I’ll definitely do that - thank you very much!

But just to be clear: the goal is to get the lowest Kaggle score possible, correct? 0.93 would then be worse than my current score of 0.07.

@RobertJ Had to check my Cats and Dogs Redux score (from 6 months ago).

My Kaggle score of 0.08050 got is now at position 258 (username akir).

So your Kaggle score of 0.07197 should be pretty ok? Am I missing something now? :slight_smile:

This was a wrong suggestion from me. Only disable shuffling if you are using pre-calculated weights.

2 Likes

@torkku Thanks again for your help. I just didn’t know what an acceptable score was - so I didn’t know when to stop tuning. :wink: I’ll now move on to Lesson 2.

No probs. Just move on for more fun! State farm distracted detection is also really fun and a bit more interesting. :slight_smile:

You can always come back to improve your Dogs and Cats Redux results later. :slight_smile: I might do that some day.

1 Like

I submitted with clipped values. I got the score of .10230. When I try to do final submission. It is not accepting. I guess our submission only before the end of the competition is considered.

@nikhil.ikhar You should be able to do a late submission. I just did one today. The UI is a bit confusing. Select your file to upload, enter a description and click Make Submission. This will result in your submission being evaluated - took about 2 seconds for me. Scroll to the top and click on My Submissions to view your score.

You can use this score to scan the leaderboard and figure out the rank you might have received had you actually been participating in the competition.

Thanks. I did the same.