Validation loss vs Kaggle Leaderboard loss

Hello! First of all, I want to thank you for the excellent course and forums.

I’m working on applying the techniques learned in the cats vs dogs competition (currently at lesson 3).
So far as the validation set loss decreased, so did the Kaggle Leaderboard (LB) loss (and they were pretty close).
However, recently the validation loss went down to 0.04 but the best LB loss I get is no less than 0.063.
Why is there this discrepancy? Have I overfit my validation set (2k samples)? What can be done to fix this?
Thanks!
Slav

Yeah, could be :slight_smile:

There is also the question of clipping, meaning as you continue to improve your model (and your validation set results) it is likely to be the case that your model is more and more confident in its predictions (values closer to either 0 or 1, or exactly 1 and 0).

What the logloss does, is that the penalty is non-linear. You will get disproportionally penalized for outputting 0.990 vs 0.985 when the true class is 0. Thus if you are not doing clipping (not sure if that is explained in lesson 3 as I am only 3/4th way through it), getting better validation results might be offset by this factor on the LB to some extent.

BTW. what is your architecture? Were you able to get this good of a result only via retraining the FC layers + softmax layer without clipping?

1 Like

Thanks @radek, spot on with the clipping.
I had clipping in the range (0.02, 0.98). When changed to (0.005, 0.995), score improved.

The architecture I’m using (with medium amounts of data augmentation):
Conv features
Dense(4096)
Dropout(0.5)
Dense(4096)
Dropout(0.5)
Dense(4096)
Dropout(0.5)
Dense(2)

No batch norm layers - adding them increases loss (not sure why yet).

1 Like