⚠ Danger Zone - Kaggle LB Overfitting

Hello Everyone,

Here I would like to share something that 2 of my kaggle buddies @groverpr, @shik1470 and I were discussing thoroughly while doing Porto Seguro’s Insurance Challenge. We’ve managed to be at top ~10% with LB 0.285.

The methods we employed through this score were highly unorthodox in the way of modeling we had been doing before. Our final best prediction depends on this formula => max(average of 2 models, average 3 models) . Even though this gives us a really good result on LB and even though a lot of people have been applying similar techniques so far in Kaggle for this competition, this might be a way to overfitting to LB !!!

One of the best things to always check is how your CV and LB evolve together, whether indeed they have a linear relationship overtime (even if it is a low order polynomial). CV-LB graphs shown during last course were very good examples on how to keep track of this and how to truly trust your model.

The other side of the medallion is that, in fact your LB score only corresponds to a portion (30% if I am not mistaken) of the true test data, which makes it vulnerable to overfitting and holding a great danger of inability to generalize to the 100% of this test set.

In short, construct a good CV schema and trust it even though it sucks :D. You never know, don’t be discouraged, maybe most of the top runners are overfitting to LB by following other kernel combinations and LB performance! I have been reading about examples of this from past competition discussions, as well as the current one and you would be amazed on how much LB can change after once it’s evaluated on full test set.

Just wanted to share an interesting case and truth :slight_smile:

Note: I will probably choose one model that’s really good at CV as my submission at the end, even though it’s not very good at LB. Gotta keep experimenting…



Yup this is a key point that many Kagglers have discovered, often too late! :smiley:

1 Like

Jeremy recommended that I keep updating my validation versus Kaggle graph as I add new models and get data points. It’s an excellent way to keep track of progress and as a set of tests to know when I’ve screwed up something (this graph will go crazy).


I dont know if you saw, but for this competition they allow you to submit two of your results if I am not mistaken. You can submit one that gives you a high LB score and then one that you think will do better due to your model with a good CV schema.

Just a thought!

Thanks for this.


cool. thanks for info.

Thanks you are right I forgot that point :smiley: