I am trying to understand Kaggle’s evaluation mechanism. I looked at redux’s evaluation page and kaggle’s main evaluation page. The main driver for me was to understand what value of clipping is the ideal. I was curious why higher accuracy is not better and tried out following variations:
Clipping, log accuracy:
0.001 = 0.15953
0.01 = 0.11605
0.05 = 0.11954
It was not clear to me if retaining prediction accuracy is a good or a bad thing with given data points, as 0.01 turned out to be better than 0.001 and 0.05
Here is the plot of logx
From this graph I infer that lower probabilities are penalized the most which aligns with what kaggle page mentiones about higher probability wrong predictions are penalized the most.
So here are some questions/thoughts:
- What is the loss function keras calculates for each epoch. If it is not log loss by default, can we change it to log loss, so that it is in line with Kaggle?
- It might be worth looking at examples where we were confidently wrong and focusing on fixing those or being less confident.
- Seems like a big clipping will help when our accuracy is low, but eventually it will not help as we have very few negatives?
- I came across Laplace smoothing, should we consider something like that which would make the model less certain?