Please see the attached screenshots representing two distinct training runs on Redux.
I have validation accuracies of .9840 and .9856 with associated losses of .0436 and .2242. If this were test data, one of those would be pretty great result, the other would a bit above average I believe. Note that these epochs aren’t from the same batch of training.
Both of these are from finetuned VGG16 – with different learning rates/dropout.
I’m having trouble understanding the relationship between these two metrics and why there is so much difference between them. Does it have something to do with the underfitting that appears to be occurring in the second example?
EDITS: making language a bit clearer
Interesting issue… Normally this would suggest that your model is being over-confident in its predictions, and choosing to push its predictions too close to one and zero. Using clipping of the predictions should improve your loss in this case.
This is a general question about what is a good “loss” rate. It looks like I have a decent model for the State Farm competition
accuracy with the training set is overfitted, but the model (using the Jeremy’s advanced model, with only data augmentation and a lot of training with different learning rates, no bells and whistles of
precomputing or pseudolabelling etc.) was still learning and the validation set was getting better, but my loss function never went below 1. According the state farm forum upper 60s accuracy on my validation set should place above 50% on leaderboard, but this forum says that Kaggle looks at the loss. Just checking which is the more “truthful” value I should be checking against while I fiddle with a model or strategies.
The only way to know your leaderboard score is to try submitting - give it a go and see what happens. Since this is a closed competition, you can submit as often as you like. The competition evaluation page shows that its using log likelihood (same thing as cross-entropy) so it’s the first of your numbers that should be close to the kaggle leaderboard result.