Dog breeds test set predictions getting mixed up?

Part 1 (2018)
1

All:

I’ve been working on the dog breed notebook. I’m using “arch = resnet34” and my model achieves an accuracy of 0.88. However, when I submit to Kaggle I get a score of “0.41768.”

Next I tried running the “lesson1-breeds.ipynb” notebook with “arch = resnet34,” because I figured I made a typo somewhere, I figured running FastAI’s notebook would allow me to see where I went wrong. After running Jeremy’s notebook with NO changes other than the “arch” to save time I received a Kaggle score of “0.42490.”

I also tested with “arch = resnet50” and received an even lower Kaggle score even though I was getting an accuracy rating of 86% when training/validating.

I’m very much thinking the file names are getting shuffled/mixed up somewhere when being combined with the class probabilities in the submission file.

Has anyone else seen this issue? If so, how did you solve it?

Or, how do you all combine image files names w/ the probabilities?

Again, I tried this using Jeremy’s dog breed notebook exactly with no changes other than architecture, and I had a the same issue as my own notebook.

Thank you!

2

OK, big facepalm time. According to the rules of the competition:

Evaluation:

Submissions are evaluated on Multi Class Log Loss between the 
predicted probability and the observed target.

And if we examine this Wikipedia article here it says the following:

Logarithmic loss (related to cross-entropy) measures the performance of 
a classification model where the prediction input is a probability value 
between 0 and 1. The goal of our machine learning models is to minimize 
this value.

So! I’ve been working on so many problems with accuracy as the metric I totally missed the different scoring, and my attempts to improve the model that resulted in a lower and lower number were actually a good thing.

Hopefully if anyone else does the same thing this will help.

Thank you.