Thanks 
@sermakarevich based on your suggestions I have been able to resubmit and now have a log loss of 0.24. Thanks for all the help.
3 Likes
Congrats @rikiya and @jamesrequa for beating 0.16 loss level. Can you please share your approaches in general?
2 Likes
I wish I could say I had some “secret formula” but honestly I have literally just been following the general steps that we have learned so far. The only difference is for dog breed I skip the unfreeze section entirely. Because of that I am not using any differential learning rates but I am using cycle_len and cycle_mult. Also I am being a bit selective with my ensemble which simply contains bagged avg of a few models. So make sure if you are also taking the avg of multiple models that you check how each one is contributing to the overall score. If you start by just combining a bunch of them together from the start you may not realize that one of them could actually be hurting your score 
Anyway now there are a few more ppl on the LB that are around 0.10 log loss so I think there is still room for improvement (assuming they are not cheating)
7 Likes
Some of them look too solid for cheating, so yeah, it looks like <0.1 is possible.
I try to use this oof predictions on train to check what and how to blend. But I stacked on trying to make all steps from lessons work. Cant achieve any improvement when use cycle_len and cycle_mult compared to just step by step lr decrease.
There’s a kernel that gets 0.07 IIRC. The trick is to simply use the Imagenet model directly without removing any layers at all - i.e. take advantage of the pre-trained fully connected layer too. In practice this is almost never useful, but in this case because it’s a subset of imagenet in the competition, you can use these trick.
9 Likes
How would we implement this with fastai? If I understand correctly all of the models we are using here have the top layers for the original models switched out.
1 Like
I was thinking imagenet models should give ~ 80% accuracy without fine tuning as this is their score. After fine tuning we get 94-95% accuracy.
I have a couple of ideas. One look at this notebook that I just created.
https://github.com/yanneta/pytorch-tutorials/blob/master/modified_VGG.ipynb
The second approach is to train a linear model on the predictions of the imagenet networks.
3 Likes
That’s what the lb=0.07 kernel does
Thanks @sermakarevich, though, me too, no secret here 
My approach seems almost the same as @jamesrequa’s.
I skipped unfreeze, and trained with all but one training data. I have been tried 3 models so far, and submitted an ensemble of some of them, each of them scored around 0.19-0.20.
Other than that, I just followed what @jeremy taught in the lesson.
Thanks @jeremy, I’ll check out the 0.07 kernel 
3 Likes
Here he gets 0.4 with ResNet50: https://www.kaggle.com/yangpeiwen/keras-inception-xception-0-47
Is it possible to do the same with the fastai lib?
Check out the CIFAR 10 post I just posted - it shows how to use a model without changing it at all. You can use this approach to use a full imagenet model.
4 Likes
No reasons to get 0.47 when you have 0. 16
Sorry if this is a simple question but how would one “train” a model on predictions instead of the images?
My bad =D
First predict on your training images. You will get 1000 probabilities, then use your favorite machine learning package for example (sklearn) with multiclass logistic regression.
6 Likes
Hey… Congrats on the score! Could you give a hint on how to train with all the training data. Currently trained by splitting the data between training and validation- so interested to see how much it improves without splitting the data.
Thanks!
Is it as simple as moving just one image to validation? Will try that in a bit!