Thanks Jeremy. Will do it using kaggle-cli. I see a slew of submissions from fastai students already. Amazing stuff. 
I somehow have four more images than the competition has so I have some work to figure that out. I think I should have a first submission tomorrow night as long as I am able to figure out why that is happening.
I figured out my issue and now have a completed submission.
Iām really happy with that as a starting point. I used resnet34 and havenāt really done anything to help the model yet. Thank you to everybody who had helped me get to this point. I can now confirm that it is possible to use from_paths even though from what Iāve heard from everybody, from_csv is the better way to do it.
6 Likes
Models log loss on a whole training set (acquired through 5 fold cv, between folds score might very from less than 0.17 to 0.26):
inception_4_300 0.228
inception_4_350 0.211
inception_4_400 0.204
inception_4_450 0.223
inceptionresnet_2_300 0.239
inceptionresnet_2_350 0.217
inceptionresnet_2_400 0.215
inceptionresnet_2_450 0.222
Simple averaging all models except forinceptionresnet_2_300, inception_4_450 gives 0.181 on a training set and 0.172 on a leaderboard.
3 Likes
@sermakarevich Great, thanks
I am afraid of asking this kind of basic question, but what did you average (weights, probs, orā¦)?, and how did you average them in practice?
2 Likes
Yep, thats really scary question 
In practice it looks like this:
- I have 5 predictions for a test set because I do 5-fold CV
- averaging CV predictions for test set for each model so at the end I have a single test set prediction for each training config (model / image size)
- through CV i get train set predictions as well - this allows me to check how should I average predictions from different models (you might end up with just a mean, or median, or weights, or blending with additional model) and better understand all models accuracy as whole train set is better than validation set
- averaging test predictions from different models
5 Likes
Amazing! Thanks @sermakarevich!
Iām in 11th place, and Iāll try your approach.
How do you do a 5-fold CV with the fastai lib?
fastai students gonna rock it 
It turned out to be pretty easy with sklearn StratifiedKFold indexes with ImageClassifierData.from_csv method. You just need to define val_idxs parameter.
2 Likes
Thanks @sermakarevich ! Just qq.
You mean you did CV for test set rather than training/validation?
Thatās pretty cool 
Thanks!
BTW, are there any standard methods of ensembling multiple models or architectures, I mean re āweights or probabilitiesā or āmean or medianā?
- splite train set into 5 parts with
sklearn StratifiedKFold - 4 parts are used as train-1 set and 1 is used as valid-1 set
- this is done by StratifiedKFold.split method which returns indexes for train-1 set (80% of original train) and indexes for valid-1 set (20% of original train)
- tune a model
- do TTA predictions for test and valid-1 (20% of train set)
- iterate through this 5 times
@jamesrequa knows this better than me. I used two different ways:
- just avg(sum(all predictions))
- extracted features from convolutional layers from different models are stacked together and only than I feed them into FC layer.
6 Likes
Iām asking because for the dog breed competition when I tried to ensemble three models by just simply averaging their probabilities, each log loss was around 0.21, the outcome jumped up high to around 13 
. Thats why Thanks!
Check rows and columns ordering. 13 is definitely an error.
2 Likes
Thanks a lot, now itās super clear
And, sure, I was wrong somewhere in the averaging process, Iāll try again 
@sermakarevich what are these numbers (300, 350, 400, 450)? Size of the images?
2 Likes
looks like half of top 20 is fastai students so far 
5 Likes
why not get predictions for test by training on the whole dataset instead of CV ?
No reasons why not to. You only need to know how to optimise a model without a validation set. With CV one can achieve
- better understanding of accuracy
- get predictions for train set
- get mini ensemble for test set.
This mini ensemble gives 0.02 log loss improvement test vs train (which is 10%).
2 Likes
Iām assuming you mean a new model for each iteration, correct?
ā¦ and thanks for the detailed and nice writeup on using K-Fold CV!
How do I submit my results to Kaggle?
I ran some tests and built a decent classifier for my first submission, but itās not clear to me how to get those predictions into a csv file for submitting.
Look at the last few lines of this Kernel for an example of that:
https://www.kaggle.com/orangutan/keras-vgg19-starter
One step they donāt do though is:
sub.to_csv(path+"filename.csv", index=False)
6 Likes