Statefarm kaggle comp

Part 1 (2017)
93

Major understatement! It would rank in the top 25%!!!

94

I am finding very different validation accuracy results between 1) training a model using predicted features and 2) adding conv_model and dense models and then training the model. I am pretty sure I am not doing something right here … hopefully a fresh pair of eyes will point me to the right direction: https://anaconda.org/vedaustin/state-farm-aws-ver-3-1-gist/notebook

95

Since you’re using your training batches to precalculate your conv features, you need to add shuffle=False to that constructor :slight_smile:

96

haha … ofcourse . damn it! :slight_smile:

97

Hello everyone,

After getting a validation accuracy (validation set had 2000 images, only 4 misclassifications) of 99.85% on cats and dogs redux, I was pumped to get cracking on StateFarm. I tried training my own model from scratch using @jeremy’s notebook (available here) and then wanted to use the pre-computed VGG weights for convolution layers.

My bare-boned model from scratch gives OK-ish results (it overfits badly) but I’m not too concerned about that at the moment coz I’d rather use the pre-computed weights. Here’s where the problem begins…

When I use the pre-computed VGG weights (convolution), my accuracy stays pretty low and never goes above 14%. After reading this thread, I realized that my directory organization structure may be wrong; so I checked it, it was ok. I also noticed that a lot of people played with dropout, learning rates etc. I did exactly the same. I tried a bunch of Dropouts and a bunch of learning rates, both, high and low; I went as low as 1e-7 but it didn’t seem to help.

I feel a little stumped as I can’t seem to figure out what’s going wrong with my model.
My notebook may be found here.
My notebook for directory organization may be found here.
Would anyone happen to know what I’m doing wrong?

TIA.

98

Hi,
I am having the same problem. I get good results with CNN but an accuracy of 0.1 with pre-computed VGG model that @jeremy uses in statefarm notebook. Let me know if you figure out what the problem is. I will do the same.

1 Like
99

@prateek2686 When you fit your conv features, conv_feat was created from shuffled batches, but trn_labels was not shuffled, so they don’t match. You need to not shuffle the batches used to create conv_feat.

fc_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=4, 
             validation_data=(conv_val_feat, val_labels))
5 Likes
100

@layla.tadjpour see my answer above

101

Are you using a random set of images? 10% accuracy is what I remember always seeing when not creating a proper test set. Make sure that you are using different drivers in the validation set and test set. Hope that helps!

102

@rachel: Thanks so much. That makes sense. Sorry this was a silly mistake. I will fix it and hopefully the results will be better. I shall post updates.

1 Like
103

Thanks @rachel. That was it. It solved my problem.

1 Like
104

thanks for your answer @brianorwhatever . Turrns out the reason it was not working properly was me not setting shuffle=False when creating batches as @rachel said.

105

That was it @rachel . With a little bit of finetuning, I could easily get 98% accuracy.
Thanks again!

106

Hi Jeremy, I am trying to run kaggle statefarm full dataset on the model you used for the sample. It is taking 360 seconds per epoch on p2 instance. Your ipython notebook shows only 114 seconds. What do you think I am doing wrong? I am using the exact same code from ipython notebook (but not the AMI provided with this course, could this be the problem)?

Here is my code (for creating train and val split) and the model I used to train: https://github.com/singlasahil14/kaggle-statefarm

107

Hi Sarno, I am trying to finetune the vgg model without any data augmentation. The best accuracy I could get to is 68.94%, while in jeremy’s ipython notebook, the accuracy was like 79%. What do you think I am doing wrong?
Here is my code for creating training and validation split, and code for training the model (copied mostly from ipython notebook) https://github.com/singlasahil14/kaggle-statefarm

108

Reposting from ‘Lesson 3 discussion’ as this seems a more appropriate place.

Hi,
I was thinking about statefarm problem and had some thoughts and questions around it.

a. Statefarm problem seems to have lesser variety in the type of objects that are present in images i.e. each image would have a human and some objects within a car. Given VGG16 has been architectured as well as trained to detect a much wider set of objects, it feels like an overkill to use all convolution layers of VGG16 as is for statefarm problem. Any comments/insights regarding this ?

b. Statefarm image categorization would be determined majorly by the relative position of the objects w.r.t each other i.e. ‘hands-on-steering-wheel’ implies undistracted driver VS ‘hands-on-something-else’ implies distracted driver. How can the fact that ‘actual objects matter less Vs relative position of objects matter more’ be factored in to architect the model ? Any insights/guidelines on this ?

c. Unrelated to statefarm, what are relative tradeoffs of using a larger convolution kernel(55 instead of 33 for e.g.) ?

Thanks,
Ajith

1 Like
109

I’ve been using a finetuned version of VGG19 for StateFarm as a starting point to build up better models. I ran the test function(same as in vgg16.py) and it seems to be running for quite a while now. I understand that this is 79,000 images and it would take a bit but its’ been on for close to 40 minutes now.

Anyone facing similar issues? Thanks!

110

Hi there,

so did you managed to get the weights?

are they here http://www.platform.ai/models/

thanks!

111

Ajith,

I had thought about the same questions, here are my findings:

TLDR; winning solution uses VGG16 and manually crops certain parts of the image for the CNNs to focus on.

  1. Statefarm dataset is relatively small, and reusing the convolution layers from VGG would ideally help the model avoid overfitting.

Visualizing what VGG+keras is looking for:

excellent visuals (zeiler / fergus style):

  1. Its quite clear that certain parts of the image are more important than others, and the leading results certainly had many creative methods to think about this.

This discussion showed a competitor’s method to display with a very cool heatmap of what the CNNs were focusing on:

1st place solution uses VGG16 and combines 3 images; a cropped image around the head, a cropped image around where the hand may be, and the original image itself.

10th place solution involved finding the area around the drivers body, and cropping the image to around that.


Statefarm was a super cool competition to work on, and I feel like it was a quantum leap from dogs-and-cats competition in terms of deep learning understanding. Hope my findings were helpful, the fact that the winning solution used something more elegant in the spirit of deep learning was certainly very encouraging!

best,

Jerry

7 Likes
112

Hi Jerry,
Thanks for this reply. This is very informative.

Regards,
Ajith