Statefarm kaggle comp

Hi Chris,
I replicated this process using a fine-tuned Vgg16 model and my prediction dimensions were (79726, 10) . If your final layer has an output of 10 then that is what you should be getting back. Try running it again, if you get the same problem copy and paste your notebook into https://gist.github.com/ and save it with an .ipynb extension so we can see exactly what you did.

Thank you! I think I’ve figured out what I did:

  1. I called load_weights on a VGG model that had been fine tuned with a Dense(2) final layer
  2. I fine tuned again with a Dense(10) layer
  3. Then called test() without a call to fit()
1 Like

That will do it! Good catch!

Hi Garima,

I’m just cracking the book on the State Farm comp. How did you create the directory structure, if you don’t mind me asking?

I wrote a couple of little Python functions to make the directory structure. Like this:

def mkdirs(path):
    [os.mkdir('{}/c{}'.format(path, i)) for i in range(10)]
               
def mv_valid(path):
    for i in range(10):
        d = 'c{}'.format(i)
        g = glob('{}/{}/*.jpg'.format(path, d))
        shuf = np.random.permutation(g)
        for i in range(200):
            os.rename(shuf[i], shuf[i].replace('train', 'valid'))
1 Like

That validation set won’t work correctly - it’ll give very (very!) optimistic results. The competition data page says: The train and test data are split on the drivers, such that one driver can only appear on either train or test set. That means you need to do the same thing for your validation set creation!

6 Likes

@jeremy good to know, thank you!

@mattobrien415 I haven’t tried this yet, but I believe this would create the directory structure for you.

My model seems to get very poor results no matter what I do. The accuracy and validation accuracy seems to float around 10% and never increases. This is even more obvious when I look at the confusion matrix and it’s guessed that every image was of a single category.

I have moved 3 different drivers into my validation set thinking that was what Jeremy has been hinting at. I have set the layers from the first dense layer onwards to trainable since this data set isn’t predicting categories from imagenet.

I assume there is something fundamentally flawed with my model that it is only predicting 1 category. I have been considering using the techniques (data augmentation, batch normalization, dropout) but I am under the understanding that these techniques only help once you have a relatively good model. Is there something simple I’m missing?

1 Like

@brianorwhatever the challenge of getting a model to train at all on this dataset is part of what makes it such a good learning experience. So the process you’re going thru will hopefully be very rewarding :slight_smile:

It sounds to me like you’re on the right track. I’d suggest trying a variety of learning rates. Perhaps your learning rate is too high. I would suggest using batch normalization if you can - it makes initial learning much easier.

1 Like

Sounds good - I will keep plugging away. Thanks!

Brian I’m with you. My model is perfect… at classifying everything into the first class! I’m using a sample training size of 2000 and a validation set of 500 (drivers not in training set). Learning rate of .01. I popped the final VGG layer and added my new dense. I even tried going back a few layers but still everything in the first cat. Even though this is a small training set I figured I could at least get to 11% … then I wouldn’t feel like this was a bug on my part.

as suggested by Jeremy, I’ve found a lot more success with lower learning rates

2 Likes

Am I getting any warmer? This creates a validation set of 827 images.

import pandas as pd
def mv_valid():
    # read the CSV file into a pandas DataFrame
    dil = pd.read_csv(path + 'driver_imgs_list.csv')
    # group the frame by the subject in the image
    grouped = dil.groupby('subject')
    # pick a subject at random
    subject = groups.keys()[np.random.randint(0, high=len(groups)-1)]
    # get the group associated with the subject
    group = grouped.get_group(subject)
    # loop over the group and move the images into the validation directory
    for (subject, cls, img) in group.values:
        source = '{}train/{}/{}'.format(path, cls, img)
        target = source.replace('train', 'valid')
        print('mv {} {}'.format(source, target))
        os.rename(source, target)
3 Likes

Looking good :slight_smile: I’d suggest a validation set with more drivers than that however - I’m using 3, IIRC

1 Like

I have also been using 3 and was wondering if adding more would negatively or positively impact my results. Guess I’ll just have to give it a shot!

The downside is your training set will be smaller. The upside is your validation accuracy will be more stable. The best approach would be to create a bunch of models, each time holding out one driver, and then average the validation across all of them (you could also average the predictions across all of them, like the ensembling we did in MNIST last week!) But I wouldn’t bother with that until you had done all the experimenting you wanted to do, since it adds a lot of time to each experiment.

1 Like

A valid submission at least!

1 Like

That’s a great result! What was your technique?

I downloaded all the notebooks together with a transcript of the video lectures, then trained a multilayer LSTM. I took the resulting notebook and kept pressing shift-return until I hit an error. I fixed two such errors and I was done!

I am of course joking, the truth is I hacked together something that closely resembles your lesson 3 final model.

2 Likes