Approach to debugging a woeful model

I’m currently working on the statefarm problem as part of the lesson 3 assignment. I don’t want to look at the kernels on Kaggle as they would give too much away :slight_smile:

Started, with the standard VGG model, the last dense layer was removed in favour of a new layer (softmax).

`20224/20224 - 1007s - loss: 11.9468 - acc: 0.2283 - val_loss: 11.2789 - val_acc: 0.2923``

My diagnosis [1]:

  • We are under-training, because the val_acc is higher than the training acc
  • Model probably doesn’t have enough features to learn the much more complicated images of statefarm

Working on my assumption of undertraining, I tried to add multiple additional dense layers to see if the validation would get any better. But this didn’t improve things. I also then tried to set all dense layers to trainable (vs just the last one) and a learning rate of 0.01.

This did not improve things either:

`20224/20224 - loss: 11.6032 - acc: 0.2726 - val_loss: 11.3431 - val_acc: 0.2932

In addition:

  • Had a hypothesis that maybe the image variation between different drivers / setups was too high, so I tried both sample-wise and feature-wise normalisation (via Keras), which made results even worse.
  • I tried various learning rates, incase the changes were not dramatic enough - but very high learning rates seemed to make the problem worse.

My question, then, is:
a) Is broadly my intuition about what is going wrong [1] mostly right?
b) I don’t have enough experience to have an feeling for where I should be looking to debug something like this, where I’ve sort of become stuck in a rut. I’ve (attempted) to made dramatic changes to the network without much success. Should start again with a simple linear classifier to set expectations? Look at image pre-processing? Try to make the model more complex or try to make it more simple?

If your model still includes the two 4096 dense layers, that’s far beyond what is necessary for model complexity for this problem, so that’s not the issue.

And your model is training somewhat, because you are not stuck around 10% accuracy.

Did you run multiple training epochs? This looks a lot like the initial epoch of some of my models, which get much better after several epochs.

Did you separate your dataset correctly?

It seems that the test dataset will have different drivers from the ones that we get to train on. If you don’t create a good validation set, you might be getting very optimistic results. Your validation set should not contain any drivers from your training set.

If you want a few good pointers like this one, I recommend visiting the Statefarm thread: Statefarm kaggle comp

In the upcoming videos, Jeremy will show you a bunch of different ideas you can try with this dataset :slight_smile:

Cheers, and good luck!

Thanks @kensmithzzz @niazangels – I eliminated the model lacking complexity as the problem and started looking into what else may be wrong, turned out there was a problem in the training set after all, which was throwing off the results completely.

Results look much better now!

1 Like