Systematically lower accuracy e.g. on State Farm

ganesha123 · August 22, 2017, 4:04am

Hi everyone,

I’m really enjoying the course, but I’ve been struggling with my (validation) accuracies being systematically lower than they were for Jeremy. Between 20% (for preliminary sample testing) and 10% (for my final model) lower.
I started shift-entering through the notebooks (forbidden, I know…) to try to identify what the problem was. I still got much lower accuracies despite using exactly the same code. (Included is the script from the State Farm sample testing that I saved to illustrate this).

This issue has applied to both Cats and Dogs and State Farm.
Regarding State Farm, I have checked that the validation and training sets contain different drivers.
Generally, I’d love to know if anyone has any thoughts / had a similar experience.
Is there some change from Keras 1 to Keras 2 (which I’m using) that would make my results different?

Many thanks for any help!

github.com

irequireversioncontrol/my_state_farm/blob/master/state_farm_sample_tests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enter State Farm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from theano.sandbox import cuda\n",
    "cuda.use('gpu1')"
   ]

This file has been truncated. show original

danaludwig · August 19, 2018, 7:00am

Hi @ganesha123 anesha123, I’m doing the 2018 course with the fastai/pytorch models, and I’m getting about 10% lower accuracy / loss than Jeremy’s video on validation set too. I’m running my own DL box with a 1080 GPU (not 1080Ti). nvidia-smi shows driver version 384.130.

The good part is that all the code runs, and better code runs better, but it’s always a little worse than Jeremy’s results.