Statefarm VGG training error

Part 1 (2017)
#1

Hello fellow deep learning enthusiast,

i’m currently working on entering the statefarm competition.
I have made a few experiments on my sample and after that i decided to start with the whole dataset.

Unfortunatly if I try to train any model on the whole training set at some point the ImageDataGenerator throws this error:

7530/19712 [==========>…] - ETA: 327s - loss: 1.1391 - acc: 0.7891

Exception in thread Thread-4:
Traceback (most recent call last):
File “/home/jaidmin/anaconda2/envs/theano/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
self.run()
File “/home/jaidmin/anaconda2/envs/theano/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “/home/jaidmin/anaconda2/envs/theano/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/engine/training.py”, line 409, in data_generator_task
generator_output = next(generator)
File “/home/jaidmin/anaconda2/envs/theano/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/preprocessing/image.py”, line 691, in next
target_size=self.target_size)
File “/home/jaidmin/anaconda2/envs/theano/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/preprocessing/image.py”, line 191, in load_img
img = img.convert(‘RGB’)
File “/home/jaidmin/anaconda2/envs/theano/lib/python2.7/site-packages/PIL/Image.py”, line 841, in convert
self.load()
File “/home/jaidmin/anaconda2/envs/theano/lib/python2.7/site-packages/PIL/ImageFile.py”, line 290, in load
raise IOError(“cannot find loader for this %s file” % self.format)
IOError: cannot find loader for this HDF5 file

ValueError Traceback (most recent call last)
in ()
1 model.fit_generator(batches, batches.nb_sample, nb_epoch=4, validation_data=val_batches,
----> 2 nb_val_samples=val_batches.nb_sample)

/home/jaidmin/anaconda2/envs/theano/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/models.pyc in fit_generator(self, generator, samples_per_epoch, nb_epoch, verbose, callbacks, validation_data, nb_val_samples, class_weight, max_q_size, nb_worker, pickle_safe, initial_epoch, **kwargs)
932 nb_worker=nb_worker,
933 pickle_safe=pickle_safe,
–> 934 initial_epoch=initial_epoch)
935
936 def evaluate_generator(self, generator, val_samples,

/home/jaidmin/anaconda2/envs/theano/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/engine/training.pyc in fit_generator(self, generator, samples_per_epoch, nb_epoch, verbose, callbacks, validation_data, nb_val_samples, class_weight, max_q_size, nb_worker, pickle_safe, initial_epoch)
1480 '(x, y, sample_weight) '
1481 'or (x, y). Found: ’ +
-> 1482 str(generator_output))
1483 if len(generator_output) == 2:
1484 x, y = generator_output

ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

I thought there was an error in my code so I tried to run the Statefarm notebook from Jeremy, but there the same error occurred (already on the get_data() function )

I did some googling (is that a word? I’m not a native speaker) and found that hdf or h5 is a file format to store very large files, and that PIL doesn’t support operations on HDF files. I searched my training data folders for andy hdf files but found none. Next I will redownload the data and try again.

If anybody has an idea what might be going on, please let me know

Johannes Laute

#2

Do you have h5py installed? You can check by entering this in the terminal:

python
import h5py

If you get an import error you can install h5py by typing this in the terminal:

conda install h5py

or

pip install --user h5py

It’s strange that you were able to work with a sample but not the whole dataset.

It would help if you posted some of your code, especially the code you used to define batches and val_batches. Here’s a guide about asking for help.

The best would be if you uploaded your notebook to github and posted a link, that way we could see everything.

#3

Thanks for your quick answer.
H5py was installed. i fixed the problem by deleting the state folder in /data and redownloading with the kaggle-cli.
Now everything seems to work, still wondering why pil would be trying to open a hdf file…

Thanks also for the wiki entry on what things to provide when asking questions in the forums.

johannes