Conv2D Segmentation Fault (CUDA 8.0, cuDNN 5.11, Keras 2.0.2) in Dogs/Cats Classifier

My environment could be contributing to the trouble I’m running into, so let me be clear: running Ubuntu 16.04, CUDA 8.0, cuDNN 5.11, Anaconda 2 (4.4.0), and relying on Keras 2.0.2 + Theano 0.9.0, and doing all of this atop an intel based machine with a GTX 1080 TI attached. I’ve been able to make modifications to the original utils.py and vgg16.py files to make them work with Keras 2.0.2, but I consistently run into a segmentation fault when the vgg16.py calls the following:

model.add(Conv2D(filters, (3, 3), activation='relu'))

The code I’m using is only lightly adjusted for Keras 2.0.2 (mostly based on feedback others have provided here). I’ve posted all the code I’m using here: https://gist.github.com/pavgup/e6e22eb6b6c3261837881b1cf8276e96

And just in case, I also ran this in gdb to grab some additional information on the fault which I dropped into a comment on that gist.

Help! I’d love to make this work correctly, but I’m unsure what the segmentation fault really means…

For people that might run into segmentation fault problem, it turns out this is a known issue with Theano installing the wrong version of pygpu. Check it out here: https://github.com/Theano/Theano/issues/6141

Here’s the solution:

pavan@learn:~/lesson01$ conda install pygpu=0.6.2
Fetching package metadata …
Solving package specifications: .

Package plan for installation in environment /home/pavan/anaconda2:

The following packages will be DOWNGRADED:

libgpuarray: 0.6.8-0      --> 0.6.2-0
pygpu:       0.6.8-py27_0 --> 0.6.2-py27_0

Proceed ([y]/n)? y

libgpuarray-0. 100% |#############################################################################################################################| Time: 0:00:00 888.38 kB/s
pygpu-0.6.2-py 100% |#############################################################################################################################| Time: 0:00:00 2.62 MB/s

1 Like