For anybody that wished to run cuDNN 6.0.20 on CUDA 8.0.,61 the instructions here are as best I can recall. The reason to do this are the benefits of running the CUDA PASCAL code in a pascal equipped GPU's that provides large improvements in performance over cuDNN 5.1, such as using less GPU memory, cooler GPU running temperatures and lower GPU utilisation all while giving massive gains in running convolutional neural nets. This instruction was done with the help of Leon Letto on the forum who found away around the issue of running cuDNN on CUDA 8.0.61-1.
I did this because of the grief and time this has taken and there's nothing on the web in complete form yet at this time that can lead to a successful install of cuDNN 6.0.20 on CUDA 8.0 with theano 0.9.0. Theano just does not like this arrangement of CUDA 8.0 cuDNN 6.0 at all so its been trial and error. I can't find anywhere on the web, in one single instruction to do this, so both Leon and I have been back and forth over this, so this is to capture what we know to get it going from complete fresh install to working lesson 1 using CUDA 8.61.1 and cuDNN 6.0.20. with theano 0.9.0 at this time for nvidiia pascal equipped GPU's. Good Luck!
This assumes you are doing the course this forum is about so it references material from it. Also spell checker in this dialogue may change things I did not capture so beware.
Get two USB drives.
You will need to rescue some folders to save time on re-running Vgg later, these are the 'model' folder that hold the Vgg weights and 'Data'. 'Models' is in a hidden folder called ~/.keras inside that is 'model'. Dump model on a USB someplace. Rescue the DATA folder holding cats-dogs under NBS where your jupiter projects are and dump that on the data USB.
The two USB's are needed. One for data and one for ubuntu LTS 16.04, as ubuntu may give out warning if it finds other folders present. Yes we are nervous! This install has taken dozens of trial runs with no success to get to this point, everything needs to be straight down the middle no unusual deviations will help replicate success. If there's a deviation and it doesn't work there will be always doubt about a step taken that was not quite true to the instructions below. We note the order of installation at this time from what we know is very order specific especially libgpuarray and conda pygpu which are the key files that make this work we think. Yes we do install theano twice its best to 'play this game' as its what is working as of now.
Obtain a copy of ubuntu LTS 16.04 load it on the ubuntu USB and format with a windows PC using 'rufus'. https://rufus.akeo.ie
Spin up ubuntu and do nothing as yet. We will, I suggest use ssh on another machine it helps when things go wrong we can still possibly have access to the system via the command line or to run init 3 states etc. plus its easy to cut n paste these instructions on another machine straight into the ssh command line.
Instructions: (as best I can recall) There maybe parts we don't need in this installation but just follow it step by step to have a base reference. It is not script, it is the modified AWS script from the course but its better to do this by hand until someone refines it.
This instruction is designed to work with ubuntu 16.04 LTS
INSTALL SSH OPENSERVER
sudo apt-get install openssh-server
sudo service ssh status
ctrl-c to get out
locate your ip address and write it down to use on your other machine.
log into your local machine running the lab and GPU using ssh your-machine-name@your-ipaddsress
ensure system is updated and has basic build tools
sudo apt-get update
sudo apt-get --assume-yes upgrade
sudo apt-get --assume-yes install tmux build-essential gcc g++ make binutils
sudo apt-get --assume-yes install software-properties-common
download and install GPU drivers
wget "http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb" -O "cuda-repo-ubuntu1604_8.0.61-1_amd64.deb"
sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
sudo apt-get update
sudo apt-get -y install cuda
sudo modprobe nvidia
Download the cuDNN 6.0 from nvidia developers website https://developer.nvidia.com/, you will need an account, sign up. Obtain the files for Linux cuDNN 6.0 for CUDA 8.0 and place that in your ~/Downloads folder.
INSTALL cuDNN 6.0
tar -zxf cudnn-8.0-linux-x64-v6.0.tgz
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/* /usr/local/cuda/include/
install Anaconda for current user
wget "https://repo.continuum.io/archive/Anaconda2-4.2.0-Linux-x86_64.sh" -O "Anaconda2-4.2.0-Linux-x86_64.sh"
bash "Anaconda2-4.2.0-Linux-x86_64.sh" -b
echo "export PATH=\"$HOME/anaconda2/bin:\$PATH\"" >> ~/.bashrc
conda install -y bcolz
conda upgrade -y --all
install and configure theano
pip install theano
device = gpu
floatX = float32
root = /usr/local/cuda" > ~/.theanorc
configure jupyter and prompt for password and remember it!
jupyter notebook --generate-config
make a password
python -c "from notebook.auth import passwd; print(passwd())"
echo "c.NotebookApp.password = u'"$jupass"'" >> $HOME/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False" >> $HOME/.jupyter/jupyter_notebook_config.py
Place 'model' and 'data' folders back into the folder directories you took them from before ~/.keras and nbs lesson 1 respectively.
we will start Jupyter up as follows on the local and remote machines
Do this from the Local ( the machine with the GPU running the labs)
jupyter notebook --no-browser --port=8889
Do this from the remote machine you will need to modify the name and ip address
ssh -N -f -L localhost:8888:localhost:8889 machine name@ipaddress
if you have used a remote machine before for ssh you will need to get into the localhost file and erase the SHA keys against previously used local machines IP addresses.
OK now you will run the NBS course Lesson 1. It will look like it might work, but throw a compile error on In  the Vgg() section.
Next we will need to make some changes to the system to get this to work -at all-.
Close off the session jupyter session on the remote, locate the cmd line window running jupyter and ctrl-c.
close-off jupiter on the local machine, ctrl-c.
Next we are going to download the libgpuarray libraries key to getting this going.
git clone https://github.com/Theano/libgpuarray.git
cmake .. -DCMAKE_BUILD_TYPE=Release
Then change the .theanorc which should be in cd~ but is hidden use ls -a to find:
device = cuda
floatX = float32
root = /usr/local/cuda
Finally install pygpu using conda this is also key
conda install pygpu
Probably a good time for a reboot so restart the local
next restart the local machine jupyter session and remote ssh sessions. I suggest you set up an additional ssh session to the local machine as well as jupyter.
on the remote do..
on the local do...
jupyter notebook --no-browser --port=8889
Do this from the remote machine, please insert your details your-machine-name@your-ipaddress
ssh -N -f -L localhost:8888:localhost:8889 machine-name@ipaddress
bring up a http bowser window and put in local machine ipaddress:8889
re run Vgg on line  where we do the import and start theano you will get a nasty message like this or very similar
/home/sl/anaconda2/lib/python2.7/site-packages/theano/gpuarray/dnn.py:135: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to version 5.1.
warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 6020 on context None
Mapped name None to device cuda: GeForce GTX 1080 Ti (0000:02:00.0)
Using Theano backend.
Just ignore above nasty message and re-run line , lesson1 Vgg it should now work. next look at the nvidia card and see how its running.
polling the card while running the Vgg session will lower times by a couple of seconds BTW.
Next refinements: this is up to you.
create some aliases so you don't have to re-run long instructions on the local and remote.
on local machine running our lab and GPU
echo 'alias ju=‘jupyter notebook —-no-browser —-port=8889’' >> ~/.bashrc
on remote the alias assuming iMAC or similar you will need to modify the .bash_profile or use nano to get into .bash_profile
again this is for apple iMAC
echo 'alias remote='ssh -N -f -L localhost:8888:localhost:8889 sl@.localdomain' >> ~/.bash_profile
ok higher risk activities ahead! -you don't need to do this - lets upgrade to nividia driver 378.13
close jupyter notebook on the local as this may cause a crash to lightdm , causing guest login re-cycling if you do not gracefully exit jupyter notebook when rebooting.
download the nividia driver 378.13 driver from nividia.com and place it in ~/Downloads
make sure you are on the remote machine and ssh'd in - this time I really mean you will need to ssh in as you will lose the GUI completely on local. You maybe able to get in on the local cmd line using ctrl+alt+f1 but it does not work always, so ssh.
sudo service lightdm stop
local screen goes black
sudo init 3
remove the nvidia 375 driver
sudo apt-get remove --purge nvidia-375 nividia-modprobe nvidia-settings
there goes nvidia 375, now install 378.13
sudo sh NVIDIA-Linux-x86_64-378.13.run
follow screens ignore errors when it asks that the pre-install didn't go well, don't accept 32bit if offers and say yes to X configure and the reboot local.
back in local check with nvidia-smi restart jupiter on both machines and see how it goes it may shave a few seconds off.
Thats it!, sorry if there are typos or I missed something this is a recollection of events also '<>' did not come out on forum for ssh instructions usually its ssh local-machine-name@loaclmachine-ipaddress