@than3 Thanks for sharing your experience and indeed it’s totally overwhelming-and a good reminder for us that someone with your level of experience can also get frustrated.
We like to call this the secret walk through fire that all of us have to face while working on DL Problems and once you enter the club, you don’t tell anyone about how hard it is to setup a box 
I’m j/k-it actually took me 13 days IIRC to setup everything the first time so I can completely relate.
Since you’ve a strong machine and have weighed in the other options, I would be happy to help debug, here are the outlines of the steps that work for me, happy to provide more details or debug:
- Install Ubuntu (fresh)
- Download CUDA (latest) and install just that (It also installs NVIDIA Drivers for you, just follow the instructions)
- Unzip CuDNN
- Set Paths (I always forgot the export step)
- Restart and install anaconda, followed by Conda installing fastai
Thanks,
Sanyam