Are you using theano or tensorflow? And which python version? I recently built my DL machine and the CUDA sample test result was ok. But after installing python 3.6, theano and tensorflow, I tried to run Lesson 1 Vgg16 model and like your case, my GPU's usage was 0-3% percent.
I then went to Tensorlfow and Theano tutorial/installtion pages and ran their simple scripts to test if they are using GPU. Tensorflow test passed but not Theano. I could run part 2 lessons which uses Tensorflow and my GPU usage are 80-90%.
I found out that I had installed Theano using Conda but it is currently not supporting python 3.6. I have not tried installing it using pip or binary source since I don't need to run Theano right now.
I thought my experience might help you resolve this issue.