When running X windows and cuda in parallel, X will take up some amount of the GPU memory which is not available for deep learning purposes. If your mainboard came with an onboard Intel graphics card, you might consider using the Intel card for all X windows purposes, thus having the NVidia GPU 100% available for Cuda. See this thread for approaches on how to achieve that.