The variable cnmem is an attribute of the old interface which is/will be deprecated in the next version of Theano. What we have been playing around with the Libgpuarray and pygpu is all part of this new interface.
In the original lesson1 on github it took 588 seconds to run 1 epoch in cell 7 with a Tesla K80.
I appreciate you have a better understanding of the hardware architecture.
I am short on memory which is about to be fixed (only 16GB). But I don't think that is the problem as the gpu is not using all it's power as seen from the nvidia-smi script. Mostly less than 50% memory.
The bandwidthTest result is a PASS . Somewhere there is a setting that replaces cnmem for the new interface I think it is
Default: 0 (Preallocation of size 0, only cache the allocation)
Controls the preallocation of memory with the gpuarray backend.
The value represents the start size (either in MB or the fraction of total GPU memory) of the memory pool. If more memory is needed, Theano will try to obtain more, but this can cause memory fragmentation.
A negative value will completely disable the allocation cache. This can have a severe impact on performance and so should not be done outside of debugging.
'< 0: disabled'
'0 <= N <= 1: use this fraction of the total GPU memory (clipped to .95 for driver memory).
'> 1: use this number in megabytes (MB) of memory.'
So for 1 I don't have that set. First thing todo tomorrow. Although it does not say how to set it in theanorc. running
python -c 'import theano; print(theano.config)' | less
may help. (cnmem is in [lib] section) what section these new parameters take maybe revealed from that config.
I have 1080ti in the second x16 slot; the 610 I use for video was in the first x16 slot. As the driver 378.13 can't see that gnu, I may take it out and replace the 1080ti in that slot when I add more memory.
My Processor spec is here
The memory in the motherboard is ddr4 2400hz registered dimm which is different from the cpu spec
Swap is on a separate disc too the ubuntu os.
I don't see any problems with cpu performance compared to your cpu. It may just be that config parameter. Have you checked the memory the Gpu reports when running nvidia-smi.. When I run next I'll post my findings for cell 7 lesson1 from a fresh startup. Cheers