Training Time


(Asimo) #1

I have a question regarding a training time (maybe its a bit silly). Why 1 epoch takes 4-5 time slow than others (please see the attached screenshot)?


#2

We can’t answer that question without knowing you code and hardware. Sometimes the first epoch is slower (though you are showing an extreme example of that).


(Asimo) #3

I fixed the error, my CUDA Driver was not 9.2… =)
Everything looks smooth now =)


(Marc Carrion) #4

I’m having a similar issue, and I’m monitoring the GPU usage and seems not to be used, how can I make sure the libraries are using GPU and not CPU?

nvidia-smi dmon
# gpu pwr gtemp mtemp sm mem enc dec mclk pclk
# Idx W C C % % % % MHz MHz
0 6 35 - 0 2 0 0 405 139
0 6 36 - 0 2 0 0 405 139
0 6 35 - 0 2 0 0 405 139