Training Time

I have a question regarding a training time (maybe its a bit silly). Why 1 epoch takes 4-5 time slow than others (please see the attached screenshot)?

We can’t answer that question without knowing you code and hardware. Sometimes the first epoch is slower (though you are showing an extreme example of that).

I fixed the error, my CUDA Driver was not 9.2… =)
Everything looks smooth now =)

I’m having a similar issue, and I’m monitoring the GPU usage and seems not to be used, how can I make sure the libraries are using GPU and not CPU?

nvidia-smi dmon
# gpu pwr gtemp mtemp sm mem enc dec mclk pclk
# Idx W C C % % % % MHz MHz
0 6 35 - 0 2 0 0 405 139
0 6 36 - 0 2 0 0 405 139
0 6 35 - 0 2 0 0 405 139