Was previously using nvidia-smi dmon, but this is what I use to keep an eye on the GPU now
nvidia-smi --query-gpu=timestamp,pstate,temperature.gpu,memory.total,memory.free,memory.used --format=csv -l 5
It outputs in the following format
timestamp, pstate, temperature.gpu, memory.total [MiB], memory.free [MiB], memory.used [MiB]
2018/03/21 15:33:24.573, P0, 63, 11441 MiB, 8115 MiB, 3326 MiB
2018/03/21 15:33:29.574, P0, 65, 11441 MiB, 8115 MiB, 3326 MiB
2018/03/21 15:33:34.575, P0, 66, 11441 MiB, 8115 MiB, 3326 MiB
More details here. http://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries