Show GPU utilization metrics inside training loop (without subprocess call!)

When I am running training I always run watch -n1 nvidia-smi, (or thanks to Jeremy’s hint now nvidia-smi dmon).

I would love to see that integrated in the fastai training loop, but obviously using subprocess with nvidia-smi would be less than ideal so I researched this a bit.

nvida-smi apparently is only a wrapper around the nvml c-lib (NVIDIA Management Library). nvidia provides python bindings for that lib and a demo , so we can directly call this from within python, no need to run nvidia smi as a system process.

so pip install nvidia-ml-py3 and then (quick example):

import nvidia_smi

handle = nvidia_smi.nvmlDeviceGetHandleByIndex(0)
# card id 0 hardcoded here, there is also a call to get all available card ids, so we could iterate

res = nvidia_smi.nvmlDeviceGetUtilizationRates(handle)
print(f'gpu: {res.gpu}%, gpu-mem: {res.memory}%')

This could be integrated into the fastai library itself (if this is of interest I can give it a try) or of course could be used in callbacks etc. individually.

Just wanted to bring it to the attention of people that this is possible.

Additional info:
The original nvidia version (pip install nvidia-ml-py) only supports python2, the version used above (nvidia-ml-py3) is a patch by a user, see github)


Very cool - that’s a new one to me. I think you could create some pretty cool callback with this! e.g. extra statistics for the dynamic graph callback to show during training…


This is a very useful tip. Thanks!

Goodbye watch nvidia-smi in a separate tmux pane!


FWIW this is what I now use instead of watch -n1 nvidia-smi:

but I agree that callback approach would be great, leading to the potential maxbs_find to find the biggest batch size you can fit in your model.


Possibly worth noting that if you want to see that the same numbers that nvidia-smi reports, you might want to use:

mem_res = nvidia_smi.nvmlDeviceGetMemoryInfo(handle)
print(f'mem: {mem_res.used / (1024**2)} (GiB)') # usage in GiB
print(f'mem: {100 * (mem_res.used /}%') # percentage usage

rather than

res = nvidia_smi.nvmlDeviceGetUtilizationRates(handle)
print(f'gpu: {res.gpu}%, gpu-mem: {res.memory}%')

They report slightly different numbers - there is a discussion here with the details:


Also netdata ( produces some useful graphs on nv usage.

1 Like

How do you get those graphs? Is it a third-party application?

It would be wonderful, but we would need it to be on conda as well, and currently I see only nvidia-ml-py on conda. If you could ask that person who ported it to put it on conda (not conda-forge) that would be helpful.

It’s pretty easy to port pypi’s to conda: this will create the recipe for conda-build.

conda skeleton pypi nvidia-ml-py3

full explaination is here

But I see neither pypi has a wheel built, it’s just a tarball.

It’s a pure python module, correct? despite using ctypes

So it should then be noarch for the conda-package.

And it doesn’t seem to work via main:

$ python
Traceback (most recent call last):
  File "", line 873, in <module>
  File "", line 228, in XmlDeviceQuery
    strResult += '    <product_name>' + nvmlDeviceGetName(handle) + '</product_name>\n'
TypeError: must be str, not bytes
1 Like

It’s netdata (see the link in my previous post) that actually produces these graphs. It’s a linux daemon that creates a small web server containing the aforementioned graphs which are usually from per second metrics. It also creates graphs for pretty much everything on a linux box, (cpu, memory, iowait etc)

1 Like

That’s an easy fix (just use an py36 f-string). Since this module doesn’t seem very efficient or updated, we’d probably want to fork it (license permitting) and maintain our own version and conda/pip packages.

1 Like

That’s what I would have thought, and only take the parts that are needed. It’s BSD Licenced, so the copyright notice must be reproduced in the code/docs. PyPI lists this in the official NVIDIA version:


Copyright © 2011-2015, NVIDIA Corporation. All rights reserved.


Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  • Neither the name of the NVIDIA Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

plus the usual ‘as-is’ disclaimer which I ommited here…


This and gputil are now documented here: