EDIT: Adding in the bash version becasue it’s way easier:
I made a little python notebook to monitor GPU memory usage after getting a few OOMs. It uses nvidia-smi --display=MEMORY -q, to query memory stats. It can only update a few times per second because I’m using matplotlib inefficiently.
If anyone can figure out how to run it in parallel with model fitting in a single notebook, let me know!
You can probably do something like what you want with IPython widgets.
This isn’t a graph, but it will display the same information. The way I’ve written it you need to update it manually with update_widget, but I’m sure someone else will know how to get it to update at timed intervals asynchronously.
import ipywidgets as widgets
from IPython.display import display
from subprocess import check_output
return check_output(['nvidia-smi'] + options)
def update_widget(w=None, new_box=False):
if w is None:
w = widgets.Textarea(
w.value = nvidia_smi()
# display widget and get handle
w = update_widget()
# update info in widget