IPyGPULogger: GPU Logger for jupyter/ipython memory usage and exec time

fastai
1

Moved

ipygpulogger got integrated into ipyexperiments. Discussion forum is here: IPyExperiments: Getting the most out of your GPU RAM in jupyter notebook

Old info

This thread is for discussing ipygpulogger - GPU Logger for jupyter/ipython memory usage and exec time and potentially more features.

Here is how it works. You start the logger and then after each cell is executed you get the report of consumed memory (general and GPU), including peak and total used RAM, and execution time:

snapshot1

Full details are available here.

If you have problems using it, or have ideas and improvements to suggest please use this thread.

3 Likes
2

got this message:
✘ C:\Users\kl\AppData\Local\conda\conda\envs\fastai\lib\site-packages\ipygpulogger\ipygpulogger.py Something weird happened and this ran for too long, this thread is killing itself

1 Like
3

First, just to clarify, you’re saying that it’s working OK, but you got that message after it being idle for some time, right?

That was a bug in the peak used memory monitor thread. now fixed - please try again with the latest git.

4

yes according to windows GPU monitor panel it is very efficient in triming the cache. I believed that “the thread is killing itself” started during backprop. In that periode the timeline for the GPU-mem stopped being sawtoothed as if the allocation was constant or the cache-reduction was deactivated

To get the new version runnning i did: pip uninstall ipygpulogger
That worked. then in reinstalled using: pip install git+https://github.com/stas00/ipygpulogger.git
The version number did not change though!

will report later on the new version

5

I haven’t made any releases yet, giving it a few days to settle, and getting feedback, so the version won’t change until then.

You don’t need to uninstall anything, just:

pip install git+https://github.com/stas00/ipygpulogger.git

will install the latest version.

yes according to windows GPU monitor panel it is very efficient in triming the cache. I believed that “the thread is killing itself” started during backprop. In that periode the timeline for the GPU-mem stopped being sawtoothed as if the allocation was constant or the cache-reduction was deactivated

Are you saying you’re still getting this error message after installing the latest version?

6

i am not ready to ready to restart the notebook yet so i cannot tell for sure.

I just executed

from ipygpulogger import IPyGPULogger
il = IPyGPULogger().start()

and got this:

Error in callback <function IPyGPULogger.pre_run_cell at 0x000002AB25C9C488> (for pre_run_cell):
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~\AppData\Local\conda\conda\envs\fastai\lib\site-packages\backcall\backcall.py in adapted(*args, **kwargs)
    102                 kwargs.pop(name)
    103 #            print(args, kwargs, unmatched_pos, cut_positional, unmatched_kw)
--> 104             return callback(*args, **kwargs)
    105 
    106         return adapted
~\AppData\Local\conda\conda\envs\fastai\lib\site-packages\ipygpulogger\ipygpulogger.py in pre_run_cell(self)
    101 
    102     def pre_run_cell(self):
--> 103         if not self.running: return
    104 
    105         self.peak_monitoring = True
AttributeError: 'IPyGPULogger' object has no attribute 'running'

RAM: Consumed Peaked Used Total in 0.045s (In [134])
Gen: 0 0 5902 MB
GPU: 2 0 2360 MB
RAM: Consumed Peaked Used Total in 0.090s (In [134])
Gen: 0 0 5902 MB
GPU: 0 0 2360 MB

Its harmless so i just continue to use the notebook instance for the experiments i am running

7

Thank you for the logs, @Kaspar. Please remember to format code/logs with preformatted text in the post (or manually with ``` … ```)

As mentioned earlier, this is a totally new module, so its internals still go through change. It will work when the nb is restarted, since you now have the old version and the new loaded into python.

1 Like
8

hmm! how do i preformat the text ?

9

You can do it by hand: https://discourse.stonehearth.net/t/discourse-guide-code-formatting/30587
and you have a </> button in your post editor, which you can press after selecting the text you want to format.

and make sure to update ipygpulogger git again before you restart things, as I’ve been committing fixes.

10

Note: ipygpulogger got integrated into ipyexperiments. Discussion forum is here: IPyExperiments: Getting the most out of your GPU RAM in jupyter notebook

11