IPyExperiments: Getting the most out of your GPU RAM in jupyter notebook


(Stas Bekman) #41

Thank you for running the update, @balnazzar.

Yes, this is a known caveat, please see: https://github.com/stas00/ipyexperiments/blob/master/docs/cell_logger.md#peak-memory-monitor-thread-is-not-reliable. Please vote for pytorch implementing the needed support. https://github.com/pytorch/pytorch/issues/16266

It probably has to do with Tesla being a faster card than GTX, so the monitoring thread can’t keep up with it, due to python thread implementation limitations.

Currently pytorch-1.0.1 added a single counter, so perhaps I’ll just try to use it for celllogger, instead of the python monitor thread. But we can’t use it concurrently.


(Andrea de Luca) #42

Thanks! I’ll do what you suggest.


(Stas Bekman) #43

OK, I implemented locking and 0.1.16 has been released. Please let me know if the crashing still occurs (or any deadlocks).

Fixes negative peak memory reports too.


#44

Hi @stas

I have two question in this regard.

1-In the output of IPyExperimentsPytorch, CPU Ram refersto the whole system RAM or to the CPU Cache?

2- how can I get the CPU, RAM, GPU usage and Time taken for training per epoch in my log file? Ideally I would like to have a CSV log file like this :


(Stas Bekman) #45

I will let the code answer in the best way:

    def cpu_ram_total(self): return psutil.virtual_memory().total
    def cpu_ram_avail(self): return psutil.virtual_memory().available
    def cpu_ram_used(self):  return process.memory_info().rss

now you can look them up and see what they all mean.

2- how can I get the CPU, RAM, GPU usage and Time taken for training per epoch in my log file? Ideally I would like to have a CSV log file like this :

It’s already done by:
https://docs.fast.ai/callbacks.mem.html#PeakMemMetric


#46

using PeakMemMetric callback I am getting this error:

I have imported callback as :
from fastai callbacks import *


(Michael Schuldes) #47

use from fastai.callbacks.mem import PeakMemMetric

if you take a look into the __init__.pyof the callbacks folder you will see that mem is not imported by default.


(Stas Bekman) #48

What @msandroid said. And I updated https://docs.fast.ai/callbacks.mem.html#PeakMemMetric to include that missing line. Thank you, @msandroid!


#49

thank you @msandroid and @stas


(Stas Bekman) split this topic #50

A post was merged into an existing topic: Developer chat


(Stas Bekman) split this topic #51

A post was split to a new topic: PeakMemMetric