- pytorch caches RAM so nvidia-smi isn’t showing you the real picture. My code empties the cache so you get the actual reflection of the real available memory (to a degree due to fragmentation, so realistically it’s perhaps 80-90% of what you see). Note, the “no_cache” in the function name:
- if your memory is really tied up and can’t be flushed, then you won’t be able to run the cell that requires X-amount of free memory anyway.
The only caveat there is potentially an unreleased learner, that requires gc.collect() as well due to circular references. Probably should add it to the function as well.
Ideally, you should be able to free the memory and continue with your code w/o needing to restart. We are not 100% there yet, but are moving in that direction.
If you notice any situations where memory gets leaked and can’t be reclaimed please report those as it’ll help this effort.
You will need to run:
import gc, torch gc.collect() torch.cuda.empty_cache()
to be able to see the true representation via nvidia-smi, since it is not aware of pytorch cached memory. But the easiest way is to use: https://github.com/stas00/ipyexperiments which will automatically profile the memory usage for you and that way you can quickly see where the leaks are if any. You won’t need to watch nvidia-smi any longer.