Memory leak in dataloader?

[Caleb (Caleb Everett)]

This previous forum post identifies this issue as ThreadPoolExecutor greedily pulling batches for each iteration of self.sampler into memory in Python 3.6 vs. pulling them in lazily in Python 3.5.

Memory Error (Not cuda out of memory error)

I am using an Azure Data Science Virtual Machine which has a P100 GPU. This is what I am doing: Load Densenet201 with precompute=True bs = 64 ( I tried 400 | 300 | 200 | 100), the GPU memory used is 4 Gigs of 16 Gigs Available sz = 399 I am getting Memory Error, even though I am using just 30% of the total GPU memory available to me. Can anyone help me with this? Stack Trace: MemoryError Traceback (most recent call last) in () ----> 1 learn = ConvLearner.pr…

There are two workarounds:

1. Set num_workers to 0, which then runs batches in a single thread. This resulted in the consumption of a max of 3GB of memory in the scenario above.
2. Use the dataloader iterator from Pytorch as described here.

I’m continuing to research ways to make a permanent fix. Any ideas on how to do that or things to look into would be much appreciated.

https://github.com/fastai/fastai/blob/master/fastai/dataloader.py

concurrent.futures — Launching parallel tasks