RuntimeError: CUDA out of memory but terminal suggests I still have memory available


(Konrad Semsch) #1

When trying to execute another command in JN I get the following error:

RuntimeError: CUDA out of memory. Tried to allocate 49.00 MiB (GPU 0; 7.93 GiB total capacity; 7.39 GiB already allocated; 2.56 MiB free; 15.16 MiB cached)

but when I check in my terminal I can still see a lot of memory available on my GPU:

              total        used        free      shared  buff/cache   available
Mem:          30150        2549       22805          19        4795       27334
Swap:             0           0           0

Can anyone please shed some light what the issue here is?


#2

As far as I can understand, it’s a limitation with Jupyter/pytorch (which tends not to free memory when something bad happens).

My solutions so far:

  1. Make sure that you have pytorch >= 1.0
  2. Update fastai
  3. Restart the Kernel

(Konrad Semsch) #3

I’ve tried all 3 solutions along with decreasing the batch size and unfortunately the error persists.


#4

Don’t know what to suggest then. Be aware that sometimes Jupyter gets confused with various versions of the libraries. So make sure to check from within JN torch.__version__ and fastai.__version__


(Victor Bouzas) #5
CUDA out of memory. Tried to allocate 32.38 MiB (GPU 0; 4.00 GiB total capacity; 2.88 GiB already allocated; 2.17 MiB free; 69.94 MiB cached)

I had the same with the following configurations when I tried to run the tutorial on https://docs.fast.ai/text.html. Played along with bs and not even the lowest works.

=== Software === 
python        : 3.6.6
fastai        : 1.0.39
fastprogress  : 0.1.18
torch         : 1.0.0
nvidia driver : 416.34
torch cuda    : 10.0 / is available
torch cudnn   : 7401 / is enabled

=== Hardware === 
nvidia gpus   : 1
torch devices : 1
  - gpu0      : 4096MB | GeForce GTX 1050 Ti

=== Environment === 
platform      : Windows-10-10.0.17134-SP0
conda env     : Unknown
python        : C:\ProgramData\Anaconda3\envs\fastai\python.exe
sys.path      : 
C:\ProgramData\Anaconda3\envs\fastai\python36.zip
C:\ProgramData\Anaconda3\envs\fastai\DLLs
C:\ProgramData\Anaconda3\envs\fastai\lib
C:\ProgramData\Anaconda3\envs\fastai
C:\ProgramData\Anaconda3\envs\fastai\lib\site-packages
C:\ProgramData\Anaconda3\envs\fastai\lib\site-packages\IPython\extensions
C:\Users\CASAFARI\.ipython

Mon Jan 14 22:53:47 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 416.34       Driver Version: 416.34       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 105... WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   40C    P8    N/A /  N/A |   3491MiB /  4096MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      7708      C   ...amData\Anaconda3\envs\fastai\python.exe N/A      |
+-----------------------------------------------------------------------------+