Fastai memory usage bigger than torch

fastai fastai dev
1

I am debugging an issue where importing fastai on inference is using way more memory than torch.

image
image1764×1507 251 KB

The second code gets Illegal memory acces blowing up 40GB of GPU memory.
This is somewhat random, sometimes it happens sometimes it does not.
Probably is linked to some torch imports like cudnn.benchmark, any ideas?

=== Software === 
python        : 3.8.5
fastai        : 2.0.16
fastprogress  : 0.2.7
torch         : 1.6.0
torch cuda    : 10.2 / is available
torch cudnn   : 7605 / is enabled

=== Hardware === 
torch devices : 2
  - gpu0      : Quadro RTX 8000
  - gpu1      : GeForce RTX 2070 SUPER

=== Environment === 
platform      : Linux-5.4.0-52-generic-x86_64-with-glibc2.10
distro        : #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020
conda env     : fastai
python        : /home/tcapelle/miniconda3/envs/fastai/bin/python
sys.path      : /home/tcapelle/SteadySun/nvidia-colab
/home/tcapelle/miniconda3/envs/fastai/lib/python38.zip
/home/tcapelle/miniconda3/envs/fastai/lib/python3.8
/home/tcapelle/miniconda3/envs/fastai/lib/python3.8/lib-dynload

/home/tcapelle/miniconda3/envs/fastai/lib/python3.8/site-packages
/home/tcapelle/Apps/nbdev
/home/tcapelle/Apps/fastai
/home/tcapelle/miniconda3/envs/fastai/lib/python3.8/site-packages/IPython/extensions
/home/tcapelle/.ipython
no nvidia-smi is found

Update :sweat_smile: 

torch.backends.cudnn.deterministic = False
torch.backends.cudnn.benchmark = False

appears to solve the issue

1 Like