Hi,
I am working in classification problem with images, and lr.find is not working (error message: “Failed to compute the gradients, there might not be enough points”). If I try to train the model, I immediately get NaN as training loss, sometimes the computer freezes.
I tried reducing size of images and batch size, it didn’t work.
When running on CPU it does work.
=== Software ===
python : 3.7.4
fastai : 1.0.59
fastprogress : 0.1.21
torch : 1.2.0
nvidia driver : 430.50
torch cuda : 10.0.130 / is available
torch cudnn : 7600 / is enabled
=== Hardware ===
nvidia gpus : 1
torch devices : 1
- gpu0 : 11016MB | GeForce RTX 2080 Ti
=== Environment ===
platform : Linux-4.15.0-66-generic-x86_64-with-debian-buster-sid
distro : #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019
conda env : fastai
python : /home/vip/anaconda3/envs/fastai/bin/python
sys.path :
/home/vip/anaconda3/envs/fastai/lib/python37.zip
/home/vip/anaconda3/envs/fastai/lib/python3.7
/home/vip/anaconda3/envs/fastai/lib/python3.7/lib-dynload
/home/vip/anaconda3/envs/fastai/lib/python3.7/site-packages
Tue Oct 29 09:50:23 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50 Driver Version: 430.50 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:01:00.0 On | N/A |
| 38% 39C P8 11W / 250W | 268MiB / 11016MiB | 13% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1445 G /usr/lib/xorg/Xorg 18MiB |
| 0 1484 G /usr/bin/gnome-shell 57MiB |
| 0 1842 G /usr/lib/xorg/Xorg 106MiB |
| 0 2001 G /usr/bin/gnome-shell 84MiB |
+-----------------------------------------------------------------------------+