Training of planet set or even finding a learning rate for Planet data set of lesson 3 is very slow for me.
1 epoch takes close to 2 hours. Is it normal?
Could it be somehow related that it shows that I do not have much space left on my Google drive (occupied about 13.5Gb from 15)?
Maybe an obvious question but did you select GPU for the hardware accelerator via
Edit --> Notebook Settings ?
3 Likes
Yes, you were absolutely right!
I changed the environment to GPU but somehow it didn’t change and continued to train the model on CPU. After several experiments and creating a new copy of this notebook and moving a training data to different place I discovered the problem.
So, I had now my first hand experience of difference between training a model on GPU vs CPU.
The difference is about 30-40 times slower on CPU!
How to solve this problem? I am facing the same problem here. I already use the GPU setting, but it still running very slow. Check using nvidia-smi, the GPU is correctly detected. Need help please. How to solve this problem. During training it only using 2.5GB of memory.
Here is some image from the notebook
