I’m training a dataset of about 100,000 images and each batch is taking about 20 minutes on my machine. However, maybe only 3 out of those 20 minutes involves heavy CPU, GPU, and/or disk activity. The rest of the time, the training progress bars are not moving and there’s little/no GPU, CPU, or disk activity so I have no idea where the bottkeneck is and what fastai is waiting on. However I do see a litte more than usual network activity during the delay like 50-100 Kbps/s being transferred in both directions.
Oh, and I’m using Windows. It seems other Windows users are having this problem. I’ve also tried running it on a smaller subset of the data like 1000 images and am still getting an unexplained delay that’s maybe slightly shorter than on the entire 100,000 images (didn’t time it.)
My machine has 2 GPUs, a very low end 940M with 2GB and a 1080 with 11GB. Each epoch on the 1080 still takes maybe 80-95% as long as on the 940M despite the latter GPU being light years ahead of the former due to these unexplained delays.
Can anyone move me one step closer to a resolution?
Have you tried setting num_workers=0 when creating the databunch? ImageDatabunch defaults to num workers = num cpus. I know on windows pytorch has a lot of problems with multiprocessing, so it might be worth switching that off by setting it to zero. (When using images that might mean the loading/transforming on the cpu will likely become the bottleneck then, but at least that would be explainable )
Update: this seems to be fixed for me when it comes to Images and Tabular but I’m still having the same problem on Windows when initalizing a TextLMDataBunch.from_csv even if i add num_workers=0
Im using fastai 1.0.52 and pytorch gpu 1.1.0 for windows on a GTX 1080
Unfortunately, no. However, I tried using Keras on the same computer and it, too, was suffering from similar delays for no reason. Also tried upgrading to Cuda 10,still no dice.
This is on my work computer which is running Windows 10 with a 1080.
I also have this issue on my personal laptop with a 960M but it doesn’t seem as severe. Also running windows 10
I have the same issue with almost exact configuration as you.
I am currently on fastai 1.5.0, CUDA 10 on an 1080Ti, windows 10.
The pattern to me seems to be between training and validation there is a long waiting period (around 1 minute+ in the case for around 1000 images) which only one core of CPU being pulled to 100%, and then during the actual training all the CPU would be pulled to 100%.
Adding num_workers =0 improve the training speed for all three data bunch method and its now working normally as the same hardware on linux.
Judging from the performance I suspect that the problem may be some part of loading data in fastai or pytorch cannot automatically be multi-threaded, but I am not sure exactly.
)