How to make sure GPU is utilized 100% on fastaiv2(pytorch) windows

fastai
1

So, I created the segmentation data using Unity3D & then training using fastai v2. The problem I was facing is although torch.cuda.device(0) always show me that I have 2080ti as GPU but while training it I can’t get GPU utilisation to pass 50%. Like GPU utilisation is always 47-50% or close to that. Whereas CPU is always peaking up & all the 16-thread seems to be utilized in parallel

nvidia-smi also doesn’t get past 47% is it that slow on 2080Ti or I’m doing something wrong? Even after forcing msi-afterburner to 100% power consumption it only got me from 47% to 50%

here’s a video training live: https://www.youtube.com/watch?v=VjJkakMiHgc&t=274s
here’s my ipynb file: https://github.com/AbhimanyuAryan/ml-imagesynthesis/blob/master/Unity.ipynb

You can see yourself like every epoch takes like 1:35 on average. Should it not be faster than that?

training
training948×984 116 KB

2

I think on windows you can’t get always %100 gpu use because you have to use 0 workers due to pytorch on windows. So, preparing batch slows down the training and gpu usage. You may try to increase the batch size too.

3

ok I’ll look into increasing batch size. Thanks