Model loaded on GPU but using GPU and CPU both For prediction

shubraj · February 3, 2022, 1:39pm

hello everyone,

I have trained a vision model on custom data on GPU(NVIDIA GeForce RTX 3060-12gig) using Fastai - 2.5.3. when I deployed that model in production on GPU(tesla 32 gig/ Cuda-10.2) for continuous predictions I noticed that that model is loaded on GPU but using my CPU as well on the server. so I tested that locally and the result was the same it was using both GPU and CPU on prediction.

so my question is. did I miss something or is there any problem with drivers?

here are the screenshots for both GPU and CPU utilization. it can be seen clearly model is using both GPU and CPU

please correct me if I am wrong somewhere.
Thank you

muellerzr · February 3, 2022, 1:43pm

We use CPU for some of the data preprocessing. Can you show what your prediction code is and how you load the model in?

shubraj · February 3, 2022, 1:49pm

@muellerzr thanks for the reply sir,

here is my example code -

torch.cuda.set_device(0)
model = load_learner(f’ai/fastai_recog.pkl’, cpu=False)

image_path = ‘path/to/image.jpg’
if ‘http’ in image_path:
with model.no_bar(): preds = model.predict(requests.get(image_path, stream=True).content)
else:
with model.no_bar(): preds = model.predict(open(image_path, ‘rb’).read())

print(preds)

muellerzr · February 3, 2022, 1:52pm

Your code looks fine. As I said we use the CPU for initial data preprocessing, so it will never be 100% gpu

shubraj · February 3, 2022, 1:53pm

ok Thank you sir @muellerzr