Moving inference to the CPU

fastai
1

I’ve trained a model using fast.ai v1. I now want to deploy the model and run predictions but only on CPU.

I’ve tried the following which is taken from this topic. The code looks like:

learn.model.training = False
learn.model = learn.model.cpu()

When I attempt to obtain a prediction:

lean.predict(df_test.iloc[0])

I get the following error:

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #3 ‘index’

This error has also cropped up in the Productionizing models thread and is here

Any idea how to resolve the error?

1 Like
2

You model is on the CPU but your data is automatically on the GPU, that’s why you have this error. Either don’t put the model on the CPU or change the default device to default.device = torch.device('cpu').

4 Likes
3

Perfect! that did the trick thanks!

4

Hello @sgugger, in the notebook lesson2-download.ipynb, default.device = torch.device('cpu') does not work when I run it in my computer, neither fastai.defaults.device = torch.device('cpu')

What works is fastai.device = torch.device('cpu')

For you ?

5

Are you sure you are using the latest version?

6

I use the fastai version 1.0.39.dev0 in windows 10. It is not the last one ?

7

Yes, it’s the latest release, and defaults.device should work (if you don’t import * you will have to import defaults manually from its module).

8

You are right that defaults.device works but not (at least with my configuration) fastai.defaults.device as written in the notebook lesson2-download.ipynb.

More, fastai.device works as well.

See below my tests:

device
device1560×1560 201 KB

3 Likes
9

@sgugger, maybe a late one, but worth checking.

Regarding the inference of CPU, I have observed that half of the number CPU is using for batch preparation and the rest of half is always using for actual matrix calculation to get the predictions? Is anyway I can split usage of CPUs not half-half? I tried num_workers, but it seems do not have any impact?

10

In the first comment you stated it was default.device but in the second comment you stated defaults.device (with an s). That I think was the source of the confusion.
What worked for me was defaults.device = torch.device('cpu'), while fastai.device and default.device both did not work.

11

This could be set after my model has being loaded and trained in GPU?

2 Likes
13

This could be set after my model has being loaded and trained in GPU?

Hello,
Very good question, do you have the answer ?

14

Set dls.device = “cpu”

fastai looks at the DataLoader for the device to use. (And then adjusted the device the model is on)

Note: this is for fastai v2

1 Like
15

Thank you.

I have a last question.
On my computer I have a good GPU (NVIDIA GeForce GTX 1060 6GB), and I use my computer to create and test my model.

But I know my model wil be use on computer without GPU, so I want to test my model in real condition (without GPU).
My question is :
Can I create the model on my computer with my GPU, and after test and use it (on my computer) without GPU (to know the prediction time) ?

Or I need to create my model without GPU ?

I think I can create the model with my GPU, and when I use it I add this line :
defaults.device = torch.device(‘cpu’)
…
preds = learn_inf.predict(img)

16

Hi, you can create the model on your computer with GPU and use after that in a computer without it.

2 Likes