Moving inference to the CPU

#1

I’ve trained a model using fast.ai v1. I now want to deploy the model and run predictions but only on CPU.

I’ve tried the following which is taken from this topic. The code looks like:

learn.model.training = False
learn.model = learn.model.cpu()

When I attempt to obtain a prediction:

lean.predict(df_test.iloc[0])

I get the following error:

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #3 ‘index’

This error has also cropped up in the Productionizing models thread and is here

Any idea how to resolve the error?

1 Like

Tabular data not moved to gpu automatically
#2

You model is on the CPU but your data is automatically on the GPU, that’s why you have this error. Either don’t put the model on the CPU or change the default device to default.device = torch.device('cpu').

4 Likes

#3

Perfect! that did the trick thanks!

0 Likes

(Pierre Guillou) #4

Hello @sgugger, in the notebook lesson2-download.ipynb, default.device = torch.device('cpu') does not work when I run it in my computer, neither fastai.defaults.device = torch.device('cpu')

What works is fastai.device = torch.device('cpu')

For you ?

0 Likes

#5

Are you sure you are using the latest version?

0 Likes

(Pierre Guillou) #6

I use the fastai version 1.0.39.dev0 in windows 10. It is not the last one ?

0 Likes

#7

Yes, it’s the latest release, and defaults.device should work (if you don’t import * you will have to import defaults manually from its module).

0 Likes

(Pierre Guillou) #8

You are right that defaults.device works but not (at least with my configuration) fastai.defaults.device as written in the notebook lesson2-download.ipynb.

More, fastai.device works as well.

See below my tests:

2 Likes

(Martin Bai) #9

@sgugger, maybe a late one, but worth checking.

Regarding the inference of CPU, I have observed that half of the number CPU is using for batch preparation and the rest of half is always using for actual matrix calculation to get the predictions? Is anyway I can split usage of CPUs not half-half? I tried num_workers, but it seems do not have any impact?

0 Likes