About predicting on CPU

If I want to predict upon an image on an inference machine (fastai cpu-only install), I have zero problems.

If I want to predict upon an image on my development machine (gpu install) BUT using the cpu, I get some problems. Indeed, I set the defaults to cpu, and move the model to the cpu with .cpu().

Then I open an image with open_image() (this returns a fastai Image object which wraps a pytorch tensor).

But, it seems, the image is acquired using the gpu, since it complains:

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

But the most surprising thing is that a call to {imagename}.data.type() returns torch.FloatTensor, so it seems the data type is the correct one.

Despite of this, as I move my model back to .cuda(), the prediction works flawlessly.

Any suggestion? :roll_eyes:


Sometimes, I use the same config: training with GPU and inference on cpu in the same machine.
When I do that, during the inference, I first start the recipe with:

import torch

I can run the recipe without error and make prediction.
I’m sure the GPU is not used because I monitor it
Hope it helps…

1 Like

import torch


Strange. It should be torch.device, not torch.cuda.device.

Thanks, however, I’ll try ASAP.

1 Like

There is also torch.device but I haven’t used it.
I confirm I use:
import torch


You need to set the fastai device default to cpu. This will ensure that all tensors, either data or model weights, will be run on the cpu. You can do this with a single line of code:

fastai.defaults.device = 'cpu'

Then when you want to go back to running on the GPU run the command:

fastai.defaults.device = 'cuda'

Hope this helps!

source: approximately minute 40 on video 2 of the practical deep learning course 2019


I tried that too. But I think I was not clear enough in describing my issue.

Like I said, if I set the default device to cpu, and the open an image and inspect the tensor, I find it to be a cpu tensor. Nonetheless, the predictor says it is a cuda tensor (while it requires a cpu tensor).

Another thing to notice, I do NOT take the image to be predicted from the (validation) dataset. I open it with fastai’s open_image(filename).


Ah, I see it now, I must have skipped over that. Is it possible to post the code that you use for setting up your data set and training?

and setting up the model.


Sure. The gist is:

data = ImageDataBunch.from_folder(path,
                                  size=299, bs=bs,

learn = create_cnn(data, models.resnet34, metrics=accuracy, ps=0.8)

The training goes on successfully.

Then, the inference: I put fastai/torch in cpu mode, export the model, and the load it.

inf_learn = load_learner(path='/my/path',fname='name.pkl')

im = open_image('name.jpg')


And it throws the error.


Ok, my problem seems to be solved.

Strange things do happen: if I put fastai/pytorch in cpu mode with:

fastai.defaults.device = 'cpu'

It accepts it and defaults.device does confirm ‘cpu’. But prediction doesn’t work.

If I activate cpu mode with (it should be perfectly equivalent):

fastai.torch_core.defaults.device = 'cpu'

It works.

Thanks, however: you did stimulate me in trying even the most improbable things :slight_smile:


Great! Glad its working. I am curious to know why that was the case. Anyone have any idea why we are seeing this behavior?


guys you saved my life thanks


Am probably late to the party but for future readers, you can just call cpu() on the FloatTensor as detailed here.

1 Like

Done of these attempts work for me. I’m trying to run ML1 cpu only.
Ubuntu 16.04
python 3.6.3
created fastai-cpu with environment-cpu.yml
I removed many errors with:
conda install pytorch torchvision cpuonly -c pytorch <- from pytorch
conda install -c conda-forge python-graphviz <- from anaconda

presently my error in ml1…Lession4-mnist_sgd
net = nn.Sequential(
nn.Linear(28*28, 100),
nn.Linear(100, 100),
nn.Linear(100, 10),
AssertionError Traceback (most recent call last)
5 nn.ReLU(),
6 nn.Linear(100, 10),
----> 7 nn.LogSoftmax()
8 ).cuda()

~/anaconda3/envs/fastai-cpu/lib/python3.6/site-packages/torch/nn/modules/module.py in cuda(self, device)
309 Module: self
310 “”"
–> 311 return self._apply(lambda t: t.cuda(device))
313 def cpu(self):

~/anaconda3/envs/fastai-cpu/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
206 def _apply(self, fn):
207 for module in self.children():
–> 208 module._apply(fn)
210 def compute_should_use_set_data(tensor, tensor_applied):

~/anaconda3/envs/fastai-cpu/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
228 # with torch.no_grad():
229 with torch.no_grad():
–> 230 param_applied = fn(param)
231 should_use_set_data = compute_should_use_set_data(param, param_applied)
232 if should_use_set_data:

~/anaconda3/envs/fastai-cpu/lib/python3.6/site-packages/torch/nn/modules/module.py in (t)
309 Module: self
310 “”"
–> 311 return self._apply(lambda t: t.cuda(device))
313 def cpu(self):

~/anaconda3/envs/fastai-cpu/lib/python3.6/site-packages/torch/cuda/init.py in _lazy_init()
176 raise RuntimeError(
177 "Cannot re-initialize CUDA in forked subprocess. " + msg)
–> 178 _check_driver()
179 torch._C._cuda_init()
180 _cudart = _load_cudart()

~/anaconda3/envs/fastai-cpu/lib/python3.6/site-packages/torch/cuda/init.py in _check_driver()
90 def _check_driver():
91 if not hasattr(torch._C, ‘_cuda_isDriverSufficient’):
—> 92 raise AssertionError(“Torch not compiled with CUDA enabled”)
93 if not torch._C._cuda_isDriverSufficient():
94 if torch._C._cuda_getDriverVersion() == 0:

AssertionError: Torch not compiled with CUDA enabled

Any help would be appreciated.
I found the error.
I removed “.cuda” from the nn.Sequential statement.

1 Like

Noting for future readers and for myself in the future: this method of forcing inference to be on the CPU (fastai.torch_core.defaults.device = 'cpu') works only if I call it before loading the learner from disk.