Cuda error 45: no kernel image - Pytorch 0.3.1.post2

April 23 2018: Just noticed this on the Pytorch forum:


UPDATE :

I was nervous about getting the peterjc123 system form the internet in case it has also been updated, so I approached this as a manual exercise of copying over what I thought might be needed:

I removed cuda90-1.0-h4c72538_0.json and pytorch-0.3.1-py36_cuda90_cudnn7he774522_2.json from the fastai/conda-meta directory and replaced them with the equivalent cuda80-1.0-h205658b_0.json and pytorch-0.3.0-py36_0.3.0cu80.json from my working base system.

I likewise removed torch and torch-0.3.1.post2-py3.6.egg-info directories from fastai/Lib/site-packages and replaced them with torch and torch-0.3.0b0+591e73e-py3.6.egg-info from my base system.

Now when I fire up ipython and test - things work, and it reports the same information as my base system.

There may be gotchas with this approach, if anyone knows of any please let me know. I will post updates if I need to do any more work to get it functioning.

I have recently updated my fastai environment (under Windows 10) and now I have errors using even basic functionality. Pytorch reports it is now version 0.3.1.post2 and says it no longer supporting my old graphics card, so I want to revert to the earlier version 0.3.0b0+591e73e, which worked, or find some other solution apart from buying a new graphics card - which is not justified or even supported in this PC (I intend to buy something better but am not there yet).

Note that the error may be related to the cuda windows library version, as this also differs in my fastai environment (cudart64_90 vs cudart64_80).

I am able to reproduce the differences between the versions like this:

  1. In fastai environment:
import torch
print(torch.__version__)
0.3.1.post2

torch.cuda.find_cuda_windows_lib()
<CDLL 'cudart64_90', handle 7ffc97ed0000 at 0x1b2ebaa9860>

torch.cuda.set_device(0)
torch.cuda.get_device_capability(0)
(3, 0)

x = torch.cuda.FloatTensor(1)
D:\Anaconda3\envs\fastai\lib\site-packages\torch\cuda\__init__.py:116: UserWarning:
    Found GPU0 GeForce GTX 650 Ti which is of cuda capability 3.0.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))

x
 0
[torch.cuda.FloatTensor of size 1 (GPU 0)]

y = torch.cuda.FloatTensor(1)
y
 0
[torch.cuda.FloatTensor of size 1 (GPU 0)]

x + y
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-33-cd60f97aa77f> in <module>()
----> 1 x + y

D:\Anaconda3\envs\fastai\lib\site-packages\torch\tensor.py in __add__(self, other)
    303     # TODO: add tests for operators
    304     def __add__(self, other):
--> 305         return self.add(other)
    306     __radd__ = __add__
    307 

RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at c:\anaconda2\conda-bld\pytorch_1519501749874\work\torch\lib\thc\generic/THCTensorMathPointwise.cu:301

=======================================================

  1. On the other hand, in my base environment, which is still back on 0.3.0b0+591e73e (which I believe is a peterjc123 version), this works fine:
print(torch.__version__)
0.3.0b0+591e73e

torch.cuda.find_cuda_windows_lib()
<CDLL 'cudart64_80', handle 7ffc367f0000 at 0x24586fa5e80>

x = torch.cuda.FloatTensor(1)
x
 0
[torch.cuda.FloatTensor of size 1 (GPU 0)]

y = torch.cuda.FloatTensor(1)
y
 0
[torch.cuda.FloatTensor of size 1 (GPU 0)]

x + y
 0
[torch.cuda.FloatTensor of size 1 (GPU 0)]

Install pytorch 0.3.0 wheel works for me. The support for <=5.0 is gone since 0.3.1. My GPU is geforce 930m , window 10.

Not sure if we have pytorch 0.3.1 specific function will break the code though, just start going through lesson 9.

I have struggle for a while on this. The other choice is building pytorch from source, works for me on the same computer with Ubuntu 16.04, did not try on window yet.

Yes, it’s a pain and a bit nasty, but a relief to find that I’m not the only one!

Where / how do you get wheel 0.3.0? I see on the Pertjc123 site mentions of CUDA compatibility and links to CI Packages but not wheels with version numbers…

I am thinking that there’s an advantage in building Pytorch from source as then we can get debugging into the code as well - something I’ll look into it soon (I’m on Windows 10).

It’s on peterjc123 page already, you have to read carefully though as I overlook too. As the instruction look too complicated but actually straight forward.

Click the icon on window gpu 0.3.0(build/pass). Inside that page, there are multiple versions. Read the doc carefully, click artifact and select the corresponding package u need (it is base on your window os, gpu architecture and python version). Will add some screenshots when I get access to computer.

I wrote a quick blog post on this. Please let me know if this helps. I can run first 1/3 lesson 9 notebook without seeing any error so far.

I install a GPU PyTorch 0.3.0 version.

2 Likes

That’s great - thanks very much. I will look at it tomorrow - got to get some sleep right now!

BTW there is an space in the link, between the p and y of pytorch,

Cheers
Chris

If this is what you mean (below) I still don’t know what I am doing with the page I’ve landed on…

Do you mean this:
image

Leading to this?

Hi Chris, did you read the blog already? I create a little gif hope it helps…

1 Like

I have now, thanks Nok!

Can you clarify that if we compile then we will compatibility with our GPU even with the get 0.3.1 and beyond. otherwise (if using a wheel) we need to stick with no more that 0.3.0?

I cannot speak for the PyTorch community, but from this same thread it said compile from source should work.

I have no idea what’s the difference between pip install .whl file and build from source, does that mean someone build from source already and make a .whl file?

(Edited: 0.3.1.wheel Seems work for me too, but I ran into memory issue that I didn’t have when I use 0.3.0.whl. I suggest a reboot after you install the wheel file, make sure you uninstall any torch first then pip install the file. I can run the notebook after reboot.)

Thanks

I had read that thread a couple of days back, but got to the bit at the end where @smb was saying he encountered a problem (quoted below) - and thought that it wasn’t therefore very straigtforward.

After pip installing the 0.3.1 wheel, did you still get the warning, but it continued to run your code? Did you get over your memory problems with the 0.3.1 wheel installation?

Quoted from the Pytorch thread:

No warning, after reboot seems smooth and no memory issue (My GPU has 2GB ram only).

I believe my ubuntu was 16.04, I did compile from source before and things went smooth. I have not compile on Window as it is more work than in Linux. I think as long as your GPU are not too old, it would still work.

Thanks nok

Can you please confirm - was this after installing 0.3.1 using the wheel, or after compiling from source?

Wheel. (I believe wheel file is just someone complie from source already and generated a binary file after some googling) Thats why in peterjc123 repo there are many Wheel built for different machine type. Just make sure you install the correct one.

Thanks nok.

I pip installed the 0.3.1 wheel for my architecture (Keplar) and VS (2015), and so far things work.

BTW, a big part of my slowness to upgrade is the cost of buying a new PC, but another is my confusion about how far I can go with a better GPU for my current PC, which is 4 years old and has a Gigabyte H87M-D3H motherboard. When I google for compatible GPU’s I find this site, but I am not sure what might be the best choice from the old GPUs listed - or even if its possible to buy one. I wonder if I can get current GPUs on to this board, and they are just not listed on this site because its not maintained. If you have any expertise in this area I would love to hear from you!

You’ve been really helpful, I deeply appreciate your involvement and willingness to help!

Thank again for all the effort you put into this!

Chris

1 Like

I am also struggling to put up a cheap box. You may want to check this thread.

From the link that you provided, GPU compatibility seems fine as it supports 1080Ti, but the problem is it only support DDR3 Ram and 4th gen intel CPU which maybe hard to find.

Yes, I’ll look into that again - nice to know I can support more modern cards on this motherboard - do you get this information from a parts website? Though I think there is not much going on in the PCI lanes of this motherboard - Jeremy recommends 8 lane PCI, I’m not sure what this has. I already have RAM (but only 16GB) and CPU i7, this is my current PC - I just thought I should start with upgrading it with something better than my GTX 650 Ti

I just check the compatible GPU on the website that you provided. You may want to double check before you buy a GPU. I guess 16GB RAM and i7 will do the job for this course. You can always upgrade afterward if needed.

Shows my ignorance! I had completely overlooked that section…

But I am pretty sure that my motherboard would not do a good job of this, since its an older design. What GPU do you have and how much RAM does it have? Do you find it is good enough for the fast.ai code, or do you need to reduce batch sizes when models are unfrozen?

1 Like

I was only 1/2 the way on lesson 9 notebook. Does not have memory issue so far. MY GPU was 930M 2GB, speed is not awesome. If you want to have instant response it is not enough.

I am not advance enough so at this stage most of the time I spent on understanding every line of code. I do not run line by line, I run the whole section before I want to work on it. So I can experiment each variable when I come back and avoid waiting for training time.

My laptop has a i7 CPU and 12 GB ram.