Cuda error 45: no kernel image - Pytorch 0.3.1.post2

Chris_Palmer · March 31, 2018, 9:11pm

Thanks

I had read that thread a couple of days back, but got to the bit at the end where @smb was saying he encountered a problem (quoted below) - and thought that it wasn’t therefore very straigtforward.

After pip installing the 0.3.1 wheel, did you still get the warning, but it continued to run your code? Did you get over your memory problems with the 0.3.1 wheel installation?

Quoted from the Pytorch thread:

Unfortunately I couldn’t resolve

ldd /home/../anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so

[....]
libmkl_gf_lp64.so =&gt; not found
libmkl_gnu_thread.so =&gt; not found
libmkl_core.so =&gt; not found
[....]

Therefore, to run pytorch code, prefix it with

LD_LIBRARY_PATH=/home/.../anaconda3/lib64/:/home/.../anaconda3/lib/ python my_pytorch_code.py

nok · April 1, 2018, 2:52am

No warning, after reboot seems smooth and no memory issue (My GPU has 2GB ram only).

I believe my ubuntu was 16.04, I did compile from source before and things went smooth. I have not compile on Window as it is more work than in Linux. I think as long as your GPU are not too old, it would still work.

Chris_Palmer · April 1, 2018, 11:34am

Thanks nok

Can you please confirm - was this after installing 0.3.1 using the wheel, or after compiling from source?

nok · April 1, 2018, 11:39am

Wheel. (I believe wheel file is just someone complie from source already and generated a binary file after some googling) Thats why in peterjc123 repo there are many Wheel built for different machine type. Just make sure you install the correct one.

Chris_Palmer · April 1, 2018, 6:46pm

Thanks nok.

I pip installed the 0.3.1 wheel for my architecture (Keplar) and VS (2015), and so far things work.

BTW, a big part of my slowness to upgrade is the cost of buying a new PC, but another is my confusion about how far I can go with a better GPU for my current PC, which is 4 years old and has a Gigabyte H87M-D3H motherboard. When I google for compatible GPU’s I find this site, but I am not sure what might be the best choice from the old GPUs listed - or even if its possible to buy one. I wonder if I can get current GPUs on to this board, and they are just not listed on this site because its not maintained. If you have any expertise in this area I would love to hear from you!

You’ve been really helpful, I deeply appreciate your involvement and willingness to help!

Thank again for all the effort you put into this!

Chris

nok · April 2, 2018, 5:04am

I am also struggling to put up a cheap box. You may want to check this thread.

From the link that you provided, GPU compatibility seems fine as it supports 1080Ti, but the problem is it only support DDR3 Ram and 4th gen intel CPU which maybe hard to find.

Chris_Palmer · April 2, 2018, 7:40am

Yes, I’ll look into that again - nice to know I can support more modern cards on this motherboard - do you get this information from a parts website? Though I think there is not much going on in the PCI lanes of this motherboard - Jeremy recommends 8 lane PCI, I’m not sure what this has. I already have RAM (but only 16GB) and CPU i7, this is my current PC - I just thought I should start with upgrading it with something better than my GTX 650 Ti

nok · April 2, 2018, 8:13am

I just check the compatible GPU on the website that you provided. You may want to double check before you buy a GPU. I guess 16GB RAM and i7 will do the job for this course. You can always upgrade afterward if needed.

Chris_Palmer · April 2, 2018, 8:43am

Shows my ignorance! I had completely overlooked that section…

But I am pretty sure that my motherboard would not do a good job of this, since its an older design. What GPU do you have and how much RAM does it have? Do you find it is good enough for the fast.ai code, or do you need to reduce batch sizes when models are unfrozen?

nok · April 2, 2018, 10:21am

I was only 1/2 the way on lesson 9 notebook. Does not have memory issue so far. MY GPU was 930M 2GB, speed is not awesome. If you want to have instant response it is not enough.

I am not advance enough so at this stage most of the time I spent on understanding every line of code. I do not run line by line, I run the whole section before I want to work on it. So I can experiment each variable when I come back and avoid waiting for training time.

My laptop has a i7 CPU and 12 GB ram.

Chris_Palmer · April 2, 2018, 10:59am

OK, I also have only 2 GB ram. Are you on Pascal notebook, if you are only halfway then you may not have got to the section where the last 3 layers of the model are unfrozen. I had to reduce batch size to work with that - I picked 32 (half of 64), and it was OK, I had to increase the number of cycles to get anything near to what Jeremy was showing, but even then the accuracy was really bad…

nok · April 2, 2018, 1:27pm

I cannot speak for this. I don’t think a GPU will affect the accuracy a lot, it is more about the speed. However, batch size did play a role in accuracy due to Batch Normalization, where performance start to become unstable when batch size was too small. But in your case Batch Size is still 32, which should be fine.

Did you get similiar performence compare to Jeremy’s notebook before you reach the part that you have to change batch size?

jeremy · April 2, 2018, 1:53pm

If you decrease the batch size 2x you’ll need to re-run the LR finder.

MaheshBhosale · April 2, 2018, 6:52pm

Hi Nok and chris,
I am in the same trouble as that of chris, I have updated the conda as a result pytorch has been updated to 0.3.1.post2 . I tried installing wheel from peterjc123 from here, but unable to install it through pip.
It gives me error,
(fastai) C:\Users\Mahesh.Bhosale\Downloads>pip install “torch-0.3.0b0+591e73e-cp36-cp36m-win_amd64 (1).whl”
torch-0.3.0b0+591e73e-cp36-cp36m-win_amd64 (1).whl is not a supported wheel on this platform.

Though my GPU is Quadro M1200 which is of Maxwell architecture.

Any help is much appreciated.

nok · April 3, 2018, 1:40am

Hi, Maxwell should be supported, just make sure you select the correct wheel. Do you have Win10 as your OS, since your wheel file looks like a Win10, python3.6 and Maxwell wheel file.

Chris_Palmer · April 3, 2018, 1:58am

Thanks Jeremy

I’ll insert that step and try again.

What is the best strategy - begin with the batch size that you’ll eventually need, or adjust down to it as required?

Chris_Palmer · April 3, 2018, 1:59am

Did you choose a wheel appropriate to your version of visual studio (2015 or 2017)?

Chris_Palmer · April 3, 2018, 2:10am

Yes, performance was similar - Jeremy has indicated that I need to re-establish the learning rate - so when I get home tonight I’ll incorporate that and see where it leads me.

MaheshBhosale · April 3, 2018, 3:38am

Hi nok, yes my OS is Windows10. Below is system information.

But one point worth noting is, I have intel proocessor but wheel file has “amd” at last, would that make any difference?

MaheshBhosale · April 3, 2018, 3:43am

I believe visual studio is needed for compiling from the source, as we have compiled binaries and packaged .whl, is visual studio needed while installing these as well? If yes, I wonder why would that be a possible case?

Though I do see README of peterjc123 mentions,

APPVEYOR_BUILD_WORKER_IMAGE : VS 2015/2017 This does not indicate the compiler it used, but the system it can run on. The former one can be used on Windows 7/8/8.1 and Windows Server 2008/2012, while the latter on Windows 10 and Windows Server 2016.