Howto: installation on Windows

ecdrid · January 29, 2018, 5:09am

I am having everything in the default enviornment setup by Anaconda3

This setup works perfectly for fast.ai till now… ( i had done conda install everything i want and whenever i want to use my jupyter notebooks, I had just made it point to start at the fast.ai directory by changing the jupyter-config file…

balnazzar · January 29, 2018, 5:21am

Thanks @FourMoBro and @ecdrid.

I cannot really figure out why the heck my pytorch does not find cuda. TF works with it flawlessly, and even pytorch installed with my “old instructions” did find it.

Can you confirm that you called torch.cuda.is_available() and it responded True?

Another thing: how did you install the ipykernel for fastai env?

Thanks.

balnazzar · January 29, 2018, 5:54am

In my darkest hour (), I installed cuda 9.

Now pytorch finds Cuda.

Uncanny.

Now I have to install TF 1.5… Heck of a day…

ecdrid · January 29, 2018, 6:35am

As I told you , I haven’t used the environment file provided by fast.ai…

I had created everything at one place (the local environment of Anaconda 3)

pip3 install ipython

jeremy · January 29, 2018, 1:50pm

environment.yml installs a conda package of cudnn and cuda90, so in theory your installed versions (if any) shouldn’t matter. If they do, it suggests you have something in your path ahead of anaconda - which means perhaps you aren’t using the choosing “Anaconda Prompt” from the start menu (since I believe that sets Anaconda’s path to be the first ones).

Note that you don’t have to do anything special to choose a kernel if you install using the method shown in this thread.

balnazzar · January 29, 2018, 3:15pm

I was, like I am now. Still, it required systemwide installation of cuda 9 to get pytorch able to find cuda. Possibly, something was not right with the whole organization of environment vars, and cuda installation straightened them up…!

I noticed it now. It works with my plain python 3 kernel.

Thanks guys, you have have been very supportive, beginning from you, Jeremy!

How did you do that? Just for the sake of curiosity. Manually installed every single package listed in the .yml within base env?

ecdrid · January 29, 2018, 4:18pm

Just a few…(Anaconda does the hard work to create an environment…)

PyTorch
Keras
Tensorflow
Bcolz
Graphviz
Spacy

And then when I type jupyter notebook in the Anaconda terminal,
It points to directory where I keep all my notebooks…

Simple And Smooth

balnazzar · January 29, 2018, 5:27pm

Cool. I gather you installed fastai with pip, right?

hafidz · January 29, 2018, 5:29pm

Ran the above instructions from scratch, with CUDA 9.0 installed.

Currently going through Lesson 1. No hiccups so far

Update (30/1/2018): No problems on Lesson 1.

balnazzar · January 29, 2018, 6:13pm

Uhm, strange things keep happening.

Pytorch uses the gpu the first time I run the model (resnet34). Then, it uses the CPU (you can tell just because it’s kind of 50x slower, no messages are displayed).

It begins to use the gpu again only if I reboot, but just for a run. Then it falls back to the cpu.

AFAIK there is no function in pytorch to perform some gpu reset. Regardless, any idea about what causes that weird behaviour? Note that pytorch finds cuda and cudnn even when it falls back to the cpu.

miguel_perez · January 29, 2018, 6:30pm

That is great news! Very convenient to have this Windows option for fast.ai for quick local prototyping.

Just tested it a bit, Lesson 1 works smoothly for me, then tried lesson 5, collab. filtering and got error:

ecdrid · January 29, 2018, 6:37pm

Nopes…

GitHub repository is cloned in a seperate directory…(rather they are drive aparts…)

balnazzar · January 29, 2018, 7:29pm

Everything seems to be ok, and still, pytorch uses the GPU just one time after every reboot, then falls back to the CPU.

FourMoBro · January 29, 2018, 7:50pm

what makes you think that it is using CPU over GPU? What times are you seeing for the learn.fit operations in lesson1 during data augmentation (3 epochs) and the fine tuning (7 epochs)?

balnazzar · January 29, 2018, 8:00pm

During what I think to be GPU operation, it takes a fraction of second per iteration, while when using the cpu it takes an average of 1.8 secs per iteration. The entire process was taking a few seconds per epoch on the gpu, minutes on the cpu.

What about the wall time on your system, and what GPU are you using?

jeremy · January 29, 2018, 8:06pm

That should be fixed now. Sorry about that.

miguel_perez · January 29, 2018, 8:44pm

Wow that was fast! Thank you @jeremy, confirmed, it works like a charm now.

FourMoBro · January 29, 2018, 8:49pm

The numbers that you show appear to be from a different area of the notebook than what I initially asked. I am more interested in the epochs later in the notebook in the corresponding sections of Data augmentation, and then fine tuning. I logged my results in this thread.

Chris_Palmer · January 29, 2018, 10:08pm

I get the following experience when trying to get the spacy en installed - anyone had this and know how to work around it?

(D:\Anaconda3) D:\>python -m spacy download en
Collecting https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)
    100% |████████████████████████████████| 37.4MB 1.4MB/s
  Requirement already satisfied (use --upgrade to upgrade): en-core-web-sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz in d:\anaconda3\lib\site-packages
Requirement already satisfied: spacy>=2.0.0a18 in d:\anaconda3\lib\site-packages (from en-core-web-sm==2.0.0)
Requirement already satisfied: numpy>=1.7 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: murmurhash<0.29,>=0.28 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: cymem<1.32,>=1.30 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Collecting thinc<6.11.0,>=6.10.1 (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Exception:
Traceback (most recent call last):
  File "D:\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "D:\Anaconda3\lib\site-packages\pip\commands\install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "D:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "D:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 554, in _prepare_file
    require_hashes
  File "D:\Anaconda3\lib\site-packages\pip\req\req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "D:\Anaconda3\lib\site-packages\pip\index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "D:\Anaconda3\lib\site-packages\pip\index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "D:\Anaconda3\lib\site-packages\pip\index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "D:\Anaconda3\lib\site-packages\pip\index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "D:\Anaconda3\lib\site-packages\pip\index.py", line 811, in get_page
    inst = cls(resp.content, resp.url, resp.headers)
  File "D:\Anaconda3\lib\site-packages\pip\index.py", line 731, in __init__
    namespaceHTMLElements=False,
TypeError: parse() got an unexpected keyword argument 'transport_encoding'

jeremy · January 29, 2018, 10:12pm

Looks like your anaconda isn’t installed in your home dir. That may cause permissions problems - at the very least, it’s untested.