Howto: installation on Windows

bsalita · February 17, 2018, 9:23am

I use NGrok to workaround firewalls and serve Jupyter using a domain name. Not sure if it applies to your situation.

Jonas · February 17, 2018, 8:52pm

Hi!

this is step-by-step what I did to get lesson 1 running (fastai\courses\dl1\lesson1.ipynb) on my Windows 7 machine (HP ZBook 15 G3), which features an NVIDIA Quadro M2000M with 4 GB.

Thank you for all the help in this thread, and especially to @jeremy. This post is basically a collection of what I already learned here.

You must have an NVIDIA GPU and have installed the NVIDIA drivers.
(I did install Cuda and Cudnn but it should not be necessary ! In order to install Cudnn I had to register for the NVIDIA Developer account, and I also needed to install VisualStudio which seems unnecessary. Meanwhile I have deinstalled VisualStudio again)

Install Anaconda (Python 3.6)
Choose the option to install for “Just Me”
Install it in a Windows directory where the path does not contain any spaces or other special characters.
I installed under the default directory, which is
C:\Users\[my_user_name]\AppData\Local\Continuum\anaconda3
Install Git for Windows
Clone fastai from GitHub
Run Anaconda Prompt from your start menu, and type:
git clone https://github.com/fastai/fastai.git
Setup Python environment / install packages
I was not able to perform the “conda env update” for the reasons described by @prusswan above. In particular I get “The program can’t start because api-ms-win-core-rtlsupport-l1-2-0.dll is missing” error message.
So I installed all required packages manually. This can be done via the Anaconda GUI or via the Anaconda Prompt
Please also use the “base” environment when installing. I had the dll-problem also when doing this process with a new environment.

5.1 Anaconda GUI (more “Windows-like”…)
Start Anaconda Navigator
Click on “Enviroments” and leave the “base” environment selected
Click on “Channels” and define the following channels beside “defaults”:

fastai
pytorch
peterjc123

Select “Not Installed” (packages) and begin searching for and installing (by clicking the box and applying) the following packages:

bcolz
opencv
graphviz
tqdm
pytorch

For the pip packages (not available from conda) you still need to go to the Anaconda Prompt and install them running the pip command:
pip install graphviz
pip install sklearn-pandas
pip install pandas_summary
pip install isoweek
pip install torchvision
pip install torchtext

5.2 Anaconda Prompt
You might as well install everything using the Anaconda Prompt (also here do not create / activate any new environment). Open Anaconda Prompt (normal user) and type:
conda install -y bcolz
conda install -y opencv
conda install -y -c fastai tqdm
conda install -y -c peterjc123 pytorch
conda install -y -c peterjc123 cuda90
conda install -y -c peterjc123 cudnn
pip install graphviz
pip install torchvision
pip install torchtext
pip install sklearn-pandas
pip install pandas_summary
pip install isoweek

Download data for cats and dogs
If you do not have wget for Windows installed and not using cygwin, just do like this:
Open the Windows Explorer and go to your home directory, then change to the fastai directory.
Create a new directory named "data"
Now open your favorite browser and paste
http://files.fast.ai/data/dogscats.zip
as URL
Download the file and move it to your new data directory
Unzip the file with the menu entry “Extract Here”
Create symbolic links
Open Anaconda Prompt as Administrator (necessary for this step): "Run as administrator"
You land in the Windows system32 directory so you need to change to your home directory, e.g.
cd C:\Users\[my_user_name]
and then:
cd \fastai\courses\dl1
You may need to remove an existing symbolic link file from the GitHub repository first:
del fastai
now:
mklink /d fastai ..\..\fastai
and:
mklink /d data ..\..\data
Launch Jupyter Notebook
In Anaconda Navigator go back to "Home"
Under Jupyter Notebook select “Launch”.
Click through to courses and then dl1 and try lesson1.ipynb.
I haven’t tried anything else than this notebook so it might be necessary to install further packages when proceeding through the lessons.
I got it all running and
torch.cuda.is_available()
as well as
torch.backends.cudnn.enabled
return True

Jonas · February 18, 2018, 3:33pm

UPDATE: running the same experiment a few times it seems GPU is being used after all! It is only used 6 - 8 % whereas CPUs are going at 100 %, but it is used…

I am using GPU-Z tool tool to monitor GPU activity and I had the same impression, that GPU seems not to being used…

Queries continue returning expected result though:
torch.cuda.is_available()
True
torch.backends.cudnn.enabled
True
torch.cuda.device_count()
1
torch.cuda.get_device_name(0)
’Quadro M2000M’

beezerlm · February 18, 2018, 5:03pm

I have also noticed the process reserves memory on the gpu, but it seems odd that it cranks the CPU and hardly utilizes the GPU. In the first series of courses (pre-2018) running cats/dogs, my GPU was maxed while my CPU was relatively low/stable. I could literally feel the heat coming from the tower. I tried to re-run my old notebooks for comparison but was riddled with errors because it seems my libraries have been updated.

Jonas · February 18, 2018, 5:41pm

agree @beezerlm hope someone has an explanation.

beezerlm · February 18, 2018, 9:53pm

Looking at This thread by FourMoBro, My elapsed time for learn.fit operations seems to be inline with the others.

3 epoch - 04:07
7 epoch - 14:54

Maybe high CPU / low GPU usage is by design for these operations?

Moody · February 19, 2018, 5:12am

Hi @Roger, I got the same error message while the GPU is working properly. Could you step me through how to sort it? Thanks in advance.

ModuleNotFoundError: No module named 'fastai.imports'

Brad_S · February 19, 2018, 5:52am

Hi Sarada

it means your step 8 from Jeremy’s post isn’t working,
i.e. the link / symbolic link isn’t finding fastai

figure out your working directory.
go there - is there a link to “fastai” and does clicking it take you to your fastai install directory

I’m guessing not and maybe slight edits of the below will get it to work

hth

Jonas · February 19, 2018, 11:58am

I have ~14 s/it with my 32 GB main memory, 8 CPUs, and 4 GB GPU

Kasianenko · February 20, 2018, 7:14pm

Hello Jeremy,
And thank you for this detailed guide. I’ve read all 99 posts in this topic and spent several hours to make it run on my Win10 with GeForce 960m.
According to comments it is required to have CUDA. It failed running torch.cuda.is_available() with CUDA v.7.5 and now it’s running with v.9.1
My humble suggestion is to add CUDA v.9 to first post.
Second question: In the lecture video you told how to reboot remote machine to free up GPU memory. Is there any trick similar to this on Windows machine?
P.S. I’m so happy to run it locally, without cloud computing and creating several accounts and connecting credit cards to every account, that I’m happy to contribute to this topic.

hafidz · February 20, 2018, 7:35pm

I think the idea of rebooting the remote machine is simply an act to restart the instance. If you’re running it on Windows I don’t imagine that you literally need to restart the PC. A mere restart of the conda instance will work I assume.

Roger · February 21, 2018, 3:07am

Hi,

I’m trying to get the NLP notebook in lesson 4 (dl1/lesson4-imdb.ipynb) to work but got

Can’t find model ‘en’.

I found the post about installing spacy and then downloading en (results below).

However, it results in the error

Creating a shortcut link for ‘en’ didn’t work

Can someone let me know if there’s a simple fix for this in Windows?

(fastai) C:\Local\fastai\fastai>python -m spacy download en
Collecting https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)
100% |████████████████████████████████| 37.4MB 51.1MB/s
Requirement already satisfied: spacy>=2.0.0a18 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from en-core-web-sm==2.0.0)
Requirement already satisfied: numpy>=1.7 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: murmurhash<0.29,>=0.28 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: cymem<1.32,>=1.30 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: thinc<6.11.0,>=6.10.1 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: six in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: pathlib in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: ujson>=1.35 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: dill<0.3,>=0.2 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: regex==2017.4.5 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: ftfy<5.0.0,>=4.4.2 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: msgpack-python in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: msgpack-numpy==0.4.1 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: wrapt in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from thinc<6.11.0,>=6.10.1->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from thinc<6.11.0,>=6.10.1->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: cytoolz<0.9,>=0.8 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from thinc<6.11.0,>=6.10.1->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: termcolor in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from thinc<6.11.0,>=6.10.1->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: pyreadline>=1.7.1 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from dill<0.3,>=0.2->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: idna<2.7,>=2.5 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: html5lib in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from ftfy<5.0.0,>=4.4.2->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: wcwidth in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from ftfy<5.0.0,>=4.4.2->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: toolz>=0.8.0 in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from cytoolz<0.9,>=0.8->thinc<6.11.0,>=6.10.1->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: webencodings in c:\users\rh_la\appdata\local\conda\conda\envs\fastai\lib\site-packages (from html5lib->ftfy<5.0.0,>=4.4.2->spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Installing collected packages: en-core-web-sm
Running setup.py install for en-core-web-sm … done
Successfully installed en-core-web-sm-2.0.0

Error: Couldn't link model to 'en'
Creating a symlink in spacy/data failed. Make sure you have the required
permissions and try re-running the command as admin, or use a
virtualenv. You can still import the model as a module and call its
load() method, or create the symlink manually.

C:\Users\rh_la\AppData\Local\conda\conda\envs\fastai\lib\site-packages\en_core_web_sm
-->
C:\Users\rh_la\AppData\Local\conda\conda\envs\fastai\lib\site-packages\spacy\data\en


Download successful
Creating a shortcut link for 'en' didn't work (maybe you don't have
admin permissions?), but you can still load the model via its full
package name:

nlp = spacy.load('en_core_web_sm')

Brad_S · February 21, 2018, 11:42am

when you ran this, were you running as an administrator?

Roger · February 21, 2018, 6:45pm

Thanks, Brad. That was the problem.

gigaom · February 22, 2018, 7:14pm

I get this error when I execute below lines in notebook (Windows)

from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

`C:\Program Files\Anaconda3\envs\fastai\lib\site-packages\torch_init_.py in ()
74 pass
75
—> 76 from torch._C import *
77
78 all += [name for name in dir(_C)

ImportError: DLL load failed: The specified module could not be found.`

C:\Users\wecando>conda info
active environment : None
user config file : C:\Users\wecando.condarc
populated config files : C:\Users\wecando.condarc
conda version : 4.4.10
conda-build version : not installed
python version : 3.6.0.final.0
base environment : C:\Program Files\Anaconda3 (read only)
channel URLs : https://conda.anaconda.org/anaconda-fusion/win-64
https://conda.anaconda.org/anaconda-fusion/noarch
https://repo.continuum.io/pkgs/main/win-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/win-64
https://repo.continuum.io/pkgs/free/noarch
https://repo.continuum.io/pkgs/r/win-64
https://repo.continuum.io/pkgs/r/noarch
https://repo.continuum.io/pkgs/pro/win-64
https://repo.continuum.io/pkgs/pro/noarch
https://repo.continuum.io/pkgs/msys2/win-64
https://repo.continuum.io/pkgs/msys2/noarch
package cache : C:\Program Files\Anaconda3\pkgs
C:\Users\wecando\AppData\Local\conda\conda\pkgs
envs directories : C:\Users\wecando\AppData\Local\conda\conda\envs
C:\Program Files\Anaconda3\envs
C:\Users\wecando.conda\envs
platform : win-64
user-agent : conda/4.4.10 requests/2.12.4 CPython/3.6.0 Windows/10 Windows/10.0.16299

What I am missing?

nyxynyx · February 24, 2018, 5:18am

When doing step 4 conda env update, I get the error

Using Anaconda API: https://api.anaconda.org
Fetching package metadata ...............
Solving package specifications: .
NoPackagesFoundError: Packages missing in current win-64 channels:
  - spacy
  - expat
  - libxml2
  - pexpect
  - xz

Running

Windows 10 (64 bit)
Python 3.6.2 :: Continuum Analytics, Inc.
conda 4.2.9
Nvidia 1080

What went wrong?

Moody · February 25, 2018, 8:22am

Thanks @Brad_S. I re-did everything but skipped the following two lines. Then, it works.

del fastai
mklink /d fastai ..\..\fastai

Moody · February 25, 2018, 2:18pm

Under the following directory, C:\Users\rh_la\AppData\Local\conda\conda\envs\fastai\lib\site-packages\ to create symbolic link as below:

mklink /d en_core_web_sm C:\Users\rh_la\AppData\Local\conda\conda\envs\fastai\lib\site-packages\spacy\data\en

Moody · February 27, 2018, 4:09am

Side note: It is recommended to add the Jupyter Notebook Extensions, especially Gist-it and Collapsible Headings. Finally, I can replicate the whole environment in Windows.

Leo_Zhou · February 28, 2018, 6:14am

After installing with conda on my notebotebook, I had the same problem.

torch.cuda.is_available() and torch._C._cuda_isDriverSufficient() both return False.

After spending a lot of time diagnosing, I believe it could be related to the Nvidia driver

cuda9 was released not long ago, and it requires a quite newer version of Nvidia driver. Upgrading the driver works for me.