How to set up Windows 10 for fast.ai

balnazzar · January 12, 2018, 12:04pm

NOTE: this thread id obsolete. Use the following, instead:

Howto: installation on Windows

Setting up Windows 10 for fast.ai part1v2

Important Note for Beginners: Before you venture in a windows installation, take a look at post #64 and #65 below. Always prefer python functions to shell commands in your notebooks.

New Installation Instructions (updated Jan 26, 2018)

The following instructions are somewhat easier than the preceeding ones, but they have not been tested on windows 10 yet. Should an installation attempt fail, fall back to the older instructions below.
Update: They have actually been successfully tested by @jeremy in person.

A) This tutorial assumes you already have a functional anaconda installation.
Moreover, if you don’t have git installed on your machine, install it by following these instructions: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

B) Now, move into a directory where you are comfortable installing the Fastai repo, with its libraries and required packages.
Now you got to clone that repo as follows:

> git clone https://github.com/fastai/fastai

C) Once the cloning process finishes, be sure to be in the directory created by git for the Fastai repository, and type:

> conda env create -f environment-win.yml

followed by:

> activate fastai

This will install the required packages and their dependencies into the fastai environment by leveraging the file environment-win.yml prepared by Fast.ai.

D) Now, open jupyter:

(fastai)> jupyter notebook

and create a new notebook (or open one of the lessons’ notebooks). Check if an appropriate kernel has been installed by the preceeding command(s). If you find out you don’t have any kernel other than the base one (usually named python 3.6), you got to install it manually:

(fastai)>python -m ipykernel install --user --name fastai --display-name "fastAI custom"

Once you open jupyter, you will select fastAI custom from the available kernels.

That would be all.
Remind that you will probably need to edit some lessons’ notebooks code, or at least the parts where you interact with the filesystem from inside the notebook.
Indeed, Fastai notebooks are written down aiming at linux systems, so if you call, for example, ls -la | head, it won’t work (quite obviously) on Windows.

Accessing the workstation remotely:
one may want to log in into such workstation remotely. In that case, there are various options, for example:

Using remote desktop (mind that it’s slow if you have a slow or high-latency connection)
Just leaving the notebook server on (unpractical: you would not be able to run administration tasks)
Using the Windows Subsystem for Linux, AKA “linux on windows” with tmux (install LoW from windows store)
Using an ssh server (which, contrarily to popular belief, is natively supported by Windows, although a bit clumsy to get working. Google for it).

Windows 10: Old Installation Instructions.

Use the following instructions just in case something goes awry with the preceeding ones.

a) Create a new anaconda env with python 3.6 and some additional packages:

(base)>conda create -n fastai python=3.6 numpy cython statsmodels opencv

This will create a new environment named fastai with some additional useful packages.

b) Install pytorch for cuda. If you already have a keras/TF setup which works only with cuda 8.0, and want to avoid headaches, you would rather install the cuda 8.0 version:

(base)>activate fastai

(fastai)>conda install -c peterjc123 pytorch cuda80

If you don’t care about Tensorflow, then you’ll be fine with cuda 9.0. In that case:

(fastai)>conda install -c peterjc123 pytorch cuda90

However, keep us posted should you manage a successful installation of the cuda9 version alongside TF and cuda8

c) Install fastai library

(fastai)>pip install fastai

be patient, that will install a lot of stuff.

d) Now you are done, but you may want to install the ipython kernel in order to use lessons’ notebooks and do your own experiments. In my case:

(fastai)>python -m ipykernel install --user --name fastai --display-name "fastAI custom"

That would be all.

Now, one may want to log in into such workstation remotely. In that case, there are various options, for example:

Using remote desktop (mind that it’s slow if you have a slow or high-latency connection)
Just leaving the notebook server on (unpractical: you would not be able to run administration tasks)
Using the Windows Subsystem for Linux, AKA “linux on windows” with tmux (install LoW from windows store)
Using an ssh server (which, contrarily to popular belief, is natively supported by Windows, although a bit clumsy to get working. Google for it).

Hope this helps.

Additional notes:

Some users report various issues, which I will try to address as I hear of them.

bcolz: Shuld you get an error related to package bcolz, it’s because it was installed via pip when you installed fastai library, and the pip version doesn’t quite match. Remove it by pip uninstall bcolz, then install it by conda install bcolz. Quick test: python -c "import bcolz"
torchvision: Should you get import errors related to torchvision, install it with pip: pip install torchvision.
Should you get errors related to PyYAML, remove it if you installed it with conda, then reinstall it with pip: pip install PyYAML. Mind that another, different, package named pyaml do exist. Furthermore, the conda package pyyaml (all lowercase) did produce errors on my machine, but you can try it if you want. Obviously, do not install pip PyYAML alongside conda pyyaml.
(important) Once you start a notebook, don’t forget selecting the right kernel from inside Jupyter, otherwise you’ll get a ton of import errors. In the example above, the kernel we would select is “fastAI custom”.

ricardomang · January 13, 2018, 8:22am

Hey Balnazzar, I appreciate you taking the time.

I installed keras with conda install keras.

Then I started with your instruction.

I copied the commands exactly, they all seemed to work.

Then I tried to run the notebook and I got the error:

No module named ‘keras’:

Before that (not sure how), I had gotten keras to work, but then was unable to instantiate vgg16 with error:

NameError: name ‘Vgg16’ is not defined

And tried to troubleshoot with this link unsuccessfully:

Honestly this community is soo awesome, and I really appreciate all the help! I’m honestly at a loss at what to do, this marks the 10th hour I’ve spent trying to configure this environment. I thought I’d be able to teach myself (being that I’ve developed other apps and games) but I’m so confused. If you have any insight please let me know. Otherwise I’ll try and find a different way! Thanks again Balnazzar, I wish you much success!

ecdrid · January 13, 2018, 8:45am

I guess the problem is that you have created the notebook in the wrong place?

Also did you installed Keras in the same environment as you are currently working in?

balnazzar · January 13, 2018, 9:38am

@ricardomang: no problem.

I suggest to start creating different envs for keras/TF/whatever and for fastai/pytorch. You will be able to run experiments with both fastai library and keras by just switching kernels from inside a jupyter notebook.

Please create a fresh anaconda env by following the above instructions, and report your results. I’m farly confident you will be able to run the first lesson OOB.

pekoto · January 14, 2018, 4:15am

There are instructions here too which might be useful. They use CUDA 9.0.

github.com

cklukas/fastai/blob/master/doc/windows_setup.md

# Installation under Windows 10

## Why Windows?
Well, if you have the possibility to just use Linux, work in a dual-boot configuration or have an GPU-equipped Linux server available, thats clearly 
the preferred way to work with pytorch and is currently also the preferred environment for developing deep learning solutions. Nevertheless, Windows 10 is
a big step-up for developers and users, supports multiple desktops, the Linux subsystem, and more. So if you don't want to split your hard disk space
for a dual-boot configuration or don't know Linux well, this document may be useful to you.
## Ubuntu Linux subsystem?
Windows 10 Professional (version 1709) makes it possible to activate a natively running Linux sub-system directly within Windows (Ubuntu Linux). 
Under this sub-system python and pytorch and thus the whole fastai framework (including Jupyter server etc.) may be installed and used. 

Unfortunately, it is currently not possible to utilize GPU acceleration under this environment. 
Therefore, the Linux subsystem is not recommended to be used. The training speed would in most cases be too slow for practical work.

If you still would like to use this environment, first activate the subsystem, then install Ubuntu from the store.
Then you can proceed by cloning the git repo, installing python 3.6 and then by installing all 
required Linux and python packages as normal. If you get a UTF-8 error during the PyYAML installation (required for Keras), 
you need to install that python package separately (the package is not yet adapted for Python 3.6). Before installing the PyYAML package, you need to enter a command to 
switch the current code page (enter 'chcp 1252'). After issuing that command the install should work.

This file has been truncated. show original

balnazzar · January 15, 2018, 1:05pm

Those instructions are far more detailed than mine. Thanks!

SalmonSoba · January 24, 2018, 3:21pm

Hi,

I have followed the instructions here until step d) and i am unsure what to do hereafter.
I tried to launch jupyter and got the following error message.

Question:

Do i have to download fastai-master from github? Or is the pip install library part installing all the materials required?
if 1) is a yes, where to i put the folder ? I presume that is the root of the problem below.

Apologies in advance for the very basic question asked.

ModuleNotFoundError Traceback (most recent call last)
in ()
1 # This file contains all the main external libs we’ll use
----> 2 from fastai.imports import *

ModuleNotFoundError: No module named ‘fastai’

In [ ]:

Arghz · January 24, 2018, 9:01pm

Hi SalmonSoba,

Here are instructions to install on windows 10:
DeepLearningEssentials - installation
just be careful which version (cuda or not) of pytorch

with pip fastai installation:
in C:\Anaconda\envs[EnvironmentName]\Lib\site-packages\fastai
in file model.py

replace loss = raw_loss = self.crit(output, y) by loss = raw_loss = self.crit(output, y.long())
replace self.crit(preds,y) by self.crit(preds,y.long())

Some tutorials works with pytorch 0.2, as Cats & Dogs
Some with pytorch 0.3
So i did 2 environments, one with pytorch 0.2, one with pytorch 0.3

SalmonSoba · January 24, 2018, 10:47pm

Thanks Arghz for the response. I believe i have made all the installations.

What i have now for folder directories is this:
C:\ProgramData\Anaconda3
C:\Users\Username\AppData\Local\conda\conda\envs\fastai
C:\Users\Username\fastai-master

From where should i open C:\Users\Username\fastai-master\courses\dl1\lesson1.ipynb ?

I tried Anaconda prompt > Activate fastai > Jupyter notebook.
It launches the notebook interface.

Thereafter, i went here: http://localhost:8889/notebooks/fastai-master/courses/dl1/lesson1.ipynb

And i got this:

balnazzar · January 25, 2018, 12:31am

@SalmonSoba

I see your kernel above is named python 3.

I imagine, however, that you installed fastai in a dedicated env. What is its name? Have you activated it? Was the ipython kernel installation successful?

Since you said “I have followed the instructions here until step d)”, I assume that step d) was performed successfully. You have to activate your_env_name, then you have to run jupyter from here, and switch kernel to (in my example) fastAI custom.

Then you can run the lesson notebook.

Keep us posted, mate

balnazzar · January 25, 2018, 12:56am

I see some issues:

In torchvision installation instruction, Pip is written with a capital P. I think it won’t work in OSX.
You will end up with two versions of bcolz, one installed with conda, the other by fastai, in the same env.
I think you should explicitly add some instruction for GPU version of pytorch, since a beginner will just copy & paste your instructions no matter whether he has a GPU or not.
Instructions for creating a separate env should be provided, otherwise a beginner will install everything in the base env, which is generally not advisable.

SalmonSoba · January 25, 2018, 12:56am

@ balnazzar
Thanks for the response.

I have activated fastai. Ipython kernel installation was successful.

Though i must admit that during the installation process at step C), I had problem with not having Microsoft Visual C++ 14.0. I went to install Microsoft Visual C++ 14.0, and continued the process without activate fastai in the first go.

But i realised that and did activate fastai before pip install fastai again.

I have to be frank that i took the (fastai)>python -m ipykernel install --user --name fastai --display-name "fastAI custom"
as it is. Am i supposed to change the user, name and display-name part ?

balnazzar · January 25, 2018, 12:59am

Uhm, pytorch by peterjc123 should have installed the required VC runtimes.
Do a conda list to be sure. The runtimes should be there. Then call a python interpreter and do an import torch. If it gets correctly imported, you are ok.

no, if your new env is named fastai you are ok with copying and pasting it (but just the part that comes after the > sign).

Now, you have to select the fastAI custom kernel from inside jupyter, because the base kernel knows nothing about fastai.
Do it by switching kernel in the Kernel menu, between Cell and Widgets, top of the page.

I’m going to get some sleep, but I’ll answer tomorrow morning, should you get further issues!

Note:

It doesn’t matter. Just be sure to activate the right env, and then select the right kernel from inside Jupyter. You can then navigate the directory tree from inside jupyter.

marc · January 25, 2018, 1:33am

Hi Balnazzar… Thanks for taking the time to make the instructions for Windows! Everything went smooth and now I have the environment set up on my local PC.
However, the notebook lesson1.ipynb in courses/dl1 contains Linus specific commands like “!ls {PATH}” or “img = plt.imread(f’{PATH}valid/cats/{files[0]}’)”.
I’m not sure how to adapt these to Windows. Have you done that and can you share it with us?
M.

balnazzar · January 25, 2018, 1:51am

Begin with dir in place of ls. Furthermore, should / not work, try with \.
Tomorrow morning I’ll provide more detailed instructions. In the meantime, try and do some experiments.

marc · January 25, 2018, 2:53am

Replacing ls with dir and / with \ works. But files = !dir {PATH}valid\cats | head doesn’t. (Head is not recognized as an internal or external command …)
I assume this code line puts all the filenames from {PATH}valid\cats in a Python list, but I don’t know how to do this in Windows.

SalmonSoba · January 25, 2018, 2:03pm

@balnazzar

Thanks for the detailed step by step clarification. It works now.
Main problem i had was not knowing how to switch the kernel to custom previously.

Thereafter, i had problem with the torchvision. Followed the additional notes and things worked.

Onward to the lecture ! : )

balnazzar · January 25, 2018, 5:55pm

Windows hasn’t an exact equivalent for head/tail commands. However, you got the more native windows CLI command.
Search for it: with proper flags, you can emulate Unix’ head.

You have other options too (e.g. the windows subsystem for linux, etc), but first of all you should learn how to work with filenames and the OS in python/numpy. Dig into the documentation, you will discover a ton of useful features.

marc · January 25, 2018, 6:21pm

Windows Subsystem for Linux doesn’t support Cuda, so that’s not an option.
Frankly, I don’t understand why Fast.ai uses platform dependent shell commands and not independent pure python commands for accessing files. Now I’m putting more time in solving os problems than learning ML.

balnazzar · January 25, 2018, 8:15pm

You wouldn’t use it for cuda operations. Rather, you would perform filesystem operations into a separate notebook (you can read/write windows files from inside WSL).

AFAIK, no such things do exist (again: AFAIK). The os library, for example, is platform dependent, so I think they don’t really have a choice.
But you could ask @jeremy for additional informations.

In the meantime, don’t let such trifles distract you. More often than not, in data science and AI you will be forced to work out tricks here and there to get things straight. Get used to it.