How to set up Windows 10 for fast.ai


(Andrea de Luca) #1

NOTE: this thread id obsolete. Use the following, instead:

Howto: installation on Windows

Setting up Windows 10 for fast.ai part1v2

Important Note for Beginners: Before you venture in a windows installation, take a look at post #64 and #65 below. Always prefer python functions to shell commands in your notebooks.

New Installation Instructions (updated Jan 26, 2018)

The following instructions are somewhat easier than the preceeding ones, but they have not been tested on windows 10 yet. Should an installation attempt fail, fall back to the older instructions below.
Update: They have actually been successfully tested by @jeremy in person.

A) This tutorial assumes you already have a functional anaconda installation.
Moreover, if you don’t have git installed on your machine, install it by following these instructions: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

B) Now, move into a directory where you are comfortable installing the Fastai repo, with its libraries and required packages.
Now you got to clone that repo as follows:

> git clone https://github.com/fastai/fastai

C) Once the cloning process finishes, be sure to be in the directory created by git for the Fastai repository, and type:

> conda env create -f environment-win.yml

followed by:

> activate fastai

This will install the required packages and their dependencies into the fastai environment by leveraging the file environment-win.yml prepared by Fast.ai.

D) Now, open jupyter:

(fastai)> jupyter notebook

and create a new notebook (or open one of the lessons’ notebooks). Check if an appropriate kernel has been installed by the preceeding command(s). If you find out you don’t have any kernel other than the base one (usually named python 3.6), you got to install it manually:

(fastai)>python -m ipykernel install --user --name fastai --display-name "fastAI custom"

Once you open jupyter, you will select fastAI custom from the available kernels.

That would be all.
Remind that you will probably need to edit some lessons’ notebooks code, or at least the parts where you interact with the filesystem from inside the notebook.
Indeed, Fastai notebooks are written down aiming at linux systems, so if you call, for example, ls -la | head, it won’t work (quite obviously) on Windows.

Accessing the workstation remotely:
one may want to log in into such workstation remotely. In that case, there are various options, for example:

  1. Using remote desktop (mind that it’s slow if you have a slow or high-latency connection)
  2. Just leaving the notebook server on (unpractical: you would not be able to run administration tasks)
  3. Using the Windows Subsystem for Linux, AKA “linux on windows” with tmux (install LoW from windows store)
  4. Using an ssh server (which, contrarily to popular belief, is natively supported by Windows, although a bit clumsy to get working. Google for it).

Windows 10: Old Installation Instructions.

Use the following instructions just in case something goes awry with the preceeding ones.

a) Create a new anaconda env with python 3.6 and some additional packages:

(base)>conda create -n fastai python=3.6 numpy cython statsmodels opencv

This will create a new environment named fastai with some additional useful packages.

b) Install pytorch for cuda. If you already have a keras/TF setup which works only with cuda 8.0, and want to avoid headaches, you would rather install the cuda 8.0 version:

(base)>activate fastai

(fastai)>conda install -c peterjc123 pytorch cuda80

If you don’t care about Tensorflow, then you’ll be fine with cuda 9.0. In that case:

(fastai)>conda install -c peterjc123 pytorch cuda90

However, keep us posted should you manage a successful installation of the cuda9 version alongside TF and cuda8

c) Install fastai library

(fastai)>pip install fastai

be patient, that will install a lot of stuff.

d) Now you are done, but you may want to install the ipython kernel in order to use lessons’ notebooks and do your own experiments. In my case:

(fastai)>python -m ipykernel install --user --name fastai --display-name "fastAI custom"

That would be all.

Now, one may want to log in into such workstation remotely. In that case, there are various options, for example:

  1. Using remote desktop (mind that it’s slow if you have a slow or high-latency connection)
  2. Just leaving the notebook server on (unpractical: you would not be able to run administration tasks)
  3. Using the Windows Subsystem for Linux, AKA “linux on windows” with tmux (install LoW from windows store)
  4. Using an ssh server (which, contrarily to popular belief, is natively supported by Windows, although a bit clumsy to get working. Google for it).

Hope this helps.

Additional notes:

Some users report various issues, which I will try to address as I hear of them.

  1. bcolz: Shuld you get an error related to package bcolz, it’s because it was installed via pip when you installed fastai library, and the pip version doesn’t quite match. Remove it by pip uninstall bcolz, then install it by conda install bcolz. Quick test: python -c "import bcolz"

  2. torchvision: Should you get import errors related to torchvision, install it with pip: pip install torchvision.

  3. Should you get errors related to PyYAML, remove it if you installed it with conda, then reinstall it with pip: pip install PyYAML. Mind that another, different, package named pyaml do exist. Furthermore, the conda package pyyaml (all lowercase) did produce errors on my machine, but you can try it if you want. Obviously, do not install pip PyYAML alongside conda pyyaml.

  4. (important) Once you start a notebook, don’t forget selecting the right kernel from inside Jupyter, otherwise you’ll get a ton of import errors. In the example above, the kernel we would select is “fastAI custom”.


Part 1 FAQ
Another treat! Early access to Intro To Machine Learning videos
Howto: installation on Windows
(Richard Fortenberry) #2

Hey Balnazzar, I appreciate you taking the time.

I installed keras with conda install keras.

Then I started with your instruction.

I copied the commands exactly, they all seemed to work.

Then I tried to run the notebook and I got the error:

No module named ‘keras’:

Before that (not sure how), I had gotten keras to work, but then was unable to instantiate vgg16 with error:

NameError: name ‘Vgg16’ is not defined

And tried to troubleshoot with this link unsuccessfully:

Honestly this community is soo awesome, and I really appreciate all the help! I’m honestly at a loss at what to do, this marks the 10th hour I’ve spent trying to configure this environment. I thought I’d be able to teach myself (being that I’ve developed other apps and games) but I’m so confused. If you have any insight please let me know. Otherwise I’ll try and find a different way! Thanks again Balnazzar, I wish you much success!


(Aditya) #3

I guess the problem is that you have created the notebook in the wrong place?

Also did you installed Keras in the same environment as you are currently working in?


(Andrea de Luca) #4

@ricardomang: no problem.

I suggest to start creating different envs for keras/TF/whatever and for fastai/pytorch. You will be able to run experiments with both fastai library and keras by just switching kernels from inside a jupyter notebook.

Please create a fresh anaconda env by following the above instructions, and report your results. I’m farly confident you will be able to run the first lesson OOB. :wink:


#5

There are instructions here too which might be useful. They use CUDA 9.0.


(Andrea de Luca) #7

Those instructions are far more detailed than mine. Thanks! :slight_smile:


Making your own server
#8

Hi,

I have followed the instructions here until step d) and i am unsure what to do hereafter.
I tried to launch jupyter and got the following error message.

Question:

  1. Do i have to download fastai-master from github? Or is the pip install library part installing all the materials required?
  2. if 1) is a yes, where to i put the folder ? I presume that is the root of the problem below.

Apologies in advance for the very basic question asked.


ModuleNotFoundError Traceback (most recent call last)
in ()
1 # This file contains all the main external libs we’ll use
----> 2 from fastai.imports import *

ModuleNotFoundError: No module named ‘fastai’

In [ ]:


(Dominique Gilleman) #9

Hi SalmonSoba,

Here are instructions to install on windows 10:
DeepLearningEssentials - installation
just be careful which version (cuda or not) of pytorch

with pip fastai installation:
in C:\Anaconda\envs[EnvironmentName]\Lib\site-packages\fastai
in file model.py

replace loss = raw_loss = self.crit(output, y) by loss = raw_loss = self.crit(output, y.long())
replace self.crit(preds,y) by self.crit(preds,y.long())

Some tutorials works with pytorch 0.2, as Cats & Dogs
Some with pytorch 0.3
So i did 2 environments, one with pytorch 0.2, one with pytorch 0.3


#10

Thanks Arghz for the response. I believe i have made all the installations.

What i have now for folder directories is this:
C:\ProgramData\Anaconda3
C:\Users\Username\AppData\Local\conda\conda\envs\fastai
C:\Users\Username\fastai-master

From where should i open C:\Users\Username\fastai-master\courses\dl1\lesson1.ipynb ?

I tried Anaconda prompt > Activate fastai > Jupyter notebook.
It launches the notebook interface.

Thereafter, i went here: http://localhost:8889/notebooks/fastai-master/courses/dl1/lesson1.ipynb

And i got this:


(Andrea de Luca) #11

@SalmonSoba

I see your kernel above is named python 3.

I imagine, however, that you installed fastai in a dedicated env. What is its name? Have you activated it? Was the ipython kernel installation successful?

Since you said “I have followed the instructions here until step d)”, I assume that step d) was performed successfully. You have to activate your_env_name, then you have to run jupyter from here, and switch kernel to (in my example) fastAI custom.

Then you can run the lesson notebook.

Keep us posted, mate :slight_smile:


(Andrea de Luca) #12

I see some issues:

  1. In torchvision installation instruction, Pip is written with a capital P. I think it won’t work in OSX.
  2. You will end up with two versions of bcolz, one installed with conda, the other by fastai, in the same env.
  3. I think you should explicitly add some instruction for GPU version of pytorch, since a beginner will just copy & paste your instructions no matter whether he has a GPU or not.
  4. Instructions for creating a separate env should be provided, otherwise a beginner will install everything in the base env, which is generally not advisable.

#13

@ balnazzar
Thanks for the response.

I have activated fastai. Ipython kernel installation was successful.

Though i must admit that during the installation process at step C), I had problem with not having Microsoft Visual C++ 14.0. I went to install Microsoft Visual C++ 14.0, and continued the process without activate fastai in the first go.

But i realised that and did activate fastai before pip install fastai again.

I have to be frank that i took the (fastai)>python -m ipykernel install --user --name fastai --display-name "fastAI custom"
as it is. Am i supposed to change the user, name and display-name part ?


(Andrea de Luca) #14

Uhm, pytorch by peterjc123 should have installed the required VC runtimes.
Do a conda list to be sure. The runtimes should be there. Then call a python interpreter and do an import torch. If it gets correctly imported, you are ok.

no, if your new env is named fastai you are ok with copying and pasting it (but just the part that comes after the > sign).

Now, you have to select the fastAI custom kernel from inside jupyter, because the base kernel knows nothing about fastai.
Do it by switching kernel in the Kernel menu, between Cell and Widgets, top of the page.

I’m going to get some sleep, but I’ll answer tomorrow morning, should you get further issues! :wink:

Note:

It doesn’t matter. Just be sure to activate the right env, and then select the right kernel from inside Jupyter. You can then navigate the directory tree from inside jupyter.


#15

Hi Balnazzar… Thanks for taking the time to make the instructions for Windows! Everything went smooth and now I have the environment set up on my local PC.
However, the notebook lesson1.ipynb in courses/dl1 contains Linus specific commands like “!ls {PATH}” or “img = plt.imread(f’{PATH}valid/cats/{files[0]}’)”.
I’m not sure how to adapt these to Windows. Have you done that and can you share it with us?
M.


(Andrea de Luca) #16

Begin with dir in place of ls. Furthermore, should / not work, try with \.
Tomorrow morning I’ll provide more detailed instructions. In the meantime, try and do some experiments.


#17

Replacing ls with dir and / with \ works. But files = !dir {PATH}valid\cats | head doesn’t. (Head is not recognized as an internal or external command …)
I assume this code line puts all the filenames from {PATH}valid\cats in a Python list, but I don’t know how to do this in Windows.


#18

@balnazzar

Thanks for the detailed step by step clarification. It works now.
Main problem i had was not knowing how to switch the kernel to custom previously.

Thereafter, i had problem with the torchvision. Followed the additional notes and things worked.

Onward to the lecture ! : )


(Andrea de Luca) #19

Windows hasn’t an exact equivalent for head/tail commands. However, you got the more native windows CLI command.
Search for it: with proper flags, you can emulate Unix’ head.

You have other options too (e.g. the windows subsystem for linux, etc), but first of all you should learn how to work with filenames and the OS in python/numpy. Dig into the documentation, you will discover a ton of useful features.


#20

Windows Subsystem for Linux doesn’t support Cuda, so that’s not an option.
Frankly, I don’t understand why Fast.ai uses platform dependent shell commands and not independent pure python commands for accessing files. Now I’m putting more time in solving os problems than learning ML.


(Andrea de Luca) #21

You wouldn’t use it for cuda operations. Rather, you would perform filesystem operations into a separate notebook (you can read/write windows files from inside WSL).

AFAIK, no such things do exist (again: AFAIK). The os library, for example, is platform dependent, so I think they don’t really have a choice.
But you could ask @jeremy for additional informations.

In the meantime, don’t let such trifles distract you. More often than not, in data science and AI you will be forced to work out tricks here and there to get things straight. Get used to it.