Platform: Local Server - Ubuntu

For those on a mac, as of right now you may get an error with pip install sentencepiece. For example, I got

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Fortunately, there is a fix:

pip install https://github.com/google/sentencepiece/releases/download/v0.1.85/tf_sentencepiece-0.1.85-py2.py3-none-macosx_10_10_x86_64.whl

Note: the install was successful, but I haven’t tested the package (yet).

For mac users:

In addition to the other problem I had (with sentencepiece), I had issues with graphviz. In fact, there are two things:

  1. Graphviz, the graph visualisation software, installed via
brew install graphviz

and
2. graphviz, the python package, installed via

pip install graphviz

Somehow the python package knows where to look for the software, but it worked for me.

Thanks for the guide, @FourMoBro. A few updates for the Ubuntu installation. I’m running 18.04 LTS.

pip install azure - no longer works, but doesn’t seem necessary to run the nbs
pip install "ipywidgets >= 7.5.1" - requirement already satisfied
pip install scikit_learn - requirement already satisfied

Otherwise, I haven’t had any issues thus far.

1 Like

I installed fastai2 according to your guide in the top post. Thanks for the clear instructions!

How do I keep the fastai2 that my notebooks use up-to-date? I have been using pip install fastai2 fastcore --upgrade.

The reason I may be confused is that the above command replies
Requirement already satisfied: fastai2 in /home/malcolm/anaconda3/envs/fastai2/lib/python3.7/site-packages (0.0.16)

Yet when in the fastai2 directory, git pull updated many files. Also, I saw in another user’s notebook that they are using fastai2 0.0.17.

To be clear, I do not want to contribute yet to fastai2. I only want to be using the latest released version.

Thanks for your help!

@Pomo, When you run:

pip install fastai2 fastcore --upgrade

You are getting your fastai2 from pypi. That version is 0.0.16, it was released on March 30th 20220.

The reason why your are seeing the following message:

is because, you are trying to install the fastai2 last version found at pypi, and you already have it on your local machine. Therefore, the Requirement already satisfied

As for the other notebook:

In that notebook, they most likely installed the latest version of fastai2 (0.0.17 as of today) using one the 2 following options:

Option 1: From Github - Non editable version
you have to install directly from the fastai2 master branch like this:

pip install git+https://github.com/fastai/fastai2.git

As a good practice, you have to install fastcore at the same time, like this:

pip install git+https://github.com/fastai/fastcore.git

Option 2: From Github - Editable version

  • Installing the fastai2 editable version
git clone https://github.com/fastai/fastai2
cd fastai2
pip install -e .
  • Installing the fastcore editable version
git clone https://github.com/fastai/fastcore
cd fastcore
pip install -e .

Every time, you want to upgrade to latest version of both fastai2 and fastcore, you run (from the corresponding folder):

git pull

This is already a long answer but if you would like to learn more about this subject, I wrote this blog post: 3 ways to pip install a package: fastai2 use-case where you will find a more detailed answer.

4 Likes

Thanks so much! Your blog post explained exactly how to proceed. Now updated to v 0.0.17.

What was confusing was the need to clone the fastai2 repo before creating the new conda environment, while the fastai2 in actual use is coming from pypi. Is that fastai2 clone now irrelevant?

Again, thanks for your willing response. It saved me a lot of frustration.

1 Like

@Pomo, You are very welcome!

The cloning will copy the environment.yml file on your local machine. Then, it’s used when creating the conda virtual environment like this:

conda env create -f environment.yml

If you are not interested in cloning the fastai2 repo, you can create an empty environment.yml text file on your local machine, and copy the content found in the repo and paste in your newly created file, and run the command above. Afterwards, you pip install fastai2.

Even better, I think you can just add fastai2 as a dependency in the environment.yml file, and just run the same command ( without the pip install fastai2 needed here above):

conda env create -f environment.yml
I don't remember trying this last option but I think it should work. I will try it whenever I will have time.

I just tried the option (with the strike-through) mentioned here above, and it doesn’t work because fastai doesn’t have yet a fastai2 package in the Anaconda Repository. Until then, one have to create a virtual environment using the the environment.yml file found in the fastai2 repo, and then pip install fastai2.

Since it has been more than a month, and the post is not a wiki, I cannot edit the first post, but here is a 4/24/20 Update:

With the new release of Ubuntu 20.04 LTS in the past 24 hrs, I decided to rebuild my VM using this new LTS image. I then proceeded to follow my original instructions to see if they still work. Everything works, for the most part. Here is what I found, similar to what others have since posted, for step 9:

  1. Replace pip install graphviz with conda install graphviz or as @bsalita suggested, conda install python-graphviz

  2. DO NOT run pip install azure You will get errors, and it appears that it may not be needed. (credit @neuradai )

Otherwise, the install steps still work as originally written, even if some of the last few commands can be skipped or changed slightly. Happy learning!!!

4 Likes

Any particular reasons for going to 20.04? I purposely went 18.04 hoping all the bugs were ironed out.

1 Like

20.04 is an LTS just like 18.04. It just came out, so there may be interest by others to try it out. For me it is in a virtual machine running as a headless server that I ssh into, and this VM only runs fastai. If it breaks, so what, I have the 18.04 server in a VM to fall back on. So no real reason other than, “why not?”. I will try out the desktop build in another VM maybe tomorrow.

For those with an older CUDA or Nvidia driver (in my case CUDA 10.1 and driver 418.67), a workaround is to force pytorch 1.4 and CUDA 10.1

You can do this by editing a copy of environment.yml (https://github.com/fastai/fastai2/blob/master/environment.yml) to replace the line “pytorch>=1.3” with the lines:
cudatoolkit=10.1
pytorch=1.4

Otherwise follow the instructions by FourMoBro above

Just a heads up for part 2 and Swift4Tensorflow much of the early release were on LTS 18.04.

Perhaps not sure if that’s included in V4 Part2 this year. The latest from the toolchain suggest 18.04 but for 20.04 would probably need a local compile from source.

S4TF Toolchain

I had no problem upgrading 18.04 to 20.04 with these instructions:

Fast.ai environment setup with local GPU - Ubuntu 20.04

I’ve a laptop with Intel Core i9, 8 GB RAM (to be upgraded to 16GB after the crisis), Nvidia 1650 with 4GB vRAM. Though there are multiple cloud options, running your notebooks locally feels great, at least for me. So I setup dual boot with Ubuntu 20.04 on my laptop. I ran the first 3 notebooks without any issues (except for the first lesson, the pet dataset takes longer than a while to download). All the information you’re seeing below is already present on the forums and FastAI Github page and course page. I had to do this twice since I messed up my OS setup and I had to search in the forums again to complete this. So I thought of putting up a note for myself and share it with you people. Thanks to @FourMoBro and all my other classmates who pitched in the forums.

Github Links

  1. The Part 1 - 2020 course is running on fastai v2, which isn’t generally available yet. So, we need to install it from the github source (since under active development)
git clone https://github.com/fastai/fastai2
cd fastai2
conda env create -f environment.yml
source activate fastai2
  1. Once done, cd into the cloned directory and install from the latest code
git clone https://github.com/fastai/fastai2
cd fastai2
pip install -e ".[dev]"

I made a silly mistake in forgetting to install after creating the environment above. So, when I tried running the course notebooks,it was not able to find the fasti2 library

  1. To use fastai2.medical.imaging you’ll also need to:
conda install pyarrow
pip install pydicom kornia opencv-python scikit-image
  1. fastai2 installation is now done.

  2. For the course/fastbook notebooks to run, there are some extra dependenices that you might want to install

pip install graphviz
pip install azure-cognitiveservices-vision-computervision
pip install azure-cognitiveservices-search-websearch
pip install azure-cognitiveservices-search-imagesearch
pip install "ipywidgets>=7.5.1"
pip install sentencepiece
pip install scikit_learn
  1. Once the installation is done, you can cd into the coursev4 or fastbook and start running the notebooks.

Link to Google Docs version of the same

I’ll try and update the notes with how I installed the Nvidia GPU driver as well.

2 Likes

I have just setup a DL workstation using an RTX 2070 Super and have used Unigene Heaved and Superposition to compare my setup and they seem just about there.

Are there any such standardized deep learning benchmarks I can use? I have been trying to replicate notebooks in Fastbook which I have previously run on Colab Pro and GCP using a Tesla P100, but I don’t think that gives an accurate representation…

Not to the best of my knowledge: Here’s what I did, bench marked individual parts and compared their scores online (after averaging over 5).

2 Likes

Any specific tests you used to benchmark the individual parts? I used the Unigene suite to benchmark the GPU, specifically Heaven and Superposition and compared my results to the leaderboard scores with the similar configs.

pip install graphviz

Thank you very much, worked beautiful (ubuntu 18.04, GeForce 2080Ti, driver version 440.100, CUDA v 10.2)

Hi everyone,
I’m totally new to this topic and i’ve been struggling with the set up part.
I have a GeForce MX150 on an Asus Laptop.
I am unable to install properly the Nvidia driver (tried with 390 and 440 version) and obtain the first picture of this post. I have followed the instructions but get the error :
“NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.”
I’ve looked at different forums but most of them seems to reinstall the driver and things work which is not my case.
Does someone encounter the same issue ?
If you need more info on my config or previous installation let me know.