Jeremy's Harebrained install guide

(Jeremy Howard (Admin)) #1

Installing from scratch, plus Jeremy’s preferred tweaks

  • Install Ubuntu 18.04 LTS Server
  • Run the steps in this gist (but probably one step at a time to ensure it’s all OK)

Minimal subset of the full gist

If you already have Linux configured the way you want, here’s the steps to install CUDA and cudnn (note that your existing pytorch anaconda versions aren’t enough - you need to install it at system level):

cd
mkdir download
cd download
wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux
chmod u+x cuda_10.0.130_410.48_linux
./cuda_10.0.130_410.48_linux --extract=`pwd`
sudo ./cuda-linux.10.0.130-24817639.run -noprompt
echo /usr/local/cuda-10.0/lib64 | sudo tee -a /etc/ld.so.conf 
sudo ldconfig
echo 'export PATH=/usr/local/cuda-10.0/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

cd ~/download
wget http://files.fast.ai/files/cudnn-10.0-linux-x64-v7.5.0.56.tgz
tar xf cudnn-10.0-linux-x64-v7.5.0.56.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
sudo ldconfig

Now we have to set up a python 3.6 environment (s4tf currently doesn’t work with 3.7):

conda create -y --name swift python=3.6
conda activate swift
conda install -yc pytorch -c fastai pytorch fastai jupyter
python -c 'import torch; print(torch.tensor(1).cuda())'

jupyter notebook --generate-config
cat << 'EOF' >> ~/.jupyter/jupyter_notebook_config.py
c.NotebookApp.open_browser = False
c.NotebookApp.token = ''
EOF
pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --user

Now we can setup s4tf:

cd ~/download/
wget https://storage.googleapis.com/s4tf-kokoro-artifact-testing/latest/swift-tensorflow-DEVELOPMENT-cuda10.0-cudnn7-ubuntu18.04.tar.gz
sudo apt-get -y install git cmake ninja-build clang python uuid-dev libicu-dev icu-devtools libbsd-dev libedit-dev libxml2-dev libsqlite3-dev swig libpython-dev libncurses5-dev pkg-config libblocksruntime-dev libcurl4-openssl-dev systemtap-sdt-dev tzdata rsync
tar xf swift-tensorflow-DEVELOPMENT-cuda10.0-cudnn7-ubuntu18.04.tar.gz
cd
mkdir swift
cd swift
mv ~/download/usr ./
cd
echo 'export PATH=/usr/local/cuda-10.0/bin:$PATH' >> ~/.bashrc
echo 'export PATH=~/swift/usr/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

conda activate swift
mkdir -p git
cd ~/git
git clone https://github.com/google/swift-jupyter.git
cd swift-jupyter
python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs   --swift-toolchain ~/swift
cd ~/git
git clone git@github.com:fastai/fastai_docs.git
cd fastai_docs/
jupyter notebook
pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --user
jupyter notebook
12 Likes

New feature: Installing SwiftPM packages in Jupyter
(James Bradbury) #2

Note: hitting http://yann.lecun.com/exdb/mnist/ from dozens of computers at once is a bad idea; you can replace that folder with https://storage.googleapis.com/cvdf-datasets/mnist/ to use a Google-backed mirror.

0 Likes

(Thom M) #3

Just confirming this worked for me on a new google cloud compute Ubuntu 18.04 instance. (And I’m happy about it).

To use the notebook, I’m connecting using

gcloud compute ssh --zone=$ZONE jupyter@$INSTANCE_NAME -- -L 8888:localhost:8888 (google cloud specific but that’s the port forwarding config).

Only trick to remember is to remember to conda activate swift once you log in & before you start jupyter notebook otherwise jupyter won’t see your Swift kernel. I’m sure you could (and I probably will) append my .bashrc to do this step automagically.

3 Likes

#4

It does for me by renaming the python3-6 folder to python3-7 in swift/usr/lib (but this is a bit hacky). Here is a gist of a script I use to update my S4TF to the most recent build. Remove the line that renames the python directory if you followed Jeremy’s install, but otherwise it should work I believe.

0 Likes

(Thom M) #5

Not strictly part of the S4TF instructions, but I also had to pip install fire in the swift conda env to be able to run the course Python notebooks & exports as well as the Swift stuff.

0 Likes

#6

Yeah, I also ran into that. Perhaps fire should be included as a fastai dependency via Conda?

1 Like

(Jeremy Howard (Admin)) #7

Actually we shouldn’t be using that. Instead we should be using fastai.script.call_parse. Kinda embarrassed that I’m not actually using the CLI lib that I wrote!..

0 Likes

(tyoc213) #8

Nice, finally I can use swift… I dont know what exactly fixed the error error: failed to stop process at REPL breakpoint but it is probably the cuda and cudn setup and maybe the 3.6 because probably I was using 3.7.

But now it is working, I have used the minimal subset of the full list. Now I need to clean all the other installs and builds.

Forgot to mention… but my notebook always… always no matter what I do on the config ask me for a password (not even entering the token)… so probably we should run some like

python -c 'from notebook.auth import passwd; passwd()'

And use the value to write to the config

c.NotebookApp.password = 'sha1:68...3d9:53e678f....2'

with the other things writed in the configuration file for jupyter-notebook.

0 Likes

(Thom M) #9

I also had to install the pytorch nightly to run the current notebooks; I just forced conda to splat the main pytorch package: conda install -c pytorch pytorch-nightly --force-reinstall (make sure you’re in the swift conda env).

0 Likes

(Jeremy Howard (Admin)) #10

@marcrasi I’ve done a PR that removes the hard-coded python version (3.6) for conda. https://github.com/google/swift-jupyter/pull/57

It’s complaining about my CLA for some reason, so please either force merge it or just do the edit yourself.

0 Likes

(Thom M) #11

Another PSA: After following this script, my git username & email was set to Jeremy Howard,info@fast.ai :slight_smile: Don’t forget to change yours, or you might wonder how “Jeremy” is able to commit to your github repos…

git config --global user.name=<your name>
git config --global user.email=<your github email>

5 Likes

#12

FYI on a machine with preexisting Nvidia 418 driver and CUDA 10.1, it worked using the minimal steps starting with conda create.

I wasn’t sure Tensorflow was going to work since it seems to want CUDA 10.0 - but then so far it’s only doing the download of MNIST (I think) so may be it hasn’t gotten to any CUDA version conflicts.

Very cool!

0 Likes

(Andrew Nguyen) #13

Reporting that I have successfully installed and ran 00_load_data.ipynb on a fresh Ubuntu 18.04 LTS image on GCP. I followed Jeremy’s script line by line.

A couple of notes for those who experience strange messages when running source ~/.bashrc.

  • Line 50 is inaccurate and redundant. The Anaconda3 install adds lines to .bashrc that handle that part, and my Anaconda3, for whatever reason, lives in /home/jupyter not /home/ubuntu. I did decide to add conda activate swift after the fact, though, which means I didn’t need line 51 either.
  • Line 127 is a repeat of line 61. I just deleted the extra one.

Thanks, Jeremy!

0 Likes