Jeremy's Harebrained install guide

(Jeremy Howard (Admin)) #1

Installing from scratch, plus Jeremy’s preferred tweaks

  • Install Ubuntu 18.04 LTS Server
  • Run the steps in this gist (but probably one step at a time to ensure it’s all OK)

Minimal subset of the full gist

If you already have Linux configured the way you want, here’s the steps to install CUDA and cudnn (note that your existing pytorch anaconda versions aren’t enough - you need to install it at system level):

cd
mkdir download
cd download
wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux
chmod u+x cuda_10.0.130_410.48_linux
./cuda_10.0.130_410.48_linux --extract=`pwd`
sudo ./cuda-linux.10.0.130-24817639.run -noprompt
echo /usr/local/cuda-10.0/lib64 | sudo tee -a /etc/ld.so.conf 
sudo ldconfig
echo 'export PATH=/usr/local/cuda-10.0/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

cd ~/download
wget http://files.fast.ai/files/cudnn-10.0-linux-x64-v7.5.0.56.tgz
tar xf cudnn-10.0-linux-x64-v7.5.0.56.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
sudo ldconfig

Now we can setup s4tf:

cd ~/download/
wget https://storage.googleapis.com/swift-tensorflow-artifacts/releases/v0.3/rc1/swift-tensorflow-RELEASE-0.3-cuda10.0-cudnn7-ubuntu18.04.tar.gz
# apt-fast is recommended because it's fast. Can you just `apt` if you prefer
sudo apt-fast -y install git cmake ninja-build clang python uuid-dev libicu-dev icu-devtools libbsd-dev libedit-dev libxml2-dev libsqlite3-dev swig libpython-dev libncurses5-dev pkg-config libblocksruntime-dev libcurl4-openssl-dev systemtap-sdt-dev tzdata rsync
tar xf swift-tensorflow-RELEASE-0.3-cuda10.0-cudnn7-ubuntu18.04.tar.gz
cd
mkdir swift
cd swift
mv ~/download/usr ./
cd
echo 'export PATH=~/swift/usr/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

mkdir -p git
cd ~/git
git clone https://github.com/google/swift-jupyter.git
cd swift-jupyter
python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs   --swift-toolchain ~/swift
cd ~/git
git clone https://github.com/fastai/fastai_docs.git
cd fastai_docs/
jupyter notebook

Updating

Here’s how to update to the latest release (replace the file name below with the latest release filename):

wget https://storage.googleapis.com/swift-tensorflow-artifacts/releases/latest/swift-tensorflow-RELEASE-LATEST-cuda10.0-cudnn7-ubuntu18.04.tar.gz
tar xf swift-tensorflow-RELEASE-LATEST-cuda10.0-cudnn7-ubuntu18.04.tar.gz
cd ~/swift/
rm -rf usr
mv ~/download/usr ./
cd ~/git/swift-jupyter
git pull
python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs   --swift-toolchain ~/swift
31 Likes

New feature: Installing SwiftPM packages in Jupyter
Platform: GCP ✅
S4TF in colab (error)
cudaGetDevice() failed 01_matmul.ipynb
(James Bradbury) #2

Note: hitting http://yann.lecun.com/exdb/mnist/ from dozens of computers at once is a bad idea; you can replace that folder with https://storage.googleapis.com/cvdf-datasets/mnist/ to use a Google-backed mirror.

2 Likes

(Thom M) #3

Just confirming this worked for me on a new google cloud compute Ubuntu 18.04 instance. (And I’m happy about it).

To use the notebook, I’m connecting using

gcloud compute ssh --zone=$ZONE jupyter@$INSTANCE_NAME -- -L 8888:localhost:8888 (google cloud specific but that’s the port forwarding config).

Only trick to remember is to remember to conda activate swift once you log in & before you start jupyter notebook otherwise jupyter won’t see your Swift kernel. I’m sure you could (and I probably will) append my .bashrc to do this step automagically.

5 Likes

#4

It does for me by renaming the python3-6 folder to python3-7 in swift/usr/lib (but this is a bit hacky). Here is a gist of a script I use to update my S4TF to the most recent build. Remove the line that renames the python directory if you followed Jeremy’s install, but otherwise it should work I believe.

0 Likes

(Thom M) #5

Not strictly part of the S4TF instructions, but I also had to pip install fire in the swift conda env to be able to run the course Python notebooks & exports as well as the Swift stuff.

2 Likes

#6

Yeah, I also ran into that. Perhaps fire should be included as a fastai dependency via Conda?

2 Likes

(Jeremy Howard (Admin)) #7

Actually we shouldn’t be using that. Instead we should be using fastai.script.call_parse. Kinda embarrassed that I’m not actually using the CLI lib that I wrote!..

0 Likes

(tyoc213) #8

Nice, finally I can use swift… I dont know what exactly fixed the error error: failed to stop process at REPL breakpoint but it is probably the cuda and cudn setup and maybe the 3.6 because probably I was using 3.7.

But now it is working, I have used the minimal subset of the full list. Now I need to clean all the other installs and builds.

Forgot to mention… but my notebook always… always no matter what I do on the config ask me for a password (not even entering the token)… so probably we should run some like

python -c 'from notebook.auth import passwd; passwd()'

And use the value to write to the config

c.NotebookApp.password = 'sha1:68...3d9:53e678f....2'

with the other things writed in the configuration file for jupyter-notebook.

0 Likes

(Thom M) #9

I also had to install the pytorch nightly to run the current notebooks; I just forced conda to splat the main pytorch package: conda install -c pytorch pytorch-nightly --force-reinstall (make sure you’re in the swift conda env).

0 Likes

(Jeremy Howard (Admin)) #10

@marcrasi I’ve done a PR that removes the hard-coded python version (3.6) for conda. https://github.com/google/swift-jupyter/pull/57

It’s complaining about my CLA for some reason, so please either force merge it or just do the edit yourself.

0 Likes

(Thom M) #11

Another PSA: After following this script, my git username & email was set to Jeremy Howard,info@fast.ai :slight_smile: Don’t forget to change yours, or you might wonder how “Jeremy” is able to commit to your github repos…

git config --global user.name=<your name>
git config --global user.email=<your github email>

9 Likes

#12

FYI on a machine with preexisting Nvidia 418 driver and CUDA 10.1, it worked using the minimal steps starting with conda create.

I wasn’t sure Tensorflow was going to work since it seems to want CUDA 10.0 - but then so far it’s only doing the download of MNIST (I think) so may be it hasn’t gotten to any CUDA version conflicts.

Very cool!

0 Likes

(Andrew Nguyen) #13

Reporting that I have successfully installed and ran 00_load_data.ipynb on a fresh Ubuntu 18.04 LTS image on GCP. I followed Jeremy’s script line by line.

A couple of notes for those who experience strange messages when running source ~/.bashrc.

  • Line 50 is inaccurate and redundant. The Anaconda3 install adds lines to .bashrc that handle that part, and my Anaconda3, for whatever reason, lives in /home/jupyter not /home/ubuntu. I did decide to add conda activate swift after the fact, though, which means I didn’t need line 51 either.
  • Line 127 is a repeat of line 61. I just deleted the extra one.

Thanks, Jeremy!

0 Likes

(Jeremy Howard (Admin)) #14

Many thanks - both fixed now. (The 1st is now required, since I use “batch mode” when installing conda to avoid prompts.)

2 Likes

(Edward Ross) #15

I found this worked except I got an error with the line

The glob expansion didn’t work (I think because lib64 was not readable by my user), so I found the names of all the libcudnn files (ls cuda/lib64/libcudnn*) and executed chmod on them.

0 Likes

(John) #16

I tried the renaming trick you mention:
mv ~/swift/usr/lib/python3.7 ~/swift/usr/lib/python3.7-bak
mv ~/swift/usr/lib/python3.6 ~/swift/usr/lib/python3.7

but still get the following error:

python3 register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs --swift-toolchain ~/swift
Traceback (most recent call last): File "register.py", line 239, in <module> main() File "register.py", line 139, in main kernel_env = make_kernel_env(args) File "register.py", line 99, in make_kernel_env libpython = glob(sys.prefix+'/lib/libpython*.so')[0]
IndexError: list index out of range

Any ideas how to make it cooperate?

EDIT: added renaming steps

0 Likes

(Kevin Bird) #17

image
To help people searching:
Kernel not found
Could not find a kernel matching Swift. Please select a kernel:

I had this issue when I first opened my notebook, it turned out, my problem was that I didn’t do this step properly:

cd swift-jupyter
python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs   --swift-toolchain ~/swift

Once I reran this, it worked properly. Hopefully this can help somebody else if they see this issue.

0 Likes

(Lankinen) #19

What’s this?


I have gtx 1080 ti, ubuntu, and I followed the guide.

1 Like

(Pedro Cuenca) #20

Did you install the nvidia drivers? They need to be present even if you have installed cuda and cuDNN. What does nvidia-smi say? Please, refer to lines 35-39 of the complete script Jeremy posted.

0 Likes

(Bart Fish) #21

Don’t know if this is the appropriate place to ask, but would like to get this ecosystem installed on the NVIDIA Jetson Nano. It uses the ARM64 architecture and supports UBUNTU-18.04 debian style installs. I looked at various websites yesterday for install packages for SWTF and came up with this one, but could not get it to play with jupyter-swift. Can someone in the SWTF team get me pointed in the correct direction.

0 Likes