Lesson 1 using Google Cloud VM( step by step installation with shell script)

Thanks @steffenix for putting the link here. I just started the course. I followed several Posts that share the steps to setup GCP for this course without going through the Free Tier documentation updates. Each time i raise a Quota request, i got back a mail with upfront payment steps. Guess we cant use GCP Free Tier for this course anymore.:expressionless:

Seems like you have to upgrade the account to paid account (but you can keep on using the free 300usd) . Then you will have to ask for a quota change https://console.cloud.google.com/iam-admin/quotas

After I’ve got an automated email asking me to make a payment of $35 US to help us ensure that this is a legitimate request.

I have spent 35 usd and I have made the request to increase the quota, I have now access to GPUs.

GCP is indeed a great option. But I am currently working out all notebooks through PaperSpace, their response was pretty quick.

Hi all, I have tried the update script and at the end I am told to:

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8888/?token=a06bfxcf2j4ndknjfnkf&token=dfggdgfdgrdhjsnsjskns

I copied and pasted the link to a browser, however, I could not connect to the VM . In the last iterations I was asked to connect to a real ip with 8888 port. Does anybody know how to revert back to that option.
Thanks

Problem is solved .

Hi all,

I have a question regarding running Jupyter notebook on Google Cloud. Are we able to run Jupyter without running a GPU instance, hence saving GPU hours. I believe that’s how Crestle works? Thanks.

Well you can use CPU to run Pytorch , however it will be really slow.

I see. In that case, how do you run it? On the gpu instance even when you’re editing and writing code?

Everything goes fine until I try to connect to jupyter notebook.
Both methods don’t work. I have checked firewall rules and those are as specified.
When running through second method i get following error when opening a new terminal and trying to run following command:

gcloud compute ssh fastai-vm --ssh-flag=“-L” --ssh-flag=“8888:localhost:8888”

ssh: Could not resolve hostname \342\200\234-l\342\200\235: Name or service not known
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Any help is appreciated

For those interested, found a working solution by following these instructions:

1 Like

While setting up instance on google cloud platform with free credits(mentioned this because could not upgrade quota for using gpu) my region(mumbai) does not support gpu. Should I use different region, but this would add to the latency. I have never used any cloud computing platform before.

Clouderizer Fast.ai community template can be used on GCP VMs as well. Here are the instructions

https://help.clouderizer.com/running-on-cloud/running-on-google-cloud-platform

This should take care of dependency driver/package setup, datasets and remote Jupyter notebook access.

1 Like

Jupyter Notebooks are pretty good at hiding latency from remote systems, since you do the editing locally. I’ve found the Google credit program to work really well and would encourage you to carefully follow the Medium post linked earlier in the thread.

Hi,
I followed this post and able to run the notebook


But while running, i got an error saying “PackageNotFoundError:cv2”
can some one help me?

Clouderizer Fast.ai community template can be used on GCP VMs as well. Here are the instructions

The link is missing the page, please provide the correct link

This process used to work for me but it doesn’t anymore. I don’t know why but I am suspecting it has to do with active development of fastai so might be broken recently (1-2 days ago I think). After running the script and reboot, ssh into it again, running jupyter notebook gives me the following error (fastai env activated by default).

$ jupyter notebook
[I 17:10:08.556 NotebookApp] Writing notebook server cookie secret to /run/user/1002/jupyter/notebook_cookie_secret
Traceback (most recent call last):
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'allow_remote_access'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 869, in _default_allow_remote
    addr = ipaddress.ip_address(self.ip)
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/ipaddress.py", line 54, in ip_address
    address)
ValueError: '' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/pghoou/anaconda3/envs/fastai/bin/jupyter-notebook", line 11, in <module>
    sys.exit(main())
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 1629, in initialize
    self.init_webapp()
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 1379, in init_webapp
    self.jinja_environment_options,
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 158, in __init__
    default_url, settings_overrides, jinja_env_options)
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 251, in init_settings
    allow_remote_access=jupyter_app.allow_remote_access,
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 872, in _default_allow_remote
    for info in socket.getaddrinfo(self.ip, self.port, 0, socket.SOCK_STREAM):
  File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

A good way to prevent regression would be lock the dependency versions and also not just git clone the latest but check out only the stable versions, I think. Does anybody have the same problem ?

Yes, I have the same issues. That Medium post and instructions worked perfectly for me in the past, but my attempts to restart my deep learning efforts have failed and I’ve spent a lot of time on this the last two days. I also tried the instructions in the first post of this thread and that also didn’t work.

I’m admittedly very much a newbie when it comes to linux and the installation scripts. Hoping that someone can provide some updates to the instructions so that this initial barrier (which is huge if you’re very much a beginner) of entry to the courses is removed.

1 Like

I’m back in business (finally)!

I went through the script provided in the Medium post (https://medium.com/@howkhang/ultimate-guide-to-setting-up-a-google-cloud-machine-for-fast-ai-version-2-f374208be43) line by line and tried to correct all errors. The original bash script can be found here (linked in the Medium post as well): https://raw.githubusercontent.com/howkhang/fastai-v2-setup/master/setup.sh

I changed a few lines, and have copied the updated script below. Every line I changed or added includes a comment starting with ‘# JE’ preceding it. I’m not sure if all the changes are required, as I am, alas, a linux/script newbie, but it worked for me!

#! /bin/bash

DEBIAN_FRONTEND=noninteractive

sudo rm /etc/apt/apt.conf.d/*.*
sudo apt update
mkdir downloads
cd downloads
wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh
bash Anaconda3-5.0.1-Linux-x86_64.sh -b
cd
git clone https://github.com/fastai/fastai.git
cd fastai/

# JE changes git revision to older v0.7.2
git reset --hard e85667cfae2e6873b1bb026195b5d09a74dfcff9

echo 'export PATH=~/anaconda3/bin:$PATH' >> ~/.bashrc
export PATH=~/anaconda3/bin:$PATH
source ~/.bashrc

# JE required to solve "Command "python setup.py egg_info" failed with error code 1" error
sudo apt install -y build-essential

conda env update
echo 'source activate fastai' >> ~/.bashrc
source activate fastai
source ~/.bashrc
cd ..
mkdir data
cd data
wget http://files.fast.ai/data/dogscats.zip
sudo apt install unzip -y
sudo apt -y upgrade --force-yes
sudo apt -y autoremove
unzip -q dogscats.zip
cd ../fastai/courses/dl1/
ln -s ~/data ./
jupyter notebook --generate-config
# JE modified below's "c.NotebookApp.ip = '*'" to correct traitlets.py KeyError: 'allow_remote_access' when starting jupyter
echo "c.NotebookApp.ip = '0.0.0.0'" >> ~/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.open_browser = False" >> ~/.jupyter/jupyter_notebook_config.py
sudo ufw allow 8888/tcp
sudo apt -y install qtdeclarative5-dev qml-module-qtquick-controls
sudo add-apt-repository ppa:graphics-drivers/ppa -y
sudo apt update
cd ~/downloads/
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt update
sudo apt install cuda -y
wget http://files.fast.ai/files/cudnn-9.1-linux-x64-v7.tgz
tar xf cudnn-9.1-linux-x64-v7.tgz
sudo cp cuda/include/*.* /usr/local/cuda/include/
sudo cp cuda/lib64/*.* /usr/local/cuda/lib64/
pip install ipywidgets
jupyter nbextension enable --py widgetsnbextension --sys-prefix

# JE weight files required for some lessons
cd ~/fastai/fastai
wget http://files.fast.ai/models/weights.tgz
tar -xvzf weights.tgz

sudo reboot
1 Like

Apologies. Here is the updated link