GCP is indeed a great option. But I am currently working out all notebooks through PaperSpace, their response was pretty quick.
Hi all, I have tried the update script and at the end I am told to:
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8888/?token=a06bfxcf2j4ndknjfnkf&token=dfggdgfdgrdhjsnsjskns
I copied and pasted the link to a browser, however, I could not connect to the VM . In the last iterations I was asked to connect to a real ip with 8888 port. Does anybody know how to revert back to that option.
Thanks
Problem is solved .
Hi all,
I have a question regarding running Jupyter notebook on Google Cloud. Are we able to run Jupyter without running a GPU instance, hence saving GPU hours. I believe thatās how Crestle works? Thanks.
Well you can use CPU to run Pytorch , however it will be really slow.
I see. In that case, how do you run it? On the gpu instance even when youāre editing and writing code?
Everything goes fine until I try to connect to jupyter notebook.
Both methods donāt work. I have checked firewall rules and those are as specified.
When running through second method i get following error when opening a new terminal and trying to run following command:
gcloud compute ssh fastai-vm --ssh-flag=ā-Lā --ssh-flag=ā8888:localhost:8888ā
ssh: Could not resolve hostname \342\200\234-l\342\200\235: Name or service not known
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Any help is appreciated
For those interested, found a working solution by following these instructions:
1 Like
While setting up instance on google cloud platform with free credits(mentioned this because could not upgrade quota for using gpu) my region(mumbai) does not support gpu. Should I use different region, but this would add to the latency. I have never used any cloud computing platform before.
Clouderizer Fast.ai community template can be used on GCP VMs as well. Here are the instructions
https://help.clouderizer.com/running-on-cloud/running-on-google-cloud-platform
This should take care of dependency driver/package setup, datasets and remote Jupyter notebook access.
1 Like
Jupyter Notebooks are pretty good at hiding latency from remote systems, since you do the editing locally. Iāve found the Google credit program to work really well and would encourage you to carefully follow the Medium post linked earlier in the thread.
Hi,
I followed this post and able to run the notebook
But while running, i got an error saying āPackageNotFoundError:cv2ā
can some one help me?
Clouderizer Fast.ai community template can be used on GCP VMs as well. Here are the instructions
The link is missing the page, please provide the correct link
This process used to work for me but it doesnāt anymore. I donāt know why but I am suspecting it has to do with active development of fastai so might be broken recently (1-2 days ago I think). After running the script and reboot, ssh into it again, running jupyter notebook gives me the following error (fastai env activated by default).
$ jupyter notebook
[I 17:10:08.556 NotebookApp] Writing notebook server cookie secret to /run/user/1002/jupyter/notebook_cookie_secret
Traceback (most recent call last):
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'allow_remote_access'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 869, in _default_allow_remote
addr = ipaddress.ip_address(self.ip)
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/ipaddress.py", line 54, in ip_address
address)
ValueError: '' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pghoou/anaconda3/envs/fastai/bin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 1629, in initialize
self.init_webapp()
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 1379, in init_webapp
self.jinja_environment_options,
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 158, in __init__
default_url, settings_overrides, jinja_env_options)
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 251, in init_settings
allow_remote_access=jupyter_app.allow_remote_access,
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/site-packages/notebook/notebookapp.py", line 872, in _default_allow_remote
for info in socket.getaddrinfo(self.ip, self.port, 0, socket.SOCK_STREAM):
File "/home/pghoou/anaconda3/envs/fastai/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
A good way to prevent regression would be lock the dependency versions and also not just git clone the latest but check out only the stable versions, I think. Does anybody have the same problem ?
Yes, I have the same issues. That Medium post and instructions worked perfectly for me in the past, but my attempts to restart my deep learning efforts have failed and Iāve spent a lot of time on this the last two days. I also tried the instructions in the first post of this thread and that also didnāt work.
Iām admittedly very much a newbie when it comes to linux and the installation scripts. Hoping that someone can provide some updates to the instructions so that this initial barrier (which is huge if youāre very much a beginner) of entry to the courses is removed.
1 Like
Iām back in business (finally)!
I went through the script provided in the Medium post (https://medium.com/@howkhang/ultimate-guide-to-setting-up-a-google-cloud-machine-for-fast-ai-version-2-f374208be43) line by line and tried to correct all errors. The original bash script can be found here (linked in the Medium post as well): https://raw.githubusercontent.com/howkhang/fastai-v2-setup/master/setup.sh
I changed a few lines, and have copied the updated script below. Every line I changed or added includes a comment starting with ā# JEā preceding it. Iām not sure if all the changes are required, as I am, alas, a linux/script newbie, but it worked for me!
#! /bin/bash
DEBIAN_FRONTEND=noninteractive
sudo rm /etc/apt/apt.conf.d/*.*
sudo apt update
mkdir downloads
cd downloads
wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh
bash Anaconda3-5.0.1-Linux-x86_64.sh -b
cd
git clone https://github.com/fastai/fastai.git
cd fastai/
# JE changes git revision to older v0.7.2
git reset --hard e85667cfae2e6873b1bb026195b5d09a74dfcff9
echo 'export PATH=~/anaconda3/bin:$PATH' >> ~/.bashrc
export PATH=~/anaconda3/bin:$PATH
source ~/.bashrc
# JE required to solve "Command "python setup.py egg_info" failed with error code 1" error
sudo apt install -y build-essential
conda env update
echo 'source activate fastai' >> ~/.bashrc
source activate fastai
source ~/.bashrc
cd ..
mkdir data
cd data
wget http://files.fast.ai/data/dogscats.zip
sudo apt install unzip -y
sudo apt -y upgrade --force-yes
sudo apt -y autoremove
unzip -q dogscats.zip
cd ../fastai/courses/dl1/
ln -s ~/data ./
jupyter notebook --generate-config
# JE modified below's "c.NotebookApp.ip = '*'" to correct traitlets.py KeyError: 'allow_remote_access' when starting jupyter
echo "c.NotebookApp.ip = '0.0.0.0'" >> ~/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.open_browser = False" >> ~/.jupyter/jupyter_notebook_config.py
sudo ufw allow 8888/tcp
sudo apt -y install qtdeclarative5-dev qml-module-qtquick-controls
sudo add-apt-repository ppa:graphics-drivers/ppa -y
sudo apt update
cd ~/downloads/
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt update
sudo apt install cuda -y
wget http://files.fast.ai/files/cudnn-9.1-linux-x64-v7.tgz
tar xf cudnn-9.1-linux-x64-v7.tgz
sudo cp cuda/include/*.* /usr/local/cuda/include/
sudo cp cuda/lib64/*.* /usr/local/cuda/lib64/
pip install ipywidgets
jupyter nbextension enable --py widgetsnbextension --sys-prefix
# JE weight files required for some lessons
cd ~/fastai/fastai
wget http://files.fast.ai/models/weights.tgz
tar -xvzf weights.tgz
sudo reboot
1 Like
Apologies. Here is the updated link
Hi, for me that ultimate guide does not work anymore and I also found a few points I wanted to modify/improve. So I wrote a medium post that provides script and explanation that builds on top of whatās already written in ultimate guide. I hope somebody find it useful! Check out if interested!
HI, the āultimate guideā was not written by me. Thank you for writing a new Medium post.
- When you say ādoes not work anymoreā, could you summarize what have changed in Google Cloud, Anaconda, fastai library, etc., here?
- Which version of fastai library your guide (and shell scripts) were designed for?
- I recommend you notify the original author of the āultimate guideā to update their article so future readers are aware of your Medium post if they are looking for an updated Google Cloud guide for deep learning workloads.
Thanks.
Hi, @cedric thanks for the questions and recommendation.
- When you say ādoes not work anymoreā, could you summarize what have changed in Google Cloud, Anaconda, fastai library, etc., here?
=> I am actually not sure what is the cause of this break. All I know is symptom and the fact the original post used to work like last week. I can direct you to a forum post I posted a while ago where I got my problem solved. Maybe somebody will notice what actually broke it.
- Which version of fastai library your guide (and shell scripts) were designed for?
=> The original script also didnāt to much more than cloning fastai repo, activating fastai env, download cats and dogs regarding fastai specifically (the other things were installing Anaconda and NVIDIA Graphics driver) The github repo comes with v1 but also 0.7 is in old folder so I guess people can use whichever version they like to use. Please correct me if I am wrong here though.
- I recommend you notify the original author of the āultimate guideā to update their article so future readers are aware of your Medium post if they are looking for an updated Google Cloud guide for deep learning workloads.
=> I notified original author by a medium response but I am not sure he read it.
1 Like