Platform: Google Cloud Platform (GCP)

kdomnitser · March 18, 2020, 4:49am

If you want to clone course-v4 quickly without setting up ssh keys, you can use http instead as described in https://github.community/t5/How-to-use-Git-and-GitHub/why-is-public-key-needed-to-clone/m-p/38447. But will probably need to set that up anyway if you want to later push your work into your own remote repo.

sylvaint · March 18, 2020, 5:44am

You should run the commands with bash shell
switch your default shell to bash: https://www.howtogeek.com/444596/how-to-change-the-default-shell-to-bash-in-macos-catalina/

chsh -s /bin/bash
close and re open your terminal window

Then run
curl https://sdk.cloud.google.com | bash
exec -l $SHELL

zcancio · March 18, 2020, 7:07am

I’m avoiding the CLI by directly using the console.

You can spin up an instance using ai-platform notebooks.
https://console.cloud.google.com/ai-platform/notebooks/instances

I had to increase the GPU quota via
https://console.cloud.google.com/iam-admin/quotas

go_go_gadget · March 18, 2020, 3:44pm

Thanks! I do like zsh, though, so I’m a bit loathe to switch. I think I’ve got everything set up in Colab instead, so I’ll stick with that for now.

alexxcollins21 · March 18, 2020, 4:31pm

Dear all - I need some help!

My laptop OS is Ubuntu 16.04.6 LTS, and I’ve got a paid Google Cloud account which seems to be working fine (e.g. I have GPUs set up).

I’ve set up my instance (Step 3 in the fast.ai guide.)

I get errors when trying to set up ssh to Jupyter Notebook. I’ve tried to ssh into the instance without Jupyter too (i.e. gcloud compute ssh --project $PROJECT_NAME $INSTANCE_NAME ) but that gives errors too.

I’ve also noticed that in the console the VM instance has the “ssh” connection button greyed out, and the only option to connect is to use another ssh client.

These are the errors I get. I’ve verified that the firewall and instance are set up to accept ssh traffic.

If I set up an instance directly in the console it works fine and I can ssh into console. But I’m not sure quite how to set up a VM in the console with the same specs as via the guide - I’ll try to figure this out as a next step I suppose.

Any help would be greatly appreciated. If I can’t get this to work, I’ll go back to using Kaggle which I set up really quickly to do last year’s version of the course.

minh · March 18, 2020, 4:46pm

When cloning the fastai v4 folder. I got the following error.

Warning: Permanently added the RSA host key for IP address ‘140.82.112.3’ to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Does anyone know why that is the case? Thank you.

jeremy · March 18, 2020, 4:48pm

Try using http instead of ssh.

minh · March 18, 2020, 4:52pm

It works like a charm. Thank you.

alexxcollins21 · March 19, 2020, 12:56am

OMG. I finally did it after a lot of fiddling. I hadn’t noticed the pre-emtible error (wasn’t that obvious on the console) and so the reason I couldn’t ssh in was because the instance wasn’t running. So obvious now. And whilst trouble shooting I’d switched from using my native terminal to the CLI in Google Cloud. For some reason starting a Jupyter Notebook from CLI doesn’t work (localhost:8080 doesn’t connect). Finally tried from the native terminal and it all worked.

Summary: follow the instructions except for setting up a pre-emptible instance.

At least I’ve learnt quite a lot about the gcloud commands and Google Cloud in general whilst fiddling around with this.

jwuphysics · March 19, 2020, 2:32pm

Last year I had lots of issues with my instances constantly getting preempted. It probably doesn’t help that everybody in this course is using --zone=us-west1-b so it might be useful to spread out to other zones (but make sure to check GPU availability in each zone and zone-specific GPU pricing).

alexxcollins21 · March 20, 2020, 9:36am

I’m sure others have already thought of this, but I have forgotten to stop my VM instance twice since setting it up two days ago. I’ve written a (one line) bash script to stop the instance and have scheduled it to run with cron at 2am each day. That means I can only get billed a maximum of 20 hours if I forget to stop it.

I’ve also created a couple of aliases so that I don’t have to type long commands in the terminal to start the process and the tunnel. So instead of:

gcloud compute instances start alex-fastai-instance --zone=us-west1-b
gcloud compute ssh --zone=us-west1-b jupyter@alex-fastai-instance -- -L 8080:localhost:8080

I just type the following into the terminal to get started with the Jupyter notebook.

startFAI
GCssh

Hope this might help somebody save a few key strokes or some billable hours.

matt.mcclean · March 20, 2020, 9:58am

I have just published a setup guide for Amazon SageMaker.

gautam_e · March 20, 2020, 10:14pm

So putting that all together (retrospectively, hope I’m not missing anything important), I have:

Step 1: Creating your account
Step 2: Install Google CLI
Step 3: Create an instance (I didn’t use the “–preemptible” option)
Step 4:
In the command line clone the course-v4 repository with:
git clone https://github.com/fastai/course-v4
and the fastbook repository with
git clone https://github.com/fastai/fastbook.git
Note: A git pull within the course-v4 folder or the ‘fastbook’ folder keeps them up to date.
Also don’t forget to install fastai with:
pip install fastai2 in the command line and its dependencies with:
pip install -r requirements.txt after navigating to the course-v4 folder.
Step 5: Stop an instance

vijaysai · March 21, 2020, 3:51am

Trying to clone the latest course repository on gcloud does this. Anyone seen a similar issue ?

gautam_e · March 21, 2020, 10:57am

Did you try using https to clone the repository instead of ssh? The command for that is in my post above.

vijaysai · March 22, 2020, 3:07pm

https worked bro. Thanks.

steef · March 23, 2020, 1:12am

I’m getting error “sudo: /opt/anaconda3/bin/conda: command not found” in response to

sudo /opt/anaconda3/bin/conda install -c fastai fastai

In step 4 from https://course.fast.ai/start_gcp.html.

Does anyone have a solution?

steef · March 23, 2020, 1:36am

Also my remote host keeps closing my connection every 10 minutes or so.

E.g.

Which results in Jupyter Notebook kernel connection and autosave issues.

Any recommendations on how to fix this? Thanks!!

kdomnitser · March 23, 2020, 2:27am

That was happening to me as well. You can run this to find preemption events:

gcloud compute operations list --filter="operationType=compute.instances.preempted"

Probably you’ll see that your instance was preempted at the time the connection was closed. I started over with a new, non-preemptable instance. To do so, drop --preemptible from the command you use to create an instance.

gautam_e · March 23, 2020, 7:13pm

@steef, I think you want to install fastai2 and not fastai, am I right? The command you are using will install fastai and not fastai2.

I recommend using pip like jeremy stated:
pip install fastai2

Here are all the steps in my post above:

Hope it’s useful, and that I have not left anything out. Do let me know, should you try it out.