Setup problems: AWS

tentotheminus9 · July 18, 2017, 10:57am

Hi all,

I’ve been trying to create the P2 instance using Windows 10 and Cygwin for 2 days. I also tried installing Ubuntu bash to no avail.

After almost losing the will to live and having experienced just about every setup_p2.sh error under the sun, I installed a Linux virtual machine, copied the entire setup folder from the git repository to my (virtual) Ubuntu desktop, installed aws, then ran the aws configuration followed by bash setup_p2.sh from the linux command line. Everything worked like a dream, and now the P2 instance is up and running. If you’re in a similar situation, this approach may help.

dvs · July 19, 2017, 6:08am

Yep, this was really helpful. Thanks!

jp_beaudry · July 19, 2017, 1:25pm

samsonpaturi,

The error is suggestive of an error in the way ssh is invoked. In other words, one of the parameters is wrong.

Wild guess: there’s supposed to be a space after the ‘i’ in “-i/home/ss…”

If that’s not it, then closely review the ssh command line.

HTH,
JP

samsonpaturi · July 19, 2017, 11:40pm

Hi JP, thanks for your suggestion. The issue is not the space, I tried that already. I get the same output if I simply type ssh command. I am basically stuck without any progress.

jp_beaudry · July 20, 2017, 2:35am

Oh, that’s a bummer. Let’s keep trying:

Can you ssh successfully anywhere else with any set of parameters? Back at school or in some corporate setup, I had access to many servers to practice on. Nowadays, I’d Google a bit and/or would setup my own SSH server. This thread for example, looks like a promising source of test locations:

You need to isolate the fault. Is it with the specific AWS parameters? Is it a ssh client install issue? Funky interaction with your shell?

Also, are you able to determine exactly which ssh client is being called? My memory of which unix commands are available on Cygwin is fuzzy, but perhaps “which ssh” would do it.

ryan_c · July 21, 2017, 12:20am

After a LOT of time spent on this, I finally found a solution that works.

The primary issue is that the cygwin didn’t come with python installed, and doesn’t know where to find the existing Windows Anaconda version on your machine. This can be verified by running which python from within cygwin - it couldn’t find where python is saved. Note that this can be confusing because running pip install awscli likely doesn’t throw an error message. Cygwin actually installs awscli in the Window’s Anaconda installation of Python (I find this odd since we didn’t run conda install awscli).

HOWEVER, rather than try to point cygwin to the already installed version of Anaconda python on your machine it will save you a ton of headache to just install a cygwin-specific instance of python.

The initial steps to run in cygwin are (documented here: http://wiki.fast.ai/index.php/Awscli_in_cygwin)

pip uninstall awscli
wget rawgit.com/transcode-open/apt-cyg/master/apt-cyg
install apt-cyg /bin
apt-cyg install python
wget https://bootstrap.pypa.io/get-pip.py python get-pip.py pip install awscli

…Note, however, that the first command pip uninstall awscli “hung up” for me. So just escape out of it using quit() and continue with the others in order.

You can check that everything worked if you run which python in cygwin and it points to the cygin version (i.e. /usr/bin/python , as opposed to: /users/…/Anaconda2/).

The next CRITICAL step is that when you download all the shell scripts from Github setup folder (https://github.com/fastai/courses/tree/master/setup), Windows automatically CRLF line terminators! Therefore, in cygwin, run the following commands:

apt-cyg install dos2unix
dos2unix setup_p2.sh
dos2unix setup_instance.sh
then finally, bash setup_p2.sh

This should do the trick.

susanli · July 21, 2017, 4:08am

Hi, mchineLRN, I am having the same issue setting up, I ran wget https://github.com/fastai/courses/blob/master/setup/setup_p2.sh and wget https://raw.githubusercontent.com/fastai/courses/master/setup/setup_instance.sh but I am still getting
$ bash setup_p2.sh
setup_p2.sh: line 2: syntax error near unexpected token newline' setup_p2.sh: line 2:’

Can you please let me know how did you solve this?

Thank you
Susan

susanli · July 21, 2017, 4:41am

Hi, I am having the same error,
$ bash setup_p2.sh
setup_p2.sh: line 2: syntax error near unexpected token newline' setup_p2.sh: line 2:’

and I tried wget https://raw.githubusercontent.com/fastai/courses/master/setup/aws-alias.sh
wget https://raw.githubusercontent.com/fastai/courses/master/setup/setup_instance.sh
wget https://raw.githubusercontent.com/fastai/courses/master/setup/setup_p2.sh
with no luck. You said copy paste text on github, which text?
Thanks
Susan

mchineLRN · July 21, 2017, 11:44am

setup_instance.sh:
wget https://raw.githubusercontent.com/fastai/courses/master/setup/setup_instance.sh

should fix your problem if I remember correctly. Not sure the logic behind it but I installed it just playing around it worked for me.

susanli · July 22, 2017, 5:17am

Hi Josh, do you still remember how you solved this problem? my CAT command is full of HTML tags. What scripts did you use to make it right?

Thanks
Susan

z0k · July 22, 2017, 6:02am

Hi @susanli,

Did you delete the original setup_instance.sh file? If not, each new file you download with wget is appended with a number.

susanli · July 22, 2017, 7:44am

Thank you, now it works like a dream!

susanli · July 22, 2017, 7:46am

Thanks for your help, now it works!

georgemcintire · July 26, 2017, 5:18pm

When I try to ssh into the AWS instance I keep getting this:

connect to host ec2-XX-XXX-XXX-XX.us-west-2.compute.amazonaws.com port 22: Operation timed out

How do I solve this?

z0k · July 27, 2017, 7:08am

Hi @georgemcintire,

A few things come to mind:

Is your instance running?
Can you confirm that the IP address is correct?’
Do the inbound rules in your security group have your local IP in the source list?

Max0924 · July 29, 2017, 8:52am

Hi, I am having the same error,
$ bash setup_p2.sh
setup_p2.sh: line 2: syntax error near unexpected token newline’
setup_p2.sh: line 2:'
but
how to delete the original setup_instance.sh file?

z0k · July 29, 2017, 11:21am

Hi @Max0924,

It should just be the first file you downloaded. You can find all of them by typing

ls | grep setup_instance.sh

in the directory you downloaded the scripts. Once you remove the original file, you can use wget to get the proper version from the raw url.

Ubi · July 29, 2017, 6:32pm

I am using the Ubuntu Bash on Win10 because for some reasons Cygwin does not recognize commands like “conda” or “pip” on my computer ( I tried to add Anaconda to .bashrc but it still didn’t work).

Everything went fine in Ubuntu Bash until I set up the instance and tried to connect to it.

So I SSH, and it said,

I removed everything and started over but still kept receiving “Permission denied (publickey).”

What can I do?

Help much appreciated.

zeinab_sp · July 31, 2017, 2:47am

Hi,

I ran “bash setup_p2.sh” and I get the following error:

True

An error occurred (InvalidKeyPair.NotFound) when calling the RunInstances operation: The key pair ‘aws-key-fast-ai’ does not exist

An error occurred (MissingParameter) when calling the CreateTags operation: The request must contain the parameter resourceIdSet
Waiting for instance start…

Waiter InstanceRunning failed: Max attempts exceeded
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --instance-id: expected one argument

An error occurred (MissingParameter) when calling the RebootInstances operation: The request must contain the parameter InstancesSet
All done. Find all you need to connect in the fast-ai-commands.txt file and to remove the stack call fast-ai-remove.sh
Connect to your instance: ssh -i /home/Zeinab/.ssh/aws-key-fast-ai.pem ubuntu@None

I checked and I know that the file “aws-key-fast-ai.pem” does not exist, but I don’t know how to create it. Any help would be appreciated.

Elian · August 1, 2017, 8:57am

Many thanks Max, worked out fine for me !