Setup problems: AWS

Victor1 · December 25, 2017, 12:21am

Hi all,

with a lot of difficulties I got to this point: when runing “bash setup_p2.sh” I get

"Only us-west-2 (Oregon), eu-west-1 (Ireland), and us-east-1 (Virginia) are currently supported "

But if I run “aws configure get region” I get “eu-west-1”. I don’t know what’s the problem.

I’m using Cygwin on Window 10 64.

Thanks in advance. Looking forward to starting the course

Nell · December 25, 2017, 1:01am

in your “aws configure” command, try to manually add the region as us-west-2

if that doesnt fix it then go to ~/.aws/config using an editor like vi “vi ~/.aws/config/” to check what your region is.

mhutson · December 25, 2017, 5:44pm

Hi,

I ran the setup script (modified for t2_medium), which created a t2_medium instance and a 128 GB volume. I detached the 128 GB volume and created and attached a 30 GB volume (with the device name “/dev/sda1”).

But when I try to start the instance, it immediately stops and I get this message: “Client.InstanceInitiatedShutdown: Instance initiated shutdown”. Do you know why that happens?

(Somewhere else I read that this problem may occur when there’s no OS on the volume, but I don’t know how to tell if that’s the case here.)

Thanks.

Nell · December 25, 2017, 8:37pm

I think your answer lies in doing something similar to what this post tells you

https://www.codesd.com/item/why-is-the-ec2-instance-not-started-successfully-after-resizing-the-root-ebs-volume.html

mhutson · December 25, 2017, 9:40pm

Thanks. Can I just create a new instance with a 30 GB volume? I’ve done a lot of Googling on how to specify the volume size when creating an instance, without any luck.

Nell · December 26, 2017, 12:06am

yeah i dont know any way to do that either. But after you create your instance you can use a command like this

aws ec2-modify-volume-attribute --volume --size -

to reduce the size and see if that works?

Victor1 · December 26, 2017, 12:48pm

Hi Nell.

Thanks for your answer. I have tried both, but without success.

“vi ~/.aws/config” gave me

[default]
output = text
region = us-west-2

so I understand the region is properly setup.
I also have the limit increase approved from AWS for this region.
Isn’t it necesary to lounch the instance from AWS previously?

Thanks.

Nell · December 26, 2017, 4:59pm

Hi Victor,

what do you get when you do ‘aws configure get region’ ? If you get us-west-2 then there is something else wrong. If you still get eu-west-1 then try to run aws configure and add your secret key and access key there.

Victor1 · December 26, 2017, 9:18pm

I can see on the AWS console, the EC2 service limit in the EU west (Irland) for p2.xlarge instances is 1, as I requested and they approved. I have added my access key and my secret key to eu-west-1 region in Cygwin. And I get eu-west-1 with “aws configure get region” … But “bash setup_p2.sh” doen’t work yet. It still says “Only us-west-2 (Oregon), eu-west-1 (Ireland), and us-east-1 (Virginia) are currently supported”.

Any ideas.

Thanks

Nell · December 27, 2017, 5:19am

Hi Victor,

You need to create your ec2 instance in a different region since the fast ai AMI isnt there in that region. AMI’s are region specfic so you need to create your instance in a region that is supported which is either us-west-2, eu-west-1 or us-east-1. As far as what you need to change, you need to only change your region to one of those 3 to make it work.

Victor1 · December 27, 2017, 12:47pm

Hi Nell,

I have the EC2 instance limit of 1 for the us-west-2 region. I have deleted the user in the AWS console and created again. I have configured the AWS with new keys and us-west-2 region, but, unfortunately, I get the same results when runing bash setup_p2.sh.

Any other idea, please?

Thanks

WillManidis · December 29, 2017, 2:52am

Requested my AWS service increase (and had it approved) and have everything else running up to this step-- but when I run
bash setup_p2.sh
I get:
An error occurred (OptInRequired) when calling the CreateVpc operation: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.
usage: aws [options] [ …] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument --vpc-id: expected one argument
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument --vpc-id: expected one argument

An error occurred (OptInRequired) when calling the CreateInternetGateway operation: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.

and then a few hundred more lines of error finished with blank values for the results.

WillManidis · December 29, 2017, 8:19pm

So I’ve fixed my above problem, and have a new error-- the console outputs:
rtbassoc-b59dface
True
setup_p2.sh: line 13: /Users/willmanidis/.ssh/aws-key.pem: Permission denied

An error occurred (InvalidKeyPair.NotFound) when calling the RunInstances operation: The key pair 'aws-key' does not exist
Waiting for instance start...

cynosure · December 29, 2017, 8:50pm

possibilities:

The key pair on AWS doesnt exist (try recreating)?
You dont have assigned the permissions, in AWS console go to your instance in click on security groups, “fast ai security group”, under “Inbound” see if you have authorized your local machine. to make it sure click Edit and then Choose MyIP or Anywhere in the source and try again.
If you did 1 maybe the keys are not associated with your instance anymore ?

if it doesnt work, it might save you time to start over again and create AWS instance from scratch (but delete all the key pairs, elastic ip etc before).

klndeepak · December 29, 2017, 9:15pm

Dear Jeremy, Thank you for posting videos on deep learning, which are really useful. I have some issues in lesson 0 that is setting up AWS server. Everything went smoothly that is starting from configuring AWS to until the command $bash setup_P2.sh. When I ran this command in cygwin, it always says that maximum no. of attempts exceeded. I paste the message below. I would really appreciate if you could help me solving this error.
$ bash setup_p2.sh

An error occurred (VpcLimitExceeded) when calling the CreateVpc operation: The maximum number of VPCs has been reached.
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument

An error occurred (InternetGatewayLimitExceeded) when calling the CreateInternetGateway operation: The maximum number of internet gateways has been reached.
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --internet-gateway-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --route-table-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --group-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --group-id: expected one argument
setup_p2.sh: line 13: /home/klnde/.ssh/aws-key.pem: No such file or directory
chmod: cannot access ‘/home/klnde/.ssh/aws-key.pem’: No such file or directory
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --subnet-id: expected one argument
Waiting for instance start…

Waiter InstanceRunning failed: Max attempts exceeded
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --instance-id: expected one argument
securityGroupId=
subnetId=
instanceId=
instanceUrl=None
Connect: ssh -i /home/klnde/.ssh/aws-key.pem ubuntu@None

klnde@DESKTOP-KTQUM43 ~
$

cynosure · December 30, 2017, 12:21pm

IIRC there is a limit on VPCs (can found in AWS console)

Capture
Try deleting all the resources that you dont need and use (in AWS console) to see if it helps.

or delete everything and start over again.

klndeepak · December 30, 2017, 3:09pm

Hi, Thanks for help. I did exactly the same.
Below you can see that I completely deleted VPC files.

After that, I followed same instructions and still I get same error. Below I post. Would appreciate your help
klnde@DESKTOP-KTQUM43 ~
$ pip install awscli
-bash: pip: command not found

klnde@DESKTOP-KTQUM43 ~
$ pip2 install awscli
Requirement already satisfied: awscli in /usr/lib/python2.7/site-packages
Requirement already satisfied: PyYAML<=3.12,>=3.10 in /usr/lib/python2.7/site-pa ckages (from awscli)
Requirement already satisfied: rsa<=3.5.0,>=3.1.2 in /usr/lib/python2.7/site-pac kages (from awscli)
Requirement already satisfied: s3transfer<0.2.0,>=0.1.12 in /usr/lib/python2.7/s ite-packages (from awscli)
Requirement already satisfied: botocore==1.8.20 in /usr/lib/python2.7/site-packa ges (from awscli)
Requirement already satisfied: colorama<=0.3.7,>=0.2.5 in /usr/lib/python2.7/sit e-packages (from awscli)
Requirement already satisfied: docutils>=0.10 in /usr/lib/python2.7/site-package s (from awscli)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python2.7/site-packages (from rsa<=3.5.0,>=3.1.2->awscli)
Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == “2.6” or python_version == “2.7” in /usr/lib/python2.7/site-packages (from s3transfer<0. 2.0,>=0.1.12->awscli)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python2.7 /site-packages (from botocore==1.8.20->awscli)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/lib/python2.7/site -packages (from botocore==1.8.20->awscli)
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/site-packages (fro m python-dateutil<3.0.0,>=2.1->botocore==1.8.20->awscli)

klnde@DESKTOP-KTQUM43 ~
$ aws
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: too few arguments

klnde@DESKTOP-KTQUM43 ~
$ aws configure
AWS Access Key ID [****************DQPA]: AKIAIMJPMJM7CPJCDQPA
AWS Secret Access Key [****************7sbC]: zbBK/t4cP+WCFa/NqKS1j3pE8O+bj0jsJs Nk7sbC
Default region name [us-west-2]: us-west-2
Default output format [text]: text

klnde@DESKTOP-KTQUM43 ~
$ wget http://files.fast.ai/files/setup_p2.sh
–2017-12-30 09:49:30-- http://files.fast.ai/files/setup_p2.sh
Resolving files.fast.ai (files.fast.ai)… 67.205.15.147
Connecting to files.fast.ai (files.fast.ai)|67.205.15.147|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 2520 (2.5K) [application/x-sh]
Saving to: ‘setup_p2.sh.3’

setup_p2.sh.3 100%[===================>] 2.46K --.-KB/s in 0s

2017-12-30 09:49:33 (40.0 MB/s) - ‘setup_p2.sh.3’ saved [2520/2520]

klnde@DESKTOP-KTQUM43 ~
$ cat setup_p2.sh
export vpcId=aws ec2 create-vpc --cidr-block 10.0.0.0/28 --query 'Vpc.VpcId' -- output text
aws ec2 modify-vpc-attribute --vpc-id $vpcId --enable-dns-support “{“Value”:tr ue}”
aws ec2 modify-vpc-attribute --vpc-id $vpcId --enable-dns-hostnames “{“Value”: true}”
export internetGatewayId=aws ec2 create-internet-gateway --query 'InternetGatew ay.InternetGatewayId' --output text
aws ec2 attach-internet-gateway --internet-gateway-id $internetGatewayId --vpc-i d $vpcId
export subnetId=aws ec2 create-subnet --vpc-id $vpcId --cidr-block 10.0.0.0/28 --query 'Subnet.SubnetId' --output text
export routeTableId=aws ec2 create-route-table --vpc-id $vpcId --query 'RouteTa ble.RouteTableId' --output text
aws ec2 associate-route-table --route-table-id $routeTableId --subnet-id $subnet Id
aws ec2 create-route --route-table-id $routeTableId --destination-cidr-block 0.0 .0.0/0 --gateway-id $internetGatewayId
export securityGroupId=aws ec2 create-security-group --group-name my-security-g roup --description "Generated by setup_vpn.sh" --vpc-id $vpcId --query 'GroupId' --output text
aws ec2 authorize-security-group-ingress --group-id $securityGroupId --protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id $securityGroupId --protocol tcp --port 8888-8898 --cidr 0.0.0.0/0
aws ec2 create-key-pair --key-name aws-key --query ‘KeyMaterial’ --output text > ~/.ssh/aws-key.pem
chmod 400 ~/.ssh/aws-key.pem

export instanceId=aws ec2 run-instances --image-id ami-bc508adc --count 1 --ins tance-type p2.xlarge --key-name aws-key --security-group-ids $securityGroupId -- subnet-id $subnetId --associate-public-ip-address --block-device-mapping "[ { \" DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 128, \"VolumeType\": \"g p2\" } } ]" --query 'Instances[0].InstanceId' --output text
export allocAddr=aws ec2 allocate-address --domain vpc --query 'AllocationId' - -output text

echo Waiting for instance start…
aws ec2 wait instance-running --instance-ids $instanceId
sleep 10 # wait for ssh service to start running too
export assocId=aws ec2 associate-address --instance-id $instanceId --allocation -id $allocAddr --query 'AssociationId' --output text
export instanceUrl=aws ec2 describe-instances --instance-ids $instanceId --quer y 'Reservations[0].Instances[0].PublicDnsName' --output text
echo securityGroupId=$securityGroupId
echo subnetId=$subnetId
echo instanceId=$instanceId
echo instanceUrl=$instanceUrl
echo Connect: ssh -i ~/.ssh/aws-key.pem ubuntu@$instanceUrl

klnde@DESKTOP-KTQUM43 ~
$ bash setup_p2.sh

An error occurred (VpcLimitExceeded) when calling the CreateVpc operation: The m aximum number of VPCs has been reached.
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument

An error occurred (InternetGatewayLimitExceeded) when calling the CreateInternetGateway operation: The maximum number of in ternet gateways has been reached.
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --internet-gateway-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --route-table-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --vpc-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --group-id: expected one argument
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --group-id: expected one argument
setup_p2.sh: line 13: /home/klnde/.ssh/aws-key.pem: No such file or directory
chmod: cannot access ‘/home/klnde/.ssh/aws-key.pem’: No such file or directory
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --subnet-id: expected one argument

An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has be en reached.
Waiting for instance start…

Waiter InstanceRunning failed: Max attempts exceeded
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --instance-id: expected one argument
securityGroupId=
subnetId=
instanceId=
instanceUrl=None
Connect: ssh -i /home/klnde/.ssh/aws-key.pem ubuntu@None

klnde@DESKTOP-KTQUM43 ~
$

klnde@DESKTOP-KTQUM43 ~

cynosure · December 30, 2017, 4:45pm

It seems you could configure aws but there are several errors.

You haven’t installed pip
“home/klnde/.ssh/aws-key.pem: No such file or directory” It seems your user doesnt has permissions. try giving your user all permissions for this folder/file.

Did you deleted everything from your AWS account and started over again? By everything i mean all of the links that appear in the console;

PS : You are using 128 GB instance which is expensive. search on forum there is possible reduce the size of instance to reduce the costs.

klndeepak · December 30, 2017, 6:01pm

Hi,
Yes, I deleted everything and tried. You can see below the screen shot. Nothing appears.

Coming to the command pip, it did not work and hence I used pip2. Actually this information I got from the form. Anyway, I will uninstall Python and cygwin and start from the scratch. Hope it works. I am new to this
Thanks
Deepak

cynosure · December 30, 2017, 6:14pm

Sorry, i didnt mean reinstalling python and Cygwinn. I mean only AWS stuff. Just check your Python version. i am working with 2.7.

if Python version is correct, i dont think there is any problem with your local machine. its only the connection and setup settings with AWS that needs to be adjusted.

Did you tried giving permission to your usr (with which you login) to that key folder ?

PS: I worked with Oregon region and it worked fine. Maybe you can try changing the region?