Setup problems: AWS


(Luis BLANCHE) #796

Hello,
I get the same error :

An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation: The image id ‘[ami-bc508adc]’ does not exist

I have created everything on the eu-west-1 server as intructed for people from Europe, so do I also need to change my region and ask for a p2 instance in euw too ?

Thank you in advance


(rony) #797

Hi,
I am stuck at installation/aws set up stage.
OS: Ubuntu 16.04 LTS [in a virtual box 4 GB ram: sand box set up :slight_smile: ],
installed anaconda & AWS CLI [ no issues ]
Created AWS account.

set up aws configure…

aws configure
AWS Access Key ID [****************45EQ]:
AWS Secret Access Key [****************v2H5]:
Default region name [us-west-2]:
Default output format [text]:

tried to set up a t2 instance… and later a p2 instance… neither worked out.
this is the error message I keep getting
----start error–
bash setup_t2.sh
True

An error occurred (InvalidKeyPair.NotFound) when calling the RunInstances operation: The key pair ‘aws-key-fast-ai’ does not exist

An error occurred (MissingParameter) when calling the CreateTags operation: The request must contain the parameter resourceIdSet
Waiting for instance start…

Waiter InstanceRunning failed: Max attempts exceeded
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --instance-id: expected one argument

An error occurred (MissingParameter) when calling the RebootInstances operation: The request must contain the parameter InstancesSet

All done. Find all you need to connect in the fast-ai-commands.txt file and to remove the stack call fast-ai-remove.sh

–end error message–

Other things I have tried:

  1. http://wiki.fast.ai/index.php/Starting_Over_with_AWS
  2. http://wiki.fast.ai/index.php/AWS_install -> common problems -> InvalidKeyPair.NotFound error
    both removing rm ~/.ssh/aws-key-fast-ai.pem… and setting up the ssh dir with modified permissions

Peculiarities noted:

  1. the file aws-key-fast-ai.pem was created in the ubuntu file system since the first attempt… however it was empty 0 bytes… always.
  2. the chmod 700 effectively reduced the permissions for .ssh fodler… it was 753 [or higher] before I decided to chmod 700 on the folder.
  3. elastic IP addresses and VPNs are being created in AWS for every run of the setup_t2/p2 script… and yes i am deleting them before the next run/attempt
  4. Instances and Key pairs are not being created/visible at the aws console.

any advice on what I am doing wrong…?
-Rony: India


(rony) #798

So this issue is resolved…
Root cause: the aws-key-fast-ai.pem key was a read only file
Try deleting it from the command line… you get prompted "Are you sure you want to delete the read only file… etc etc " hit enter… and you would think you are good to go.
what needs to be done is chmod 777 for the file aws-key-fast-ai.pem
then delete it…
then delete the /.ssh directory in which the above file was [for a good measure]

and then proceed with
cd ~
mkdir .ssh
chmod 700 ~/.ssh

as mentioned in http://wiki.fast.ai/index.php/AWS_install

someone may ask the question: why didn’t the script work at the first place? honest answer I don’t know. :slight_smile:


(Vadym Matviichuk) #799

Hi guys,
Is gtx 1080ti enough for this course or i need to create an aws server
Thanks


#800

plenty enough :slight_smile:

you might need to decrease the batch size maybe here and there but I think you should be able to fit any of the discussed architectures in memory just fine


#801

Hi Tubs. I have encountered the same error and you. Did you ever figure out what was wrong? Is so, what did you do to fix it?
Thanks!


(Jacob Nastaskin) #802

Hi Ii’m having issues getting AWS CLI installed on my MAC.
I ran
"$ pip install awscli --upgrade --user "

to install it, however, I’m getting

" -bash: aws: command not found "

when I try aws configure or aws --version. I’m guessing this might be because I need to add the path to the executable file to my PATH variable, as suggested in this AWS setup tutorial, http://docs.aws.amazon.com/cli/latest/userguide/installing.html, but am having trouble doing so.

Any help or advice on this issue would be greatly appreciated!


(Helmuth Stockmann) #803

I successfully set up a t2 and a p2 instance on AWS, but I was wondering if I can point them both to the same storage volume (so I don’t keep copying files around). Is that possible and if so how would I set this up?


#804

It is possible but not super straightforward I am afraid.

The easiest route is to create a separate volume and be attaching + mounting it as needed. Problem is, when you create an EBS volume you also need to partition it and make a filesystem on it…

You could google for the above actions or try to work out what needs to be done following my tutorial from step 11: https://becominghuman.ai/automated-aws-spot-instance-provisioning-with-persisting-of-data-ce2b32bdc102.

Actually, if you want to be attaching it manually, I think this howto by @slavivanov might be even better: https://blog.slavv.com/learning-machine-learning-on-the-cheap-persistent-aws-spot-instances-668e7294b6d8

Good luck! :slight_smile:


(Helmuth Stockmann) #806

thanks @radek


#807

Hi. I’d like some help with my setup.

Firstly, I have a setup with nvidia gtx 1060 (zotac) with 6 gb vram, 8gb ram and a core i3-6100 processor. Is it possible to create a server using my own pc? Or will be more of a hassle than to have an AWS account?

Also, being from India, should I still opt for US (West)?

Any guidance is appreciated.


#808

If your nvidia card supports CUDA (quite sure it does) you should be good to go.

Are you already running linux? If yes, then getting set up for the course will be relatively easy.

Picking the AWS region will only impact prices (they vary by region) and latency to the server - I would pick the location closest to you that has GPU units. For me, I have to go across the entire Europe to get to my AWS region, but the latency for typing etc is unnoticeable.

Good luck with the setup!


#809

Yes, I am running Linux and I just found a guide here to setup:http://forums.fast.ai/t/making-your-own-server/174/195

Just wanted to clarify: Does having AWS account make the course easier to follow than rolling your own setup?


#810

If you have the hardware available (like you do) probably running this on local machine is actually easier.


#811

I have downloaded the setup files from github and tried bash setup_p2.sh. I get errors that I don’t fully understand why they are happening. I’m new to AWS and will appreciate any help. I am pasting the error below:

###Begin paste
bash setup_p2.sh
True

An error occurred (InvalidKeyPair.NotFound) when calling the RunInstances operation: The key pair ‘aws-key-fast-ai’ does not exist

An error occurred (MissingParameter) when calling the CreateTags operation: The request must contain the parameter resourceIdSet
Waiting for instance start…

Waiter InstanceRunning failed: Max attempts exceeded
usage: aws [options] [ …] [parameters]
To see help text, you can run:

aws help
aws help
aws help
aws: error: argument --instance-id: expected one argument

An error occurred (MissingParameter) when calling the RebootInstances operation: The request must contain the parameter InstancesSet

All done. Find all you need to connect in the fast-ai-commands.txt file and to remove the stack call fast-ai-remove.sh
Connect to your instance: ssh -i /home/venkatesh/.ssh/aws-key-fast-ai.pem ubuntu@None

###End paste


(Jacob Nastaskin) #812

Hi jeeyung,
How did you solve this problem? I’m having the same issue.
Thanks.


#813

Where are you pasting this? This looks like a file that should be run from console at some point during setup (after aws-cli configuration).

Please refer to the Getting Started instructions on the course page. Here you can find a video with a walkthrough


#814

Thank you for the reply.

I am not pasting this – this is the error message I get. I get this message on running bash setup_p2.sh that I downloaded from GitHub page given in here –http://course.fast.ai/start.html after following the instructions given in this introduction video.


#815

Ok, sorry, I misread your post.

The first error that you are getting is this:

An error occurred (InvalidKeyPair.NotFound) when calling the RunInstances operation: The key pair 'aws-key-fast-ai' does not exi

I think it is raised by this part of the code:

if [ ! -d ~/.ssh ]
then
	mkdir ~/.ssh
fi

if [ ! -f ~/.ssh/aws-key-$name.pem ]
then
	aws ec2 create-key-pair --key-name aws-key-$name --query 'KeyMaterial' --output text > ~/.ssh/aws-key-$name.pem
	chmod 400 ~/.ssh/aws-key-$name.pem
fi

export instanceId=$(aws ec2 run-instances --image-id $ami --count 1 --instance-type $instanceType --key-name aws-key-$name --security-group-ids $securityGroupId --subnet-id $subnetId --associate-public-ip-address --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 128, \"VolumeType\": \"gp2\" } } ]" --query 'Instances[0].InstanceId' --output text)
aws ec2 create-tags --resources $instanceId --tags --tags Key=Name,Value=$name-gpu-machine

This is in setup_instance.sh which is called by the setup_p2.sh that you are running.

Can you check what keys you have in ~/.ssh? Could you also log into console on AWS and check the key pair named aws-key-fast-ai exists?

@jnastaskin - are you still experiencing this problem or did you find a solution?

I am using a different setup on AWS so unfortunately can’t run this myself to verify what’s happening there…

BTW this is the description of the error code from here:

The specified key pair name does not exist. Ensure that you specify the region in which the key pair is located, if it's not in the default region.

One guess just came to my mind now that I think of it!!! Maybe you have the key already in ~/.ssh and it is not being uploaded to AWS?

From your terminal please execute rm -f ~/.ssh/aws-key-fast-ai* and give it another go :wink:


(Vadym Matviichuk) #816

Hi, i want to use my own PC as a server, but i don’t really want to install linux in dual boot for now, are there any guides how to set up a server on Windows 10? Thanks