Reduce size of volume

Hi @vshets,

Thanks a lot for your effort in creating AMI with reduced size. However, for some reason I can’t access slack (I only tried today though) through the slack link (updated one) provided on forum. I wonder whether you could paste your solution steps on forum as well, since forum is the first place we are recommended to go.

Thank you for your hard work on this issue.

Daniel

@vshets has an ‘s’ at the end of his name, so just fixing it here so he gets a notification…

thanks Jeremy

Hi @Daniel … sorry for the late reply. I will have to rejig my memory to recall the exact steps but this thread from the start is an attempt to create an ami from scratch. Pls let us know if you are running into specific issues.

Thanks @vshets

Hi Daniel,

I was able to use a smaller EBS by following the following steps:

  1. Stop and terminate all EC2 instances. Then go to the EBS section and delete/detach all volumes.
  2. Get the latest scripts from the fastai github directory onto your local machine with git. This depends on how exactly you got the scripts from the fastai github in the first place. I use mobaterm to ssh into a permanently running linux VM, and used git fetch to update the repo. You could just re-download the zip from github if you are using windows.
  3. Copy either setup_p2.sh or setup_t2.sh to setup.sh (I did cp setup_t2.sh setup.sh on linux)
  4. Open setup.sh and change two lines – change ami to the new ami that @vshets created, and change the VolumeSize to 30.

See below for the diff (the lines starting with < are from the new setup file I created):

ubuntu@ip-172-31-13-157:~/courses/setup$ diff setup.sh setup_t2.sh
12d11
<    export ami="ami-9c54f4fc" # Oregon
56c55
< export instanceId=`aws ec2 run-instances --image-id $ami --count 1 --instance-type t2.xlarge --key-name aws-key-$name --security-group-ids $securityGroupId --subnet-id $subnetId --associate-public-ip-address --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 30, \"VolumeType\": \"gp2\" } } ]" --query 'Instances[0].InstanceId' --output text`
---
> export instanceId=`aws ec2 run-instances --image-id $ami --count 1 --instance-type t2.xlarge --key-name aws-key-$name --security-group-ids $securityGroupId --subnet-id $subnetId --associate-public-ip-address --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 128, \"VolumeType\": \"gp2\" } } ]" --query 'Instances[0].InstanceId' --output text`
ubuntu@ip-172-31-13-157:~/courses/setup$

The final step is to run bash setup.sh

Hope this helps you. The VM needs a minimum EBS of 30GB. I am in the us-west-2 region and it costs $3 USD a month.

8 Likes

Thank you very much @nunb, it works on my Mac as well.

Hi @jeremy,
I am trying to access the mentioned AMI created to build a reduced volume but I cannot access the Slack. I tried to use the link mentioned by @rachel to receive an invitation but I never receive the link to slack in my email. So, I am struggling to try to create a new AMI. Any help on this?
Regards.

It would be great if the AMI and smaller volume were part of the setup scripts in the repo

@nunb I tried your steps and got the instance running. After ssh’ing in, I ran install-gpu.sh. However, running nvidia-smi fails. I get the message “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.”

@jeremy any ideas?

1 Like

what type of instance are you using @stickperson?

@nunb t2. Are p2 instances the only ones with GPUs?

yes – you need a P2 instance to use the nvidia-smi command.

If you are using a T2 the nvidia command won’t work. you will have to configure keras to use cpu (it’s a one line change) and in general everything will run, but just slower. there is a procedure to request p2 instances … you have to open a case with AWS support.

Ah, easy enough. Might give the t2 instance a try. If it’s less than 5x slower price wise it’s worth it.

For training it will be a lot slower than 5x. There is also the older G2 instance, but not that much cheaper and a much weaker GPU.

Can you provide line that needs to be added/changed in order to get this to work?

1 Like

@nunb @vshets I am a newbie to AWS. I am trying to use the ami-9c54f4fc AMI in us-east-1 but get this error possibly since the AMI is not there in us-east . How do I get the AMI to us-east-1… Is there a corresponding version for us-east-1
**An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation: The image id ‘[ami-9c54f4fc]’ does not exist. I tried coying over from Oregon to Virginia but then I got an error that I do not have access

Thanks for your help.

Perhaps just change your region to us-west-2 (Oregon)

I changed the region to us-west-2, it still did not work.

Try ami-31ecfb26 for Virginia in p2 instance.