Transfer files between two instances

Suppose I have two ec2 instances and I would like to transfer files between them, how can this be done preferably without charge?

I am assuming both the instances have to be in running state? I have tried a number of different things but none of them work: For example using this:
scp -i ~/.ssh/aws-key.pem ubuntu@ip-xx-x-x-xx.us-west-2.compute.internal:nbs/ ubuntu@ip-xxx-xx-x-xxx.us-west-2.compute.internal:nbs/
Basically scp -i <key> <source folder first instance> <destination folder second instance>

and variations of the above …
I have also tried to log into an instance and transfer files to another running instance but I keep getting the error of port 22: connection timed out. Does this have to do anything with security group settings? Pointers? @robin

You will need to ensure that the two hosts are in the same security group (By default every time you create a new instance EC2 will create a new security group, so watch out). And you will need to enable inbound TCP traffic to among hosts in the group. At Amazon we call this a “hostclass” which is a group of servers who work together.

This is a good overview:
https://blog.microideation.com/2016/05/26/secure-copy-files-scp-between-two-ec2-instances-in-aws/

Here is an explanation of security groups:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_SecurityGroups.html

2 Likes

Awesome … @brendan! Thank you … definitely the best guide I have seen so far. Will try it out …

@vshets @brendan I don’t claim to be an AWS expert, so take this with a grain of salt, but that doesn’t look right to me. Our security group by default has port 22 (ssh/scp) open to all IP addresses. So you should be able to scp from any AWS instance to any other that is configured in this way, right?

Yes, if you created both instances using Jeremy’s scripts then this should work. I see now that the script does set the same security group and open the SSH port 22.

I still think you need the private key (aws-key.pem) on the host you are starting from to get to the other host. I was unable to ssh between hosts until I copied my private aws-key to one of the hosts.

Then I could run:
ssh -i aws-key.pem ubuntu@ec2-35-161-92-82.us-west-2.compute.amazonaws.com

1 Like

Yes I created the instances manually after having observed the security settings from the result of running Jeremy’s setup file and duplicating these settings on to the newly created ones. So ports 22 on both my instances are open:
Inbound

Outbound:


I have not tried this but your last command looks like you are logging into the instance. Are you then scp ing files from one instance to the other.

Just FYI - rather than duplicating the security setting., you can simply add the new instances to your existing security group.

Yes SCP works fine.

I am typing this to increase my word count above the 20 word minimum.

1 Like

For some reason, that instance group setting originally created from the script does not show up on the list of available security groups. So I had to create a new default security setting and have been using that since.

This is an old thread but I wanted to share that when I was able to scp files between the 2 instances.
I copied the pem file into the instance requesting the file copy.

The 2 instances were also created with the same set up scripts so possibly the security groups are the same as well I havent checked. Hope this helps someone

So first from my local machine I copied over the pem file.
scp -i ~/.ssh/aws-key.pem ~/.ssh/aws-key.pem ubuntu@ec2-x-x-x-x.us-west-2.compute.amazonaws.com

Then from that instance I made the scp request to the src instance.
scp -i ~/.ssh/aws-key.pem ubuntu@ec2-y-y-y-y.us-west-2.compute.amazonaws.com:sf.zip ~/nbs/

3 Likes