Moving data and files between paperspace and local machine

Part 1 (2018) Beginner (2018)
1

I am not sure if this is the right place to ask. How do we move files and data between our local computer and paperspace ?

2

I am using filezilla to move jupyter notebooks around.
Or
You can also use github repo for all your code and then can do push/pull on either side.
Or
From linux command line you can use : scp
for more on this can use : man/info scp
:smile:
cheerio…!!

2 Likes
3

you can also use WinSCP to connect between local and VM and then sharing almost anything. :slight_smile:

1 Like
4

as pkanade said, filezilla may help you. I get it done using this tuto:

For me it worked great. I hope it works for you.

5

I think git is probably the best option, but a real quick and easy way is to use Google Drive.

If you set up google drive to share a folder on your local computer, then when you open jupyter notebook, you can upload directly from google drive.

When I am testing things like my own pictures of dogs, or simple things like that, it feels like a quick solution.

6

Thanks guys. I am using MacOS. I believe I will have to use SSH to move data to the paperspace machines. My data size is about 3GB.
Any thoughts.

7

You can use rsync.

8

I have a local dataset of about 50 GB size. What’s the best way to upload into a paperspace instance for processing? What time frame am I looking here in completion of the uploading process? I am in a fix right now. I have uploaded a small sample and tried. Got a decent baseline accuracy but need the full dataset for improving my model’s accuracy. Thanks in advance. Suggestions, please.

9

Paperspace recommends BTSync

1 Like
10

Can someone please provide a more detailed description on how to move files between a MAC and the Ubuntu machine. All my attempts have failed so far – I tried rsync, scp and FileZilla but in all cases seem to run into either authentication or other issues.

[edit] and BTSync is not really an option as I find it quite expensive. Paperspace should provide a reasonable way to do this - they mention copy & paste but that is not working at all. Tried on Mac and windows

thanks in advance

1 Like
11

Personally, I used Putty : https://it.cornell.edu/managed-servers/transfer-files-using-putty

12

Filezilla is another simple graphical option.
What you need is ‘Filezilla Client.’
You should run it and just type ‘sftp://[your_domain_or_ip]’ to the textbox, your ssh id/password and ssh port(it’s usually 22) to login to your server.
Note that the protocol is ‘sftp’ instead of ordinary ‘ftp.’ It will work if the server allows ssh access.

I just noticed that someone already mentioned Filezilla… Then maybe ftp -> sftp protocol was a mistake?

13

I did something like this

scp -rp ~/Downloads/img-data/* paperspace@66.44.73.236:/home/paperspace/data/test-data/

next you’ll get a prompt for password.

This should work.

11 Likes
14

Very easy way to copy anything from your Windows PC to any remote server is to use SFTP, for example freeware tool like Bitvise SHH Client is supporting it.

https://www.bitvise.com/ssh-client-download

Using Bitvise you can connect (SSH) to your Paperspace and then choose New SFTP window

image
image.png632×580 44.7 KB

From there you can simply copy paste, drag and drop, or use any copying metod supported by Windows. Paperspace servers are quite quick so I had no problem moving even bigger data over there.

image
image.png892×508 47.1 KB

15

@aramay - you Sir, are a legend! Just started copying my files after an hour or so of various unsuccessful attempts.

16

Thanks - this (plus googling the basics of scp) was very helpful! I transferred stuff out of my paperspace to my local machine and didn’t get prompted for a password. I was already logged into paperspace, so maybe that’s wihy? Either way, it worked smoothly!

17

This is so helpful!!!
THANKS!

18

More detailed description for those like me who struggled to connect SFTP client.
Option 1:

  1. Install FileZilla or WinSCP.
  2. For session use the following:
  • File protocol - SFTP
  • Host name - public IP address of your machine
  • Port number - 22
  • User name - paperspace
  • Password - password you received in email

Option 2 (useful for moving small files):
Open Jupyter notebook on VM and navigate to direction needed. Use “Upload” button in the right upper corner to upload a file. To download file check the check-box left to the file name and press “Download” button.

1 Like
19

This is generally a good solution. However, rsync is probably a better option under most circumstances. scp copies everything, rsync essentially just copies a diff.
try something like:
rsync -ravP path/to/my/file(s) paperspace@publicIPonterminal:/path/to/destination/

2 Likes
20

Set up an SSH between the machines and copy through SCP