Moving data and files between paperspace and local machine


(TRIDIB DUTTA) #1

I am not sure if this is the right place to ask. How do we move files and data between our local computer and paperspace ?


(Pranav Kanade) #2

I am using filezilla to move jupyter notebooks around.
Or
You can also use github repo for all your code and then can do push/pull on either side.
Or
From linux command line you can use : scp
for more on this can use : man/info scp
:smile:
cheerio…!!


(Shivam Goel) #3

you can also use WinSCP to connect between local and VM and then sharing almost anything. :slight_smile:


(Alexandre Cavalcante) #4

as pkanade said, filezilla may help you. I get it done using this tuto:

For me it worked great. I hope it works for you.


(Michael Cheich) #5

I think git is probably the best option, but a real quick and easy way is to use Google Drive.

If you set up google drive to share a folder on your local computer, then when you open jupyter notebook, you can upload directly from google drive.

When I am testing things like my own pictures of dogs, or simple things like that, it feels like a quick solution.


(TRIDIB DUTTA) #6

Thanks guys. I am using MacOS. I believe I will have to use SSH to move data to the paperspace machines. My data size is about 3GB.
Any thoughts.


(Sritanu Chakraborty) #7

You can use rsync.


(Subash Gandyer) #8

I have a local dataset of about 50 GB size. What’s the best way to upload into a paperspace instance for processing? What time frame am I looking here in completion of the uploading process? I am in a fix right now. I have uploaded a small sample and tried. Got a decent baseline accuracy but need the full dataset for improving my model’s accuracy. Thanks in advance. Suggestions, please.


(Ananda Seelan) #9

Paperspace recommends BTSync


(Dirk Liebich) #10

Can someone please provide a more detailed description on how to move files between a MAC and the Ubuntu machine. All my attempts have failed so far – I tried rsync, scp and FileZilla but in all cases seem to run into either authentication or other issues.

[edit] and BTSync is not really an option as I find it quite expensive. Paperspace should provide a reasonable way to do this - they mention copy & paste but that is not working at all. Tried on Mac and windows

thanks in advance


(Christian Tchou) #11

Personally, I used Putty : https://it.cornell.edu/managed-servers/transfer-files-using-putty


#12

Filezilla is another simple graphical option.
What you need is ‘Filezilla Client.’
You should run it and just type ‘sftp://[your_domain_or_ip]’ to the textbox, your ssh id/password and ssh port(it’s usually 22) to login to your server.
Note that the protocol is ‘sftp’ instead of ordinary ‘ftp.’ It will work if the server allows ssh access.

I just noticed that someone already mentioned Filezilla… Then maybe ftp -> sftp protocol was a mistake?


(Abid) #13

I did something like this

scp -rp ~/Downloads/img-data/* paperspace@65.49.78.236:/home/paperspace/data/test-data/

next you’ll get a prompt for password.

This should work.