Moving data and files between paperspace and local machine

Part 1 (2018) Beginner (2018)
21

Thanks a lot !

22

I use Filezilla although Iā€™d love to be able to mount a Paperspace drive to a letter drive in Windows.

23

Thanks for the suggestion. I can not get past the password. There are 2 IPā€™s; public and private. If I try private it just times me out. When I try public i get a ā€œPW authentication failedā€ error. There are two PWs; one for the paperspace login and one for the console login. I tried both and neither allowed me in, even though I can log into both paperspace and my console.

Update: I had to use ā€œpaperspaceā€ as my username (tried ā€œwhoamiā€ command). Then use PW for my console with public IP.

24

Thanks for the help!

25

Host: is just the fixed IP, port 22, Username ā€œpaperspaceā€, password from e-mail (or changed) for that machine (not the paperspace). On pop up Need to agree (not just once).

However, so far it works slow (may be due to connection, not the app).

26

Can anyone tell me how to pull file from github repo into paperspace? I follow the instruction from paperspace ( https://support.paperspace.com/hc/en-us/articles/360006416773-Using-a-Git-Repo). Then I got a .git file. I donā€™t know what to do with it. I am very new to github.

27

I would like to do the reverse, send the data from paperspace to my local machine. Has anyone done this?

The aim is to then commit and push the notebooks to my remote Github repo.

I have attempted to add and commit in Paperspace fastai GPU but Permission denied (publickey) so I thought if I could just get the data onto my PC then I can send it without publickey issues.

Thank you for any help itā€™s much appreciated.

28

@dataclass1 Rachel,

Here is what I do on my Windows 7 64-bit PC, using paperspace Gradient:

(1) Open your Gradient Notebook.
(2) Click on the "uploadā€™ button ā€“ itā€™s a bit below the top right of your screen
(3) Browse on your file system and select (click on) the file you want to upload
(4) Click ā€œopenā€
(5) when the small icon with the name of the file you want to upload appears, click the blue ā€œUploadā€ button.

Remember that only the files you place into the /storage directory on paperspace persist after you close the paperspace notebook.

I am guessing that a similar procedure would work on a mac or a windows 10 machine, but I could be wrong. Anyway, hope this helps.

1 Like
29

@jcatanza thank you. Iā€™m not using paperspace gradient notebooks just the fastai GPU machine. Good advice, the Gradient notebooks look good.

Iā€™ve just ended up using secure copy ā€˜scpā€™ through my Linux shell (terminal):
$ scp -r paperspace@184.[publiciphere]:/file(s)/to/send/from/paperspace/* /where/to/put/on/local/PC

And it works :slight_smile:

1 Like
30

On Gradient, if you have too many files to move manually using the ā€˜Uploadā€™ button, one way Iā€™ve found convenient is to zip all your data, upload it (itā€™s like uploading one file), then unzip on Gradient.

31

FYI, this doesnā€™t seem to work with Paperspace and the Gradient notebook. I tried to scp to the hostname of the running notebook but that didnā€™t work.

Maybe you have to pay for a public IP for this to work?

32

Glad to help. :+1:

33

Last time I used was back in April. I am not active on this course. However, few things you could try.

  1. Can you SSH into your box ? (paperspace)
    if yes

  2. Do you see any error when trying to SCP into your box ?
    if no

  3. then you should troubleshoot SSH before doing SCP.

  4. I did not get any public IP. I was using fastai notebook, not these gradient notebooks.

Thanks and let me know if this helps ?

34

I was faced with a problem to download a dataset in the virtual machine. These are some of the ways-

  1. First of all, we can use the untar_data method of fastai for download as well as untaring the data. It works well for the standard datasets used in the fastai course which are stored in the cloud in gzip format. However, this method cannot be used to download the file from google drive. (At least I couldnā€™t)

  2. There is code available in python which can extract any zip file, I have tested it.
    https://stackoverflow.com/questions/3451111/unzipping-files-in-python

  3. So, the problem boils down to downloading the datasets. We can use wget command to download the dataset. It works very fast, and I have downloaded the BACH test dataset which is of 3GB in less than 5 min. https://zenodo.org/record/3632035/files/ICIAR2018_BACH_Challenge_TestDataset.zip
    But this does not work for Google drive shared link directly.

  4. To download from Google drive, we can use the following bash script which uses the curl command-
    (NB: Just select this code, copy it and on the terminal paste Shift+Ctrl+V, or right-click and paste option.)

#!/bin/bash
fileid=ā€œfile id of Google drive shared fileā€
filename=ā€œWrite your filenameā€
curl -c ./cookie -s -L ā€œhttps://drive.google.com/uc?export=download&id=${fileid}ā€ > /dev/null
curl -Lb ./cookie "https://drive.google.com/uc?export=download&confirm=awk '/download/ {print $NF}' ./cookie&id={fileid}" -o {filename}

You must replace the file id with the google drive id of the file and the filename is the name of the file which u want to give the file in double-quotes. It must be made sure that the file is shared publicly (must have edit permission) then only it works. I have tested it and it works fine.


After downloading the zip file, you can unzip it with the tar command or method in step 2.
  1. The command wget can also be used to download the Google drive file.
    (Files > 100 Mb are large files) Also change docs.google to drive.google

For large files run the following command with necessary changes in FILEID and FILENAME:
wget --load-cookies /tmp/cookies.txt ā€œhttps://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate ā€˜https://docs.google.com/uc?export=download&id=FILEIDā€™ -O- | sed -rn ā€˜s/.confirm=([0-9A-Za-z_]+)./\1\n/pā€™)&id=FILEIDā€ -O FILENAME && rm -rf /tmp/cookies.txt

So, for small file run following command on your terminal:
wget --no-check-certificate ā€˜https://docs.google.com/uc?export=download&id=FILEIDā€™ -O FILENAME

https://bcrf.biochem.wisc.edu/2021/02/05/download-google-drive-files-using-wget/
It works, I have tested it.

  1. In addition, to Google drive you can also use wget to download a file from any link
    wget https://zenodo.org/record/3632035/files/ICIAR2018_BACH_Challenge_TestDataset.zip -O ā€˜ICIAR2018_BACH_Challenge_TestDataset.zipā€™

  2. Also, you can download using curl command also.
    curl -o ICIAR2018_BACH_Challenge_TestDataset.zip https://zenodo.org/record/3632035/files/ICIAR2018_BACH_Challenge_TestDataset.zip
    (example code for BACH dataset)
    https://www.howtogeek.com/447033/how-to-use-curl-to-download-files-from-the-linux-command-line/