How to download a dataset to PaperSpace

[Zora]

Hi Guys,

I am learning practical deep learning for coders course. I am new to the field, and I am stuck on how to call or download my dataset for practice.

I am using a Gradient notebook in Paperspace. I uploaded my data to PaperSpace, and I followed all the steps explained in PaperSpace tutorials here https://github.com/alexacarlson/DeepDesign2019/blob/master/paperspace_tutorials/Paperspace_uploadingdata.md#creating-a-jupyter-notebook-in-paperspace

Now, when I try to download my dataset, I am wondering how I have to do this part. I used the fastai documentation (https://docs.fast.ai/data.external
https://docs.fast.ai/tutorial.vision). However, none of those commands seem working in my case. I am always getting this error that there is not a directory with this name or length of files giving me zero all the time. Can you please help me which command I have to use to download my dataset when I uploaded a dataset to PaperSpace?

Also, I am always confused about using those datasets that are available by default inside the fastai library. I know that we used it in the first lesson of the course, but still, I think the name of the dataset (the list of the name: https://docs.fast.ai/data.external) I am using with class URLs not working all the time(error: there is not a directory with this name). Can anyone help me what my mistake is?

P.S. Besides, I read the following topics from the forum, however, I think I need more explanations here.

How to download a dataset from a URL?

So, I was working with week 1 notebook and I have a question. How can I download a dataset and work on it? In my case, I want to download the Devanagari dataset which is like the MNIST dataset but for Hindi alphabets. I found the dataset on the UCI ML repository here: https://archive.ics.uci.edu/ml/machine-learning-databases/00389/ So, I want to know how can I download it and start working on it. I have tried the untar_data() and download_data() functions but it’s not working for me. Any help …

Datasets: downloading images for multiple classes

I’ve been using this notebook (by @lesscomfortable) to get together my datasets about towers of the world. However, with more than a few classes (using 12), I found it hard to just get all the folders created and the data download without the need of my manual intervention (1 by 1). So… I went ahead and made this little code to loop it. Write down the classes (names) you will be gathering data for. Follow the notebook’s first instructions and get your urls (file) from google images. Upload…

Thank you,

[vivekharshey1 (Vivek Harshey)]

I was faced with a problem to download a dataset in the virtual machine. These are some of the ways-

1. First of all, we can use the untar_data method of fastai for download as well as untaring the data. It works well for the standard datasets used in the fastai course which are stored in the cloud in gzip format. However, this method cannot be used to download the file from google drive. (At least I couldn’t)

2. There is code available in python which can extract any zip file, I have tested it.
   https://stackoverflow.com/questions/3451111/unzipping-files-in-python

3. So, the problem boils down to downloading the datasets. We can use wget command to download the dataset. It works very fast, and I have downloaded the BACH test dataset which is of 3GB in less than 5 min. https://zenodo.org/record/3632035/files/ICIAR2018_BACH_Challenge_TestDataset.zip
   But this does not work for Google drive shared link directly.

4. To download from Google drive, we can use the following bash script which uses the curl command-
   (NB: Just select this code, copy it and on the terminal paste Shift+Ctrl+V, or right-click and paste option.)

#!/bin/bash
fileid=“file id of Google drive shared file”
filename=“Write your filename”
curl -c ./cookie -s -L “https://drive.google.com/uc?export=download&id=${fileid}” > /dev/null
curl -Lb ./cookie "https://drive.google.com/uc?export=download&confirm=awk '/download/ {print $NF}' ./cookie&id={fileid}" -o {filename}

You must replace the file id with the google drive id of the file and the filename is the name of the file which u want to give the file in double-quotes. It must be made sure that the file is shared publicly (must have edit permission) then only it works. I have tested it and it works fine.

How to download a Google Drive url via curl or wget

After downloading the zip file, you can unzip it with the tar command or method in step 2.

5. The command wget can also be used to download the Google drive file.
   (Files > 100 Mb are large files) Also change docs.google to drive.google

For large files run the following command with necessary changes in FILEID and FILENAME:
wget --load-cookies /tmp/cookies.txt “https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate ‘https://docs.google.com/uc?export=download&id=FILEID’ -O- | sed -rn ‘s/.confirm=([0-9A-Za-z_]+)./\1\n/p’)&id=FILEID” -O FILENAME && rm -rf /tmp/cookies.txt

So, for small file run following command on your terminal:
wget --no-check-certificate ‘https://docs.google.com/uc?export=download&id=FILEID’ -O FILENAME

https://bcrf.biochem.wisc.edu/2021/02/05/download-google-drive-files-using-wget/
It works, I have tested it.

6. In addition, to Google drive you can also use wget to download a file from any link
   wget https://zenodo.org/record/3632035/files/ICIAR2018_BACH_Challenge_TestDataset.zip -O ‘ICIAR2018_BACH_Challenge_TestDataset.zip’

7. Also, you can download using curl command also.
   curl -o ICIAR2018_BACH_Challenge_TestDataset.zip https://zenodo.org/record/3632035/files/ICIAR2018_BACH_Challenge_TestDataset.zip
   (example code for BACH dataset)
   https://www.howtogeek.com/447033/how-to-use-curl-to-download-files-from-the-linux-command-line/

[Zora]

@vivekharshey1

Thank you very much!

[vivekharshey1 (Vivek Harshey)]

Glad to help.