Lesson 3 - Windows commands to upload kaggle dataset aren't working

Running
! mkdir %userprofile%.kaggle
! move kaggle.json %userprofile%.kaggle

creates a folder called ‘%userprofile%.kaggle’ and then errors on the second line withh the message “/bin/sh: 1: move: not found”

I’m not sure what this is supposed to do, even after researching it a little bit. Is it supposed to create a folder called .kaggle in my user directory or in the dl1 directory with the rest of the course materials? If I knew this, at least I could maybe do it manually.

Thank you,
Warren

I’ll breakdown the commands for you

mkdir %userprofile%.kaggle

mkdir is the command for creating a folder. %USERPROFILE is a system environment variable in Windows that points to the User’s Directory in C:\ drive. Taking example of my user, the user directory for me is C:\Users\Vishesh. So this command would basically resolve to C:\Users\Vishesh.kaggle in my case, which would be incorrect as it would basically create a folder Vishesh.kaggle under my C:\Users directory. What you want to do here is create a directory .kaggle under C:\Users\Vishesh. So, the correct command would be:

!mkdir %userprofile%\.kaggle

The second command just moves the file kaggle.json to the directory that you just created. But for this to work, you need to be in the directory where the file is present or specify the complete path to the file, otherwise it would not work. I would assume the file is downloaded to the Downloads folder for user so the command would be:

!move %USERPROFILE%\Downloads\kaggle.json %USERPROFILE%\.kaggle

Why we are doing this is because Kaggle CLI expects the kaggle.json, the file that contains the API key to your Kaggle account be placed in the folder:

C:\Users\<your_username>\.kaggle

You can do this manually as well. Should work fine.

1 Like

I am facing the same issue
I have put the path right but still getting the error /bin/bash: move: command not found.
So i did it manually. I created .kaggle folder in my user directory and then moved manually the kaggle.json file into the .kaggle folder, but i can’t download the dataset from kaggle.
Any solution?
Thanks

1 Like

Hi, thank you for suggesting the propper commands. These must have been updated becuase those were the commands I found in the notebook.

! mkdir %userprofile%\.kaggle
! move %USERPROFILE%\Downloads\kaggle.json %USERPROFILE%\.kaggle

The file didn’t move into the folder.
So after manually placing the kaggle.json file into:
course-v3 / nbs / dl1 / %userprofile%.kaggle

This output still came back
/bin/sh: 1: move: not found

:thinking:

I thought if I manually moved the .json file it might work anyways.
So moving on down the notebook,
When running

! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train-jpg.tar.7z -p {path}
! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train_v2.csv -p {path}
! unzip -q -n {path}/train_v2.csv.zip -d {path}

This error appears. :no_entry:
OSError: Could not find kaggle.json. Make sure it’s located in /home/ubuntu/.kaggle. Or use the environment method.

Here is the full message.

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/bin/kaggle", line 6, in <module>
    from kaggle.cli import main
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/kaggle/__init__.py", line 23, in <module>
    api.authenticate()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/kaggle/api/kaggle_api_extended.py", line 149, in authenticate
    self.config_file, self.config_dir))
OSError: Could not find kaggle.json. Make sure it's located in /home/ubuntu/.kaggle. Or use the environment method.
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/bin/kaggle", line 6, in <module>
    from kaggle.cli import main
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/kaggle/__init__.py", line 23, in <module>
    api.authenticate()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/kaggle/api/kaggle_api_extended.py", line 149, in authenticate
    self.config_file, self.config_dir))
OSError: Could not find kaggle.json. Make sure it's located in /home/ubuntu/.kaggle. Or use the environment method.
unzip:  cannot find or open /home/ubuntu/.fastai/data/planet/train_v2.csv.zip, /home/ubuntu/.fastai/data/planet/train_v2.csv.zip.zip or /home/ubuntu/.fastai/data/planet/train_v2.csv.zip.ZIP.

I am having the same issue on Google Colab. Any recommendations on how to fix this?

For Google Colab

1- Download the kaggle.json file from your Kaggle account
2- Upload your kaggle.json to Colab
3- Run the following commands

!mkdir -p ~/.kaggle
!cp kaggle.json ~/.kaggle/
!chmod 600 ~/.kaggle/kaggle.json

4- Create your data folder (e.g. planet like in lesson3-planet)

path = Config.data_path()/'planet'
path.mkdir(parents=True, exist_ok=True)
path

5- Download the competition data file (it will be saved in the path folder:)

! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train-jpg.tar.7z -p {path} 
1 Like

Thank you. Where should the json file be uploaded to “Colab”, where should this be uploaded - in the same directory as the Notebook? I am asking because, it is not able to find this file

image

Also, I am assuming that I need to mount the Google Drive before running these commands. Let me know if this is not the case.
image

The other thing to note is that !ls ~ does not show anything so I am not sure if that is used for mkdir, it will work.

image

But !ls … seems to be pointing to some sort of root on unix. Not sure if this should be used instead or not.
image

Upload your kaggle.json by following the steps described in the image here below:

You don’t have to mount your google drive. For large dataset, it’s faster to train your model by downloading your dataset in the Colab virtual machine (your data and your fastai program are on the same virtual machine). The downside is downloading them every time you open your notebook.

The code works. But can you explain how it works?

I ran the same gives an error 401: forbidden. Please resolve. Thankyou

Hi Farid,

So grateful I came across your instructions above (I’m working in Colab because I had similar issues with Paperspace). I still have issues progressing from the step above. I can see the uploaded file but when I run the commands that follow, I get these errors
image
image
image
image

GPC Notebook error

Hi Richard,

sorry, I’m not able to zoom in your screenshots except the last one. I’m not able to read the error messages.

Hi Farid,

Basically it looks like the data files are no longer on the site.
I cut and pasted the screenshots into 2 separate word documents but I don’t see a way to attach them,



It’s not possible anymore to directly download the zip file using the kaggle api. Here below is a solution that was proposed in the following topic that gets around this problem:

Lesson 3 - Can’t Download Planet Data Images Tar Archive Part 1 (2019)

Hi all! For those of you that want to download directly on their notebook server: This assumes you are logged in into Kaggle and you have accepted the conditions of the competition. Open the Chrome browser on your local machine Install the cookie.txt extension from this link Go to the Kaggle dataset Locate the download button button of the dataset you want (see example in image below) Copy the link (right-click on that button) Export your cookies using the newly added plugin Go to your rem…

I hope you will find it helpful.

1 Like