It should be possible to do what Jeremy did in Firefox with Google Dev tools. Link to full example here https://lornajane.net/posts/2013/chrome-feature-copy-as-curl
But as Jeremy mentioned, it might not be as reliable
Yeah when I tried it on Kaggle, it didn’t show the request. If it works for others, let me know - perhaps it’s just some version or config specific issue. I’ve used it fine on other sites.
Is there a compelling reason to use this curl method over scp?
I think one reason might be to directly get the data to AWS rather than downloading the data to your local machine and then using scp to transfer the data to AWS.
1 Like
Makes sense… thanks!
1 Like
@daschumacher my best guess is that you can
- literally just copy and paste into bash on AWS
- no need to deal with two copies of the data as mentioned just before
1 Like
Some datasets can be 50GB or more, so you probably don’t want to download that over your home internet connection!..
1 Like
I found a solution for chrome users who want to curl the data set.
There is this chrome extension that you can install. It can be found here.
https://chrome.google.com/webstore/detail/curlwget/jmocjfidanebdlinpbcdkcmgdifblncg?hl=en
It will generate a curl/wget command for you.
To use it, Download the file (you can change the chrome settings to ask you to download so you dont have to download the file)
then click on the extension, and it will give you the curl/wget command for your terminal
5 Likes
Thanks David, this extension is super useful!