How to scrape the web for images?

Part 1 (2018)
#3

Adding this here and will link to it from the post you mentioned.

Here is a simple project you can use to scrape images from a google search. It uses selenium and should only be used for educational purposes. Comments, recommendations, and pull requests are welcomed!

7 Likes
Challenges while creating your own dataset
#4

@wgpubs,even if we download the dataset with the above mention resource by you.then we have to label about 70 %(only suppose) of the data , for eg:: banana1.jpg, banana2.jpg .then reserve the 10% of the data for validation set as unlabeled and 20% of the data for test set as unlabeled,so please correct me if i’m wrong about the procedure

#5

You are correct.

You’ll have to write code to create the same directory structure used in the notebooks (e.g., /train, /valid, /test, /tmp, /models) that you see in the notebooks. From there you’ll have to move the images into sub-folders under /train … and from there, move a portion (20% or whatever) into /valid.

I also create a /sample directory and put a subset of everything in there for development. It makes it faster and allows you to debug things before using the full dataset.

You may want to look at the original part 1 notebooks as there is much more info there on how you can do the above.

#6

@wgpubs where i can find original part1 notebook and just a clarification validation set will consist of labeled data or unlabeled data

#7

Its up on the fast.ai github repo: https://github.com/fastai/courses

1 Like
#8

I sometimes use this chrome extension for downloading images.

17 Likes
Fast.ai v3 2019课程中文版笔记
Lesson 1 In-Class Discussion ✅
Deep Learning na Unb (Brasília) - Parte 1 - Lição 1
Lesson 1 official resources and updates ✅
Tips for building large image datasets
#9

incidentally, i was reminded of this thread by this one on HN today - apparently quite a hot topic: Ask HN: What are best tools for web scraping?

7 Likes
#10

I found this http://www.image-net.org/ website for images. Hope that helps other people.

2 Likes
#11

Nobody has mentioned Scrapy yet. Scrapy is a python webscraper/crawler. It has very extensive documentation and it is being used by multiple prominent companies. I am currently using it myself and I love it.

4 Likes
#12

Depends on how big your needs are. Here’s a good guide to using scrapy that you can really scale up. https://learn.scrapinghub.com/

1 Like
#13

  1. I believe that image sizes being different matters, I don’t recall if the fastai library corrects for these variations?

  2. If not, is there any resource or framework that I could reference in order to prep image sizes to feed into my Neural Network?

Apologies if these are basic questions :grimacing: I’m pretty new (but excited!) to everything here!

Ian

#14

This one can also be helpful https://www.pyimagesearch.com/2017/12/04/how-to-create-a-deep-learning-dataset-using-google-images/

3 Likes
#15

A semi-manual method to download Google images:

  • Go to Google.
  • Search for your term, say “sand”.
  • Filter the search for images.
  • Right click in the top blank part.
  • Click Save As.
  • Save it as “sand - Google Search.html” on the desktop.
  • The desktop will have this file and “sand - Google Search_files” folder.
  • That folder will have many “images(*)” files without extension.
  • To add jpg extension to these files:
    • Open Command Prompt.
    • Cd to desktop folder and then to “sand - Google Search_files” folder.
    • Write: “ren images* images*.jpg”.
    • Run it.
1 Like
Lesson 1 part 1 v2 custom images
#16

This seems to be a very nice script for scraping images from the web:

I found this thanks to a tweet by Sebastian Raschka.

7 Likes
#17

I found ScrapeStorm is useful. I think it is very simple and convenient for scraping webpage images. I recommend it to you.

1 Like
#18

This is awesome!

#19

What are the best options for scraping texts rather than images from Google please? I’d like to do something like what Jeremy showed in the earlier part of lesson 4 – not texts from arXiv, but from a Google keyword search. Any pointers will be greatly appreciated!

#20

In case anyone has similar interests - am resorting to good old lynx, and wget. Finding that better than the browser plugins and extensions I explored.

#21

I use this https://github.com/althonos/InstaLooter

though it limit how much you can download daily.

#22

Works really well, thank you!