Deep Learning Utilities

I’ve written (or tweaked) several utility scripts that I have used for the first several lectures. Nothing too fancy. Perhaps others can post scripts that they have found useful.

ai_utilities

A set of scripts useful in deep learning and AI purposes, originally for use with fast.ai lectures and libraries.

image_download.py

Download images (typically limited to 1000) from a specified serach engine, currently Google or Bing.
image_download.py is useful in several respects:

  • Because is utilizes selenium, it is not limited by the search engine api and generally allows for more downloaded images.
  • It can operate in headless mode, which means it can be used on a server without access to a gui browser.
  • The default browser is Firefox. The script can be modified to use other browsers such as Chrome.
usage: image_download.py [-h] [--gui] [--engine {google,bing}]
                         searchtext num_images

Select, search, download and save a specified number images using a choice of
search engines

positional arguments:
  searchtext            Search Image
  num_images            Number of Images

optional arguments:
  -h, --help            show this help message and exit
  --gui, -g             Use Browser in the GUI
  --engine {google,bing}, -e {google,bing}
                        Search Engine, default=google

Example: image_download.py 'dog' 200 --engine 'bing' --gui

Notes:

  1. Requires Python >= 3
  2. Install selenium: conda install selenium or pip install selenium
  3. Install other dependencies from conda
  4. Install an appropriate browser and browser driver (appropriate for your browser and operating system) in PATH.
  5. For example, if using Ubuntu and Firefox:
  • tar xfvz geckodriver-v0.19.1-linux64.tar.gz
  • mv geckodriver ~/bin/, where ~/bin is a dir in PATH

make_train_valid.py

usage: make_train_valid.py [-h] [--train TRAIN] [--valid VALID] [--test TEST]
                           labels_dir

Make a train-valid directory and randomly copy files from labels_dir to sub-
directories

positional arguments:
  labels_dir     Contains at least two directories of labels, each containing
                 files of that label

optional arguments:
  -h, --help     show this help message and exit
  --train TRAIN  files for training, default=.8
  --valid VALID  files for validation, default=.2
  --test TEST    files for training, default=.0

Example: make_train_valid.py catsdogs --train .75 --valid .20 --test .05

filter_img.sh

Use file to determine the type of picture then filter (keep) only pictures of a specified type.

Images are filtered in place, i.e., non-JPEG files are deleted. (This can be modified within the script.)

Usage: filter_img image_directory

Example:filter_image dogs/

7 Likes

Hi,
I am trying to download images using the image_download.py script and I get the following error:
SessionNotCreatedException: Message: Unable to find a matching set of capabilities
Any ideas what might have gone wrong?
Thanks
JJ

Try:

  1. Requires Python >= 3
  2. Install selenium: conda install selenium or pip install selenium
  3. Place a browser driver appropriate for your browser and operating system in PATH.
  4. For example, if using linux and Firefox:
  • tar xfvz geckodriver-v0.19.1-linux64.tar.gz
  • mv geckodriver ~/bin/, where ~/bin is a dir in PATH
1 Like

Solved this issue by downloading firefox, with the first 3 steps metnioned here https://medium.com/@griggheo/running-selenium-webdriver-tests-using-firefox-headless-mode-on-ubuntu-d32500bb6af2

script runs properly, but I max at about 800 images with the following message:

> Less images found: Message: Element <input id="smb" class="ksb _kvc" type="button"> could not be scrolled into view

Any ideas how to fix?

@jjlevinemusic, I hit the same limit using selenium/firefox. Though selenium/firefox allows for more downloads from google/images than a direct api call (which caps out around 200 images), it does appear there is a cap around 750-800 images. I’ve replicated this by trying to use the selenium/chromedriver with similar results. Perhaps Bing has a higher cap. I will take a look.

I’ve updated several scripts in Deep Learning Utilities.
Notably, image_download.py can use both google.com/images and bing.com/images for search engines. I’ve refactored the script so as to simplify adding new image search engines. It can now be used as follows:

image_download.py 'dog' 500 --engine 'bing' --gui

If anyone would like me to try to add others, let me know.

I’ve also updated make_train_valid.py so that it is possible to specify the proportion of the original files be used by train, valid and test. For example, given a directory:

catsdogs/
         ..cat/[*.jpg]
         ..dog/[*.jpg]
make_train_valid.py catsdogs --train .75 --valid .25

Creates the following directory structure:

catsdogs/
         ..cat/[*.jpg]
         ..dog/[*.jpg]
         ..train/
                 ..cat/[*.jpg]
                 ..dog/[*.jpg]
         ..valid/
                 ..cat/[*.jpg]
                 ..dog/[*.jpg]

I wrote several basic functions to more easily explore the fastai code base. These are useful to quickly inspect the attributes_of and the methods_of objects. These and several other useful utilities can be found at GitHub ai_utilities

Here are example usage:

ai_utils.py

ai_utils.py
contains:
atttributes_of(obj, *exclude): -> prints obj attributes
methods_of(obj, lr=False):      -> prints obj methods

usage: import ai_utils

> data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
> attributes_of(data.trn_dl.dataset, 'fnames')
c: 2
fnames: ...
is_multi: False
is_reg: False
n: 23000
path: data/dogscats/
sz: 224
y: [0 0 0 ... 1 1 1]

> methods_of(data.trn_dl.dataset)
denorm(arr):
get(tfm, x, y):
get_c():
get_n():
get_sz():
get_x(i):
get_y(i):
resize_imgs(targ, new_path):
transform(im, y=None):
1 Like

First of all thank you for the scripts!

Is there an easy way to incorporate this on a fast.ai paperspace machine?

Best regards
Micheal