The discussion of resizing images in this weeks video, combined with doing some more work with IIIF recently sparked some interest in the possibility of using IIIF to resize images to prepare them for a training loop. One thing I noted is that some of the resize methods available in fastai are mirrored in the possible requests made via [ IIIF Image API 3.0 for example using IIIF request playground https://www.learniiif.org/image-api/playground
Original image https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/max/0/default.jpg
Can be resized to 250x250 in various ways
https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/250,250/0/default.jpg (squished aspect ratio)
https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/!250,250/0/default.jpg (preserved aspect ratio)
This means you could offload some of the resizing operations to an IIIF server rather than doing it during the training loop. As an example to load images from a Dataframe containing a column with IIIF URLs and some labels can be done in a datablock:
import requests
def get_im(x):
with requests.get(x,timeout=30) as r:
return PILImage.create(io.BytesIO(r.content))
iiif = DataBlock(
blocks=(ImageBlock, CategoryBlock),
splitter=RandomSplitter(valid_pct=0.2),
get_x=Pipeline([ColReader('iif'),get_im]),
get_y=ColReader('label'))
This is probably not usually a good idea for the training part, since there will be more latency from grabbing images via the web than a local filesystem, but it might be an option if you are already running a IIIF server which has a fast connection to the machine you are training on?
I created a crude notebook to test the differences in training speed: https://github.com/davanstrien/fastai4GLAMS/blob/master/lessons/03_lesson/iiif_resize_experiment.ipynb
One other issue is that in this simple approach to using IIIF images, is that if any of the URLs are invalid/don’t return an image/you have a network issue, the training loop will break. Unless you are working with a huge volume of images, it’s probably going to be better to use IIIF to download the images locally before training. This could still take advantage of the possibility of making images a reasonable size to start with and reducing the amount of resizing operation which happens in the training loop.