Case study: Writing a new unit test and a doc entry for image resize functionality
Recently, I was doing some training setup that involved variable sized images and got stumbled with it not working. I was only able to find examples here and there and even forums weren’t helpful. So since I needed this problem to be solved, I decided to first write a few simple tests so that I could report the bug, and have it resolved. I submitted a similar bug report earlier, but @sgugger couldn’t find what the problem was without me giving him some reproducible code to work with, which I initially failed to provide.
Part 1: Writing the test
footnote: In case you don’t know in the fastai test suite we use small subsets of real datasets, so that the test execution completes within seconds and not hours. These are the datasets that have _TINY in their name, so as of this writing in fastai/datasets.py you will find: COCO_TINY MNIST_TINY MNIST_VAR_SIZE_TINY PLANET_TINY CAMVID_TINY - these are the ones you want to use for testing.
Apparently everything worked just fine as long as transforms were involved, but without transforms it’d just break. And then I still wasn’t very clear on why some examples used the datablock api whereas others factory methods, yet, working with the same dataset. It was quite confusing.
So I started with a simple test running with a fixed size dataset that I knew will work, since I pretty much copied an existing working test, and added some extra verifications to it that weren’t there originally.
from fastai.vision import *
path = untar_data(URLs.MNIST_TINY) # 28x28 images
fnames = get_files(path/'train', recurse=True)
pat = r'/([^/]+)\/\d+.png$'
size=14
data = ImageDataBunch.from_name_re(p, fnames, pat, size=size)
x,_ = data.train_ds[0]
size_want = size
size_real = x.size
assert size_want == size_real, f"size mismatch after resize {size} expected {size_want}, got {size_real}"
and it worked.
In this test, I setup the data object just like it’s done in the first lessons of the fastai course, and then I check the size of the first object of the train dataset and check that it indeed got resized. I hope you’re with me so far.
The assert does the checking and the last part of the assert is setup to give me a meaningful debug information in the case of the failure. You will see later how it becomes useful.
So this was my baseline and then I could start doing experiments with it by changing things around.
Next, I pretty much did the same thing, but with a variable image size dataset:
path = untar_data(URLs.MNIST_VAR_SIZE_TINY)
and it worked too.
Then I replaced the factory method from_name_re:
data = ImageDataBunch.from_name_re(p, fnames, pat, size=size)
with the data block API:
data = (ImageItemList.from_folder(p)
.no_split()
.label_from_folder()
.transform(size=size)
.databunch(bs=2)
)
and it worked with the fixed images dataset, but it failed with the variable size images dataset.
So I submitted a bug report and someone else did a similar one and he had a great test case that reproduced the problem, and meanwhile I decided to expand the test to cover all the various sizes - int, square and non-square tuples, resize methods and types of datasets. First I did it separately for each way of doing it and then started to slowly refactor it to avoid duplicated code. (duplicated code often leads to bugs.)
After many iterations (many of which were just broken), the many tests morphed into a complete unit test that covered 18 different configuration permutations and did it in both possible ways of performing a resize - (1) with the factory method and (2) data block API. Here it is:
# this is a segment of tests/test_vision_data.py
from fastai.vision import *
from utils.text import *
rms = ['PAD', 'CROP', 'SQUISH']
def check_resized(data, size, args):
x,_ = data.train_ds[0]
size_want = (size, size) if isinstance(size, int) else size
size_real = x.size
assert size_want == size_real, f"[{args}]: size mismatch after resize {size} expected {size_want}, got {size_real}"
def test_image_resize(path, path_var_size):
# in this test the 2 datasets are:
# (1) 28x28,
# (2) var-size but larger than 28x28,
# and the resizes are always less than 28x28, so it always tests a real resize
for p in [path, path_var_size]: # identical + var sized inputs
fnames = get_files(p/'train', recurse=True)
pat = r'/([^/]+)\/\d+.png$'
for size in [14, (14,14), (14,20)]:
for rm_name in rms:
rm = getattr(ResizeMethod, rm_name)
args = f"path={p}, size={size}, resize_method={rm_name}"
# resize the factory method way
with CaptureStderr() as cs:
data = ImageDataBunch.from_name_re(p, fnames, pat, ds_tfms=None, size=size, resize_method=rm)
assert len(cs.err)==0, f"[{args}]: got collate_fn warning {cs.err}"
check_resized(data, size, args)
# resize the data block way
with CaptureStderr() as cs:
data = (ImageItemList.from_folder(p)
.no_split()
.label_from_folder()
.transform(size=size, resize_method=rm)
.databunch(bs=2)
)
assert len(cs.err)==0, f"[{args}]: got collate_fn warning {cs.err}"
check_resized(data, size, args)
It may look complicated, but it’s very very simple - it does exactly the same simple things I described at the beginning of this post, just tests them in 18 different ways, via 3 loops! Remember, it was written in stages and slowly improved upon.
The only new thing that I haven’t covered so far is the CaptureStderr context manager, that we have in our test utils, which helps us test whether fastai emitted some warnings, which most of the time indicates that there is a problem waiting to happen. Therefore, the test needs to make sure our data is setup correctly and doesn’t emit any warnings (this check is done by a function called sanity_check()). So this check performs the validation that nothing was sent to stderr:
assert len(cs.err)==0, f"[{args}]: got collate_fn warning {cs.err}"
You can do the same using pytest’s capsys method, but ours is a better one, because it’s a context manager, and as such it’s more of a “scalpel”, whereas capsys is a bit of a “hammer” - when it comes to localization of the stderr capturing.
I submitted the test that included also:
@pytest.mark.skip(reason="needs fixing")
def test_image_resize(path, path_var_size):
...
because it was failing. So then we know the that this test needs fixing, and it doesn’t affect our CI (Continuous Integration) checks.
The next morning, Sylvain fixed the bug, removed the test skip directive and voila - we now have the resize without transforms covered 100% and it will never break in future released versions, because the test suite will defend against it.
You can see this test as it was submitted here.
If you want to run this test, you’d just do:
pytest -sv -k test_image_resize tests/test_vision_data.py
And in case you didn’t know - we have a testing guide, which is full of useful notes.
Part 2: Writing the resize documentation
Now I will tell you a big secret. The main reason I write documentation is primarily for self-serving reasons. I’m a lazy person and I don’t like figuring things out all the time. I enjoy the process of figuring out something once, but repeating the same figuring out is just exhausting. Therefore I tend to write down everything I think I might use again in the future. That’s why I write a lot of docs. I am happy to share them with anybody who wants them, but their main use is for myself. Making it public also ensures that if I lose my copy, I can restore it later from the public copy.
Same goes for tests, I write tests so that I don’t need to figure out why my code stopped working when a new release that included some change in the fastai library broke previously working functionality. By writing tests I ensure a future peace of mind for myself. And others benefiting from it is a nice side effect. And my ego is happy!
So back to our case study, now that I wrote this test I knew everything I needed to know about the resize functionality in the fastai library (the user side of it). And since there is so much complex ever changing tech stuff I need to cope with on the daily basis, I know I will forget this hard earned knowledge, so I decided that it’ll pay off to invest a bit more time to write a summary of what I have learned.
And so I did, and now there is a new entry that documents all the possible ways you could resize images in fastai: https://docs.fast.ai/vision.transform.html#resize
It’s literally the same as the test that I wrote, except it’s done in words and organized for an easier understanding.
Then I realized that resizing images on the fly is very inefficient if you have a lot of them and they are large. Therefore I expanded that section explaining how to resize images before doing the training, stealing one example from Jeremy’s class notebook, so there was an example in python and sharing the command line way using imagemagick method I normally use. (and that part of the doc entry could use more examples and more ways of doing that including pros and cons of those different ways. hint, hint).
Conclusion
So you can now tell that most likely before you write documentation you need to understand how the API or a use case you are about to document works, and since the only way to really understand something is by using it you will have to write some code. And if you’re going to write some code anyway, why not write a unit test for the fastai library.
If it’s a use case, involving some specific actions then a small or large tutorial on the steps to repeat your process is called for.
Its totally possible - just be careful the python stuff is fun, so you might stop watching netflix and need to discipline yourself to not drop your day job 
