Project:
Write a tool that can reduce a long sequence of failing tests:
pytest testmodule1::testa testmodule1::testb testmodule1::testc testmodule2::testa testmodule2::testb testmodule2::testc testmodule3::testa testmodule3::testb testmodule3::testc
to the minimal sequence that still fails, e.g.:
pytest testmodule1::testa testmodule2::testa testmodule3::testc
The problem is to find a minimum sequence of tests that leads to failure, starting from a long one.
Say you run make test
and one test fails (out of currently 240 tests). You run that test alone and it’s fine. You try to bisect the list of tests and it’s fine too. So it’s a certain combination of the previous tests that leads to that one test failure. One or more tests leave unclean state that affects that other test that fails because of that. So in a way we have a coupled tests situation.
The brute force approach is to take the list and remove one test at a time and retest, but we are talking about 240 tests at the moment, total run ~2min, so it’d be quite slow. bisecting is not the right approach either because a group could be involved. So what efficient algorithm would you suggest to reduce 240 to say a sequence that could be 5, 149, 200. There is no telling how many tests are involved, but most likely 2-3, which may help at choosing an efficient algorithm.
To top if off the failure is not consistent and happens once in 5 runs on average, so each sequence will need to be re-run multiple times, but that’s unrelated. It just makes the reduction process even slower.
The current failure I’m dealing with has to do with gpu mem measurements that seems to be affected by previous calls to gpu, so it probably has to do with some leaking objects and some tests not succeeding at a clean teardown.
I was hoping to find a pytest plugin that did that, but couldn’t find any.
Would anybody be interested to contribute such functionality? That would be very useful to the fastai project and surely to other projects using pytest as well.
This post shows how to create a test suite that reproduces the challenge that this project is trying to solve.
There are several parts to this interesting project and you’re welcome to contribute to any or all parts if you want the challenge and, of course, feel you can figure it out:
- Part 1a. find an efficient algorithm, see this post
- Part 1b. optimize to eliminate items with slowest execution time fist, see this post
- Part 2a. integrate it into pytest as a plugin, see this post
- Part 2b. Extend the plugin to generate the initial list itself, see this post
Thank you.