Prepare a test suite
It should be easy to prepare a fake test suite that emulates the problem. For example, this is how I’d create a quick test suite (unix) to emulate the problem:
mkdir tests
echo -e "import os, os.path;\nif os.path.exists('./tmp1'): os.remove('./tmp1')\nif os.path.exists('./tmp2'): os.remove('./tmp2')" > tests/conftest.py
echo -e "def test_a(): assert True\ndef test_b(): assert True\ndef test_c():\n with open('./tmp1', 'w') as f: f.write('')\n assert True" > tests/test_1.py
echo -e "def test_a(): assert True\ndef test_b(): assert True\ndef test_c():\n with open('./tmp2', 'w') as f: f.write('')\n assert True" > tests/test_2.py
echo -e "from os.path import exists\ndef test_a(): assert True\ndef test_b(): assert True\ndef test_c(): assert not (exists('./tmp1') and exists('./tmp2'))" > tests/test_3.py
If you look at the resulting files, the sequence: tests/test_1.py::test_c tests/test_2.py::test_c tests/test_3.py::test_c will trigger a failure.
Let’s validate, by listing just the test modules, which by default will trigger this problem too.
pytest tests/test_1.py tests/test_2.py tests/test_3.py
and we get the failure indeed:
――――― test_c ―――――――
> def test_c(): assert not (exists('./tmp1') and exists('./tmp2'))
E AssertionError: assert not (True and True)
E + where True = exists('./tmp1')
E + and True = exists('./tmp2')
tests/test_3.py:4: AssertionError
This was on the test module level, but we want to get to the test level, so that the failing sequence is:
pytest tests/test_1.py::test_c tests/test_2.py::test_c tests/test_3.py::test_c
indeed it fails.
But if we swap the first test, so that it runs:
pytest tests/test_1.py::test_a tests/test_2.py::test_c tests/test_3.py::test_c
it works just fine - the test suite doesn’t fail.
We can also run the test suite in the test module order that doesn’t trigger the problem
pytest tests/test_3.py tests/test_2.py tests/test_1.py
this succeeds.
note that I created conftest.py which resets the state before each pytest run.