Another thing we need to revisit right away is randomness in tests. I don’t think this is going to fly.
@sgugger, one of the test runs broke during one of the CI runs because accuracy fluctuates from run to run:
tests/test_vision.py F [100%]
=================================== FAILURES =================================== ___________________________ vision::mnist end to end ___________________________ def test_mnist_end_to_end(): untar_data(MNIST_PATH) data = image_data_from_folder(MNIST_PATH, ds_tfms=(rand_pad(2, 28), )) learn = ConvLearner(data, tvm.resnet18, metrics=accuracy) learn.fit(1) > assert accuracy(*learn.get_preds()) > 0.98 E assert tensor(0.9799) > 0.98 E + where tensor(0.9799) = accuracy(*[tensor([[ 2.0705, -1.8396],
but it’d be very silly trying to chase that lowest possible accuracy and it’d be a mistake since something could change in the code to actually make things worse and we won’t know it.
So it’s very likely that we need to run all tests with a fixed seed. And once set, it should never be changed, since the test writers will rely on those numbers.
It’s probably the best to set it globally in
conftest.py and be it known that tests aren’t random. And a huge warning do not change the seed value. I propose 42 as a well rounded, though unoriginal seed value
Hmm, how to you create a random seed then if you needed to do so after you fixed the seed? I guess through a sub-process - or is there a magical way to produce a non-deterministic random number after the seed is fixed in python/numpy/torch?
[sidenote: In fact would it be a good idea to run
min < result < max test so that when an optimization happens it breaks the test for a good reason. I guess that is not the greatest idea - just want to make sure we detect both degradations and improvements]
And discussion aside,
tests/test_vision.py needs attention.