Improving/Expanding Functional Tests


#147

I’ve changed the way fake data is created. It’s still synthetic data, since it’s there to test functions in callbacks or basic train facility, but it’s fully compatible with the fastai library.


(benedikt herudek) #148

ok, thx! Will have a look latest after NYE to use for some testclasses.


Developer chat
(Stas Bekman) #149

FYI, I moved tests/fakes.py to tests/utils/fakes.py - and please refactor any reusable functions into test util modules as you write tests. Thank you.


(Stas Bekman) #150

added new tests:

I’m not quite sure how to update the listing as it doesn’t indicate parts that need to be covered.


(benedikt herudek) #151

great, thx ! Enough if you update here, have updated the list above and added your name (not ideal format for tracking, but good enough I guess)

I am working on more simple tests for the learner and hope to extend my PR soon … bloody day job eating up my time :wink:


(benedikt herudek) #152

@sgugger would you think there is a meaningful, at least useful test case for testing the split function on a linear model (as given in fakes.py)? Am aware of convnet and unet usages for example.

Will check some other functions meanwhile, small (almost trivial) PR with some incremental improvements here:

Was thinking btw, if we have meaningful test cases we might want to paste them in the docs as examples also. Had looked here for example usages of split, hadn’t found them and upon searching the forum found image examples for split.


(benedikt herudek) #153

@stas any ideas, how if to test split one fake data in a use- / meaningful way?


(Stas Bekman) #154

FYI, added CaptureStdout context manager for a much more compact stdout capture and clearing:


(benedikt herudek) #155

oh good, I will look at it. Quite some tests could work over screen output easily.

But here or there was a little reluctant to use it too much - imagine we change some trivial screen output in wording and the tests fail. So want to be a little careful, maybe we need a little strategy / best practices to ensure we do not add tests that break all the time.

Good stuff, thanks !


(Stas Bekman) #156

I don’t know, I have never used this.

Probably it will be easier to write meaningful test cases by prioritizing writing tests for bug reports, so then you always have a meaningful test case (assuming the report included enough of a setup to reproduce it). I’m not opposing your systematic approach, but perhaps a lot of those methods will almost never be used, so why not wait till someone asks about it, reports it not working, etc. and meanwhile focus on the small sub-set of the code that’s really important. i.e. following the 80-20% Pareto principal.


(benedikt herudek) #157

fair point, yes some of these might not too much add value - I meanwhile follow more the docs simply to test what is described.


(Stas Bekman) #158

I was in no way implying that it should be used as much as possible, it was just a refactoring step (prompted by your own refactoring recommendation) and I was sharing what was refactored. That’s all.


(benedikt herudek) #159

yes, get it! thx.


(benedikt herudek) #160

and thx for your tips and corrections @stas ! worthwhile mentioning here


(benedikt herudek) #161

submitted a PR for inspecting and asserting over fit and fit_one_cycle, hope it makes sense.

All ears, if one can improve these tests here.

@sgugger


(benedikt herudek) #162

Btw, there seem to be same regression tests broken in vision, when running this locally with a fresh pull. Getting in trouble with a.o. these lines in test_vision_data.py

from fastai.vision.data import verify_image
import PIL
***#import responses***

test_verify_image

So, while the individual tests of my changes work and the checks of my above PR works an azure, locally having errors.

Not sure what it is, tried some simple de-comments of culprit lines -didn’t solve it. Happy to help, if you give me some pointers.

@stas FYI


(Stas Bekman) #163

Please provide the output of the failing tests and any pertinent info that you think will help reproduce the problem. I have no problem running the test suite and neither azure CI. You can check how the CI is setup https://github.com/fastai/fastai/blob/master/azure-pipelines.yml (choose the the entry that’s similar to yours and compare with how yours is different).


(benedikt herudek) #164

ok, thx for the homework :wink: am pretty sure its a local problem and will check the yml setup. Let’s assume the issue is closed, otherwise would come up with an analysis here.

thx @stas


(Stas Bekman) #165

Do we have any testing-suitable datasets with variable image sizes? or perhaps it’d be handy to add an autogenerator in fakes.py? or perhaps taking MNIST_TINY and making a variable image size copy of it by random cropping it - might be handy for other testing? MNIST_TINY_VAR_SIZE?

I have a half-baked test for tests/test_vision_data.py which works, but needs to also test on variable image size, it’s really a resize/collate_fn test:

def test_from_name_re_resize(path, capsys):
    fnames = get_files(path/'train', recurse=True)
    pat = r'/([^/]+)\/\d+.png$'
    # check 3 different size arg are supported and no warnings are issued
    for size in [14, (14,14), (14,20)]:
        data = ImageDataBunch.from_name_re(path, fnames, pat, ds_tfms=None, size=size)
        captured = capsys.readouterr()
        assert len(captured.err)==0, captured.err

Test Registry project
(benedikt herudek) #166

cool @Stas and thx a lot! I

I wouldn’t know if we have such testing-suitable datasets, I really only scan the existing tests to answer such a question. Maybe @sgugger knows?

IMHO: All for extending fakes.py as we can reuse this then.