Improving/Expanding Functional Tests


(benedikt herudek) #24

understand, agree!. Will focus on these low level script asserts with above list for now. But I could well imagine one could come up with different layers of tests, like notebook tests, memory tests or more functionally orientated tests and so on …


(Kaspar Lund) #25

Here is a nice tool i have been using. Not suitable for automatic testin, but if you are suspicious about a piece of code then it is really usefull

> import tracemalloc
> tracemalloc.start()
> snapshot1 = tracemalloc.take_snapshot()
> 
> ---------your lines of code to be traced -------
> 
> snapshot2 = tracemalloc.take_snapshot()
> top_stats = snapshot2.compare_to(snapshot1, 'lineno')
> print(f"Top 10 of {len(top_stats)}")
> for stat in top_stats[:10]: print(stat)

(benedikt herudek) #26

@sgugger what would you think we create 2 additional methods in fakes.py called fake_text_data and fake_image_data. Then for e.g. fake_text_data would open and load a dummy file (like in test_text_data) and use that class to test one_item and other fastai functions that would not work with a TensorDataset. I.o.w. in test_basic_data I use fake_data, which uses TensorDataset to test e.g. one_batch. But other functions like one_item we test from test_text_data.

Or we find a way to create fake_data with not just TensorDataset but rather fast.ai functions - but not sure how - to do that.


#27

I’ve changed the way fake data is created. It’s still synthetic data, since it’s there to test functions in callbacks or basic train facility, but it’s fully compatible with the fastai library.


(benedikt herudek) #28

ok, thx! Will have a look latest after NYE to use for some testclasses.


Developer chat
(Stas Bekman) #30

5 posts were merged into an existing topic: Benchmarking fastai


(Stas Bekman) #32

FYI, I moved tests/fakes.py to tests/utils/fakes.py - and please refactor any reusable functions into test util modules as you write tests. Thank you.


(Stas Bekman) #33

added new tests:

I’m not quite sure how to update the listing as it doesn’t indicate parts that need to be covered.


(benedikt herudek) #34

great, thx ! Enough if you update here, have updated the list above and added your name (not ideal format for tracking, but good enough I guess)

I am working on more simple tests for the learner and hope to extend my PR soon … bloody day job eating up my time :wink:


(benedikt herudek) #35

@sgugger would you think there is a meaningful, at least useful test case for testing the split function on a linear model (as given in fakes.py)? Am aware of convnet and unet usages for example.

Will check some other functions meanwhile, small (almost trivial) PR with some incremental improvements here:

Was thinking btw, if we have meaningful test cases we might want to paste them in the docs as examples also. Had looked here for example usages of split, hadn’t found them and upon searching the forum found image examples for split.


(benedikt herudek) #37

@stas any ideas, how if to test split one fake data in a use- / meaningful way?


(Stas Bekman) #38

FYI, added CaptureStdout context manager for a much more compact stdout capture and clearing:


(benedikt herudek) #39

oh good, I will look at it. Quite some tests could work over screen output easily.

But here or there was a little reluctant to use it too much - imagine we change some trivial screen output in wording and the tests fail. So want to be a little careful, maybe we need a little strategy / best practices to ensure we do not add tests that break all the time.

Good stuff, thanks !


(Stas Bekman) #40

I don’t know, I have never used this.

Probably it will be easier to write meaningful test cases by prioritizing writing tests for bug reports, so then you always have a meaningful test case (assuming the report included enough of a setup to reproduce it). I’m not opposing your systematic approach, but perhaps a lot of those methods will almost never be used, so why not wait till someone asks about it, reports it not working, etc. and meanwhile focus on the small sub-set of the code that’s really important. i.e. following the 80-20% Pareto principal.


(benedikt herudek) #41

fair point, yes some of these might not too much add value - I meanwhile follow more the docs simply to test what is described.


(Stas Bekman) #42

I was in no way implying that it should be used as much as possible, it was just a refactoring step (prompted by your own refactoring recommendation) and I was sharing what was refactored. That’s all.


(benedikt herudek) #43

yes, get it! thx.


(benedikt herudek) #44

and thx for your tips and corrections @stas ! worthwhile mentioning here