Test Registry project

fastai
1

DRAFT DESIGN (SOME NOTES, SUMMARISING ABOVE)

Idea: a simple mapping from API calls to the corresponding tests and vice versa.

Incentives for this initiative:

  1. having more runable examples of how to use a certain API in addition to docs nbs
  2. entice more tests to be written where tests are lacking

api2test - find which tests, if any, exercise/test a given API function.

Examples of what one may type into a notebook:

doctest('ImageDataBunch.from_name_re')

returns:

  • pytest -sv -k test_from_name_re tests/test_vision_data.py
  • [link into GitHub tests folder to the test script]
doctest('Learner.fit_one_cycle')

returns:

  • pytest -sv -k test_fit_one_cycle tests/test_train.py
  • [link into GitHub tests folder to the test script]

Note, the user has to run these tests locally her-/himself

Users should be able to type in the shortest function like: doctest('fit_one_cycle'), and only need the class name if it’s a method that appears in more than one class. But the function should be able to give the same results for any level of qualification. For example for the unique function fit_one_cycle:

doctest('basic_train.Learner.fit_one_cycle')
doctest('Learner.fit_one_cycle')
doctest('fit_one_cycle')

docs.fast.ai built-in link

in addition to [Source] link the docs will now have [Test] link, which will give the user a pop up with the same information as if they were to run doctest('this api function name').

test2api - how a test registers that it exercises a given set of functions

Example:

def test_from_name_re(path):
    this_tests(ImageDataBunch.from_name_re, …)
    # normal test continues here

def test_lr_find(learn):
    # can be anywhere in the test, in case objects appear later in the test
    this_tests(learn.lr_find)

this_tests():

  • will automatically derive the fully qualified function name if an alias or short imported form is passed (e.g. could pass learn.fit and it should sort it out - this would be preferred way since it’ll register the correct subclass this way if any)
  • will automatically derive the test module and test name using inspect (inspect.currentframe().f_back)
  • will add the map entries into a map db

A test may register one or more functions. But tests shouldn’t register functions already tested elsewhere. (e.g. fit() is in 100s of tests.

data collection and doc integration

collection

make test needs to be instrumented to collect the output of all test_this() calls into a single map db (e.g. json file), so that it can later be used by doctest(). The docbuilder (fastai/gen_doc/*) will use it to integrate the testing information into the API docs.

The map file should probably be saved under fastai and added to MANIFEST.in so that it’s installed with the .py files if we want doctest() to work on user’s machines, which may or may not have a fastai git checkout. i.e. it should be in the distribution. or perhaps the first stage is where we only make this feature available via docs.fast.ai then we don’t need to worry about it right away.

To simplify things the map should probably be only updated on make test which will be instrumented with extra commands to save the map db. And only if it was 100% successful, otherwise the previous copy of the map db is preserved.

Normal pytest will not do anything special. reason: when running a single test module we don’t want to break the map db.

coding

Package: this is all docs, so some place under fastai.gen_docs - probably just stick it in one of the existing modules, we can move them later if need be.

map file format/contents

The elements that will be made available are:

test_file, test_name, fully qualified API function, e.g.:

test_vision_data.py, test_from_name_re, vision.data.ImageDataBunch.from_name_re

questions:

  • probably no need to add fastai. in the function name? or should we add it in case we document some other APIs? Andrew?

notes:

  • one api function may have multiple corresponding test_names,
  • and vice versa any test_name may have multiple API functions it exercises.
  • if there are no tests registered for a given function, we will have a placeholder explaining how to write such tests.

.json format? a flat list of entries?

2

159 posts were merged into an existing topic: Improving/Expanding Functional Tests

3

wanted to run an idea by you @stas @sgugger.

Imagine we had a function test(class.function) which you call from the notebook and it

  • gives you a widget showing the testclass and ideally also a cmd to be able to run it.
  • could also be simpler, e.g. a pop similar like the doc() functionality or
  • maybe super simple to start: just an additional link in the current doc(…) to the test class, including an instruction how to run (actually, that could be part of the top code comments) the test class.

Good tests can help quite a bit in understanding functionality and this way the testscripts would get surfaced in the notebooks easily for everyone.

Maybe that gets folks to help improve testscripts. (I wish I had more time - at least managed to get some tests for fit / fit_one_cycle - So, also thinking how maybe one can organise more interest to help from others)

Could that be a good idea?

4

24 posts were merged into an existing topic: Improving/Expanding Functional Tests

5

It’s an interesting idea, Basically, you are saying here is the API and here it is in action. That would be very handy.

This is already done partially in the docs_src notebooks which currently acts as a testing platform in addition to unit tests.

The problem is how do you replicate the required env on a static website that github pages is - i.e. it’ll have to be done elsewhere - not on the current website.

1 Like
6

In the notebook of the person running it? So whatever environment that person uses.
Like when you use doc() you say test () and show the source code.

To run it can pytest be run from a notebook? Simply like

! ls - ltr

You say

! pytest xyz

And in the comment of the test you describe how to use it

Or nicer:

7

I think that would be very complicated, remember a lot of tests aren’t just a simple function, but have a bunch of decorators, functions that need to be pre-run, etc.

The simplest solution I can think of is that the API could give a user the exact unit tests to run and look at, but they will still need to do it from the fastai repo.

So for example if you do test-ala-doc() for ImageDataBunch.from_name_re, it will tell you:

pytest -sv -k test_from_name_re tests/test_vision_data.py

and perhaps a link to github file where that test is (of course there could be many tests).

I’m just not sure how this would help the effort to have more/better tests. Either there will be a ready test to show to the user, or there won’t be one, do you think the user who doesn’t know anything about the fastai testing will get all fired up to write the missing test? Just as well simply correlate test coverage with docs and have an indicator next to API doc at how much this API is covered.

Otherwise it’ll take resources to change all tests to co-operate with this “gateway” idea, but I’m not sure it will lead to more/better tests - just as well use the resources to write more tests.

And as I shared earlier, the best way to write tests IMHO is when you create new functionality and when you get a bug report. So it’s the latter we really need to put an effort in - help the user to write a test with her bug report. That way the effort is invested where it’ll have the highest payoff - where the actual need is.

1 Like
8

understand, thx valid points.

the primary idea is: if you can see or even run the test in a notebook, its a great documentation. if you see asserts at least for me, it is clear what a class ‘really’ should do, independently of the details of how things are implemented. So it is a useful part of Documentation.

Having folks write test scripts, is I admit more wishful thinking :wink:

Am all happy to write test scrips differently, frankly writing test scripts on bug reports can be quite a challenge (well, depends and sometimes it is super easy) depending on a bug. Even though I fully agree that is for sure where tests add the most value.

I probably overlook all kinds of complications for running pytest from a notebook via command line, maybe should try testing. Also potentially an option (but complex I suppose): there is this azure pipeline when committing running all tests … maybe could be reused, but I suppose that would also potentially overload the server if all students just get access.

1 Like
9

the primary idea is: if you can see or even run the test in a notebook, its a great documentation. if you see asserts at least for me, it is clear what a class ‘really’ should do, independently of the details of how things are implemented. So it is a useful part of Documentation.

I agree, it’d be an excellent feature!

That’s why I suggested this:

Same as [Source] link links to where the code can be read, the [Test] link will tell you which tests exercise this particular API, including an exact way to run it (as quoted above). So that the user can see the desired API in action.

Then each test will need to register which API it is testing and the docs builder will use that info to map the API to the tests.

If down the road this can be given to a user on a saucer with a silver lining - i.e. run-in-a-box, that would be great. But it’d require the same initial effort to map tests to API docs. So just as well start low-tech.

1 Like
10

looks good ! start small sounds the best approach (run-in-box could follow) I could try to help with this little project, if useful and not urgent (as I have this long lead times to finish things with time constraints). But feel free to also go for it if you want it fast.

Design Questions I see for the simple start::

  • function name test(XYZ) - just like doc(xyz) ? Or should we add it to the doc() function?
  • where would the test, register which API it is testing: can we just add it into the source code as a comment with a tag easy to parse for like RUNTEST? This way the classes itself are also nicely documented
  • doc builder functionality? So we extend that I suppose and its somewhere in github?
  • think somewhere (in the doc builder?) we need to list all tests or can we just say everything in the test folder and per naming convention we know which test class tests which class?
  • …? …
11

addition to the run-in-box feature:

I just uploaded the tests folder into the notebook folder layout next to the tools folder. Then in a notebook I run:

!pytest tests/test_basic_train.py -s

just seems to work fine basically. I do not see the entire output I want to see (just the success message) and when I ran !pytest I did get errors

But essentially if we give the user a print out of what command (s)he should run and place all updates tests in her/his folder the run-in-box should be simple enough?

12

Ah, you were asking for it, I’m just giving you feedback.

Design Questions I see for the simple start::

  1. function name test(XYZ) - just like doc(xyz) ? Or should we add it to the doc() function?

I don’t know how doc works, but similar to whatever it does.

  1. where would the test, register which API it is testing: can we just add it into the source code as a comment with a tag easy to parse for like RUNTEST? This way the classes itself are also nicely documented

I’d say it should not be a comment, but code, which also needs to validate that it exists - since it’s too easy for comments to start mismatching the API, as the latter changes.

So if a test looked like:

def test_from_name_re(path):
    [...]
    data = ImageDataBunch.from_name_re(path, fnames, pat, ds_tfms=(rand_pad(2, 28), []))

now it’ll become

def test_from_name_re(path):
    testing(ImageDataBunch.from_name_re, ...)
    [...]
    data = ImageDataBunch.from_name_re(path, fnames, pat, ds_tfms=(rand_pad(2, 28), []))

and testing() will need to figure out the the full fastai.vision...ImageDataBunch.from_name_re
and using inspect it will automatically derive the test module filename and the test name.

testing can receive more than one function that’s being tested, but focus only on the unique test (i.e. no passing fit in all tests that use fit, just the function that is actually being tested) - i hope it makes sense, so we only have a few tests per API entry.

so similar to make coverage, make test will dump some kind of map file, which the docbuilder will use to integrate the testing information into the API docs.

and testing is probably to generic of a name, should think of something short and untuitive. tapi? (test api?)

this is just an idea, of course, to how to implement it with the least amount of noise…

  1. doc builder functionality? So we extend that I suppose and its somewhere in github?

fastai/gen_doc/* - you might have to ask help of Andrew who designed most of it.

  1. think somewhere (in the doc builder?) we need to list all tests or can we just say everything in the test folder and per naming convention we know which test class tests which class?

see my reply to your comment (2).

And if you decide to go with it, please also ask @jeremy and @sgugger that they are on board with it.

1 Like
13

Thx a lot @stas for all the input! Really great tips

1 Like
14

idea is on test(train) would identify the class test_train.py and open a widget with below 3 tips.

I just download the test folder to the instance next to the tools folder, so can also run it.

Could be useful ? (at least for me it would be ;-)) @sgugger @stas
–> took out pics, cmd not useful

15

Based on your demo I have no idea how this would be useful. It looks like you’re doing some profiling code - I have no idea why.

Most unit tests are opaque and have neither useful output nor graphical output, other than failing or not failing, so I don’t know what you’re trying to accomplish.

At the moment we just need to write more tests. I personally am not sure how building widgets will help manifest that.

16

Making the test code and a call stack visible in the notebook wouldn’t be useful? Maybe it’s just me - as good as examples in the doc and nice to have easily accessible.

Never mind :wink: Just a few after thoughts:

  • Example, prototype: Profiling here as such just an example to make the call stack visible. Agree the profiling of test scripts do not make sense - so this would go out unless I find a different visualisation. The widget is just the UI part, could be any other way to surface this on the UI, e.g. similar as the doc() function works.

  • Alternatively, a simple link in the doc() to the source of the test class (not just the source of the code itself), might already help.

17

This spec content moved to the lead post of this thread Doc_test project

2 Likes
18

Thanks for the summary, @Benudek.

Made into wiki with changes to match the original proposal and adding more thoughts to it.

Not sure whether testapi+doctest or test2api+api2test would be a better match for the 2 functions. Thinking more about it, I think the first set as it’s in the proposal is better, since one might get confused with symmetrical function names - is it api2test or test2api. and doctest is like doc, so this is a more intuitive extension.

Do note that you won’t always be able to call doctest('Learner.fit_one_cycle') without quotes, since it’s not guaranteed the function is imported into user space. So using quotes there is probably a safer approach. Inside the test I think quotes won’t be needed, since the namespace will be guaranteed to be imported as it’s being tested. In fact, as I added in the notes, we must not use the quotes in the test, since we want to make sure the function exists. But perhaps there are some unknowns there too - need to test in practice.

1 Like
19

This is such a cool idea :slight_smile:

1 Like
20

Kudus to @stas, I just summarised his comments… .:slight_smile: I try to come up with some code for it soon

1 Like