Nbdev code coverage / n-Tests

#1

Hi everyone,

I have been shifting my full-time work to use nbdev. I have been using it for developing some of our microservices and tooling. Because I have been using nbdev, I have had the most detailed documentation for my code out of my whole team. It has been awesome and I swear by nbdev now for both work and side projects.

With this said, using nbdev for production development highlighted some areas in testing that seem to be a tad limited. Before nbdev, we used pytest for developing and testing our APIs. One of the things that makes the rest of my team apprehensive in using nbdev is the inability to truely understand “how tested” my code is.

There are 2 features in pytest that would be cool to have in nbdev:

  • output the number of tests that have been run
  • provide an estimate / function for failing and passing based on code coverage

Outputting the number of tests run would be doable. Code coverage I can see being more complex.

#2

To explore a few ideas before suggesting any changes to nbdev, I’m thinking I could write a script to create test_[notebook name].py files from notebooks - like nbdev_build_lib but containing just the test cells.

This would make is possible to use coverage with something like:
coverage run test_00_core.py

If you’d be interested in trying this out on your projects, I’ll try to find time (o:

#3

I would still be interested. I will likely be able to do work/coding on my side also possibly after the first 2 weeks of July.

Seems that nbdev doesn’t generate test files themselves. If I remember, it seems to run the notebook and extract the failure outputs(?). It seems theres technically no such thing as a test cell, it’s just a cell that doesn’t get exported to python files. There’s also issues where I use the hide tag for test related code I don’t want exported to html. And also note, that “non test code” cells are still needed to run tests usually.

I’m actually starting to think that doing coverage might not actually be too bad to try to get working. Would coverage allow executing other arbitrary scripts? We could export ALL code in a given notebook to a python script and just run with coverage.

I’ll be pessimistic with my free time prob until mid this month. I like the idea of posting mvp stuff here so if I can I’ll offer code ideas/snippets also after mid july.

#4

If all of your tests run in plain python, this should get you going https://github.com/pete88b/decision_tree/blob/master/80_test_coverage.ipynb

if you’re able to share any of your project code, i’d be really interested to see how other people are using nbdev - and it’d be good for me to test out the new migrate to magic features as well as running a few coverage reports

#5

I’ve also tried pytest-cov by creating test_decision_tree.py

import nbdev.test
def test_run():
    nbdev.test.test_nb('00_core.ipynb')

then run with

pytest --cov=decision_tree

but that comes back with 0% for decision_tree/core.py

----------- coverage: platform linux, python 3.7.7-final-0 -----------
Name                                        Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------------------------
decision_tree/__init__.py                       1      0      0      0   100%
decision_tree/_nbdev.py                         6      0      2      0   100%
decision_tree/core.py                          34     34      8      0     0%
decision_tree/data.py                          52     52     14      0     0%
decision_tree/exports_to_target_module.py       4      4      0      0     0%
decision_tree/imports.py                        7      0      0      0   100%

if anyone knows how to make this way work, I’d really appreciate the help (o: maybe its not possible due to pytest-cov limitations?

#6

now i feel silly - that comes back with 0% because it’s not testing decision_tree/core.py (o: it runs 00_core.ipynb from top to bottom.

To get good coverage measures, we need to use the modules that nbdev built and run just the non-exported cells as tests.

If we added a little callback to nbdev.test.test_nb we could easily implement callback handlers to run tests in this way: https://github.com/pete88b/decision_tree/blob/master/test_nbs.py - while making it easy for nbdev to keep current behavior.

Now when I run

pytest --cov=decision_tree

I see

collected 1 item                                                                                                                       
test_nbs.py .                                                                                                                    
----------- coverage: platform linux, python 3.7.7-final-0 -----------
Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
decision_tree/__init__.py                       1      0   100%
decision_tree/_nbdev.py                         6      0   100%
decision_tree/core.py                          34      3    91%
decision_tree/data.py                          52     52     0%