Using pytest with nbdev, nbprocess to the rescue!

nbdev
1

Hi everyone! If you’re not aware, Jeremy and Hamel are heavily working on version 2 of nbdev, nbprocess. This framework looked exceptionally interesting to me because its simple, and most of all modular.

What do I mean? I took it out for a test drive this afternoon, and in no more than 5 hours I was able to have nbdev automatically export your tests as pytest modules!.
image

This is a screenshot from my nbprocess sandbox where I’ll be toying around with a few ideas as we go.

So, how does this work then Zach?

Right now it’s very limited, but you’ll get the idea.

  1. We only have one unittest.TestCase per notebook. (It’s a POC, cut me at least a little slack please :wink: )
  2. We assume that all tests related to a particular functionality will reside in the same cell. Or at least one test we want as a pytest.
  3. All we need to do is declare what is a test!

For example, here are two cells (separated out by ---)

#|export
def addition(a,b): return a+b
-------------------------------------
#|test addition
test_eq(addition(1,2), 3)
test_eq(addition(-1,0), -1)

That first #|export is how we export things in nbprocess now. It’s a bit of a nifty superpower that lets us pull off our next magic trick.
That second one, the #|test portion tells nbprocess we want to mark this as a “test”. The second is what I have deemed to be the name of said test, “addition”.

From here, you just perform the following (if you pip install from my repo):

from testing_grounds.export import create_test_modules
create_test_modules(some_notebook_name, "tests")

This will then create that beautiful pytest tests/test_core.py we saw earlier. All at the touch of your fingertips!

Currently it just keeps the fastcore testing logic and relies on those asserts, but it’s not hard to imagine being able to use pytest’s self.assertXYZ along the way.

Hope you enjoyed this brief insight to what’s coming, and that you’re just as excited as I am :slight_smile:

8 Likes
2

Looks great! Just one question. Do you need to inherit from the TestCase if you use the pytest runner? Of course, it doesn’t hurt, but could also be a plain list of test functions, right?

Also, self.assert* could be plain assert in this case. Does it make sense to support various testing systems? Like #|pytest vs #|unittest.

3

This would be the unittest runner. It’s simple enough to make this extension that you could easily just have a pytest runner that does the funcs instead in a seperate function call.

My next step is changing how the unittest class setup works to make it as customizable as you can (since we can’t assume you’ll always inherit TestCase, you could have your own TestCase superclass you implemented with custom setup/teardown, etc)

And a heavy refactor thanks to some of Jeremy’s feedback

Yes, it’s much more work getting self.assert* and self.fail* working, so for now I’d recommend just using either raw assert or using the test_* funcs. Which those will work OOTB since we just take the code under the test flag and shove it into the func

1 Like
4

Yes, absolutely, especially taking into account pytest’s special interpretation of assert statements.

Ok, got it! Agree, it should be possible to easily enable pytest runner as well, like generate a bit different test suit from the same notebook. I believe that pytest is much more “pythonic” than the standard library.

1 Like
5

I’ve now simplified it a ton, you can see the latest version here: nbprocess-sandbox/01_export.ipynb at main · muellerzr/nbprocess-sandbox · GitHub

This relies on a refactor getting merged into nbprocess so for now if you want to play with it do pip install git+https://github.com/muellerzr/nbprocess

Note: to get pytest working it will assume you import via from fastcore.test import * should you use fastcore

2 Likes
6

Great, thank you! I am considering to start looking into Jupyter development again; did that few years ago but then moved to “standard” approach. So great to see further development of notebook-enabling tools.

7

Update on this, based on how fastcore.test does things, we’ll never be able to take advantage of this functionality. And rewriting each specific fastcore.test on the fly is way too niche specific. So in that case I’d recommend using the raw assert statements etc rather than fastcore’s test_* suite :slight_smile:

1 Like