NBDev Testing workflow: Inline vs external tests?

I’m pretty new to all of this (Python, Notebooks, data science, etc), but not new to programming. I am excited about nbdev, and have a question about the recommended testing workflow.

In Jeremy’s YouTube video, “I Like Notebooks”, he recommends writing notebooks in sucjh a way that you can frequently restart the kernel and hit “Run All”.

The NBDev documentation makes the point that writing tests in the same notebook as the code is valuable. Generally I agree. However, I frequently find myself writing some pretty heavy tests that rely upon external services. So if I were to hit “Run All” in that notebook, those long-running cells would also run, but that’s not what I want. I could tag these cells as “#slow” and that would affect whether nbtest runs them, and that’s fine. But that wouldn’t help me prevent them from running when I click run all at the top of the notebook.

So these two guidelines make me think that what I really want is the ability to skip certain cells by default when hitting run all in Jupyter, and tag them such that they only run at nbdev_test time.

But in searching for guidance about how to exclude notebook cells from running by default, many folks advise against this for various reasons, so I think I’m swimming upstream with that idea.

This all leads me to think that I should be moving these longer-running tests into a notebook outside of the notebook where the code is, and then I start to think I’m losing one of the core advantages of nbdev.

I think I’m overcomplicating this. What are you clever folks doing?

I don’t think you lose much when you put heavy tests into another notebook. Not only is it automatically recognized by the nbdev_xxx commands, you can also mark a whole notebook as slow and therefore easily exclude it from nbdev_test. So, that’s what I would do. But I’m only semi-experienced with nbdev. :slight_smile:

1 Like

Oh cool - that’s really helpful. I appreciate the quick response!

+1, this is how I would do it too. I personally have a very strong preference to keep notebooks with the main logic very fast. Also, some of the more integration/end-to-end tests can be quite long which tends to disrupt the narrative in the main notebooks

1 Like

Thank you seem! I appreciate the validation.