Nbdev code coverage / n-Tests

muellerzr · July 23, 2021, 11:06pm

It doesn’t extract into new files… It overrides nbdev_test_nbs

pete88b/decision_tree/blob/master/80_test_coverage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating code coverage reports for an nbdev project\n",
    "\n",
    "We can run tests in parallel and get coverage with [pytest-cov](https://github.com/pytest-dev/pytest-cov).\n",
    "\n",
    "If you'd like to try this:\n",
    "- you might need to use an [editable install of nbdev](https://github.com/fastai/nbdev/#installing)\n",
    "- install pytest-cov and its dependencies\n",
    "- copy [test_nbs.py](https://github.com/pete88b/decision_tree/blob/master/test_nbs.py) to your nbdev project\n",
    "- then run `pytest --cov=[your lib name]`\n",
    "\n",
    "Feel free to join [the discussion](https://forums.fast.ai/t/nbdev-code-coverage-n-tests/73993/6) (o:"
   ]
  },
  {

This file has been truncated. show original

Well, overrides/introduces new behavior.

michaelaye · July 23, 2021, 11:27pm

Doesn’t run for me (reported), and I don’t know how it can run for others as Config().nbs_path is indeed a string not a Path object so it doesn’t have a glob attribute?

github.com

pete88b/decision_tree/blob/master/test_nbs.py#L109

    
      
          """
          fname:Param("A notebook name or glob to convert", str)=None,
          flags:Param("Space separated list of flags", str)=None,
          n_workers:Param("Number of workers to use", int)=None,
          verbose:Param("Print errors along the way", bool)=True,
          timing:Param("Timing each notebook to see the ones are slow", bool)=False
          """
          "Test in parallel the notebooks matching `fname`, passing along `flags`"
          if flags is not None: flags = flags.split(' ')
          if fname is None: 
              files = [f for f in Config().nbs_path.glob('*.ipynb') if not f.name.startswith('_')]
          else: files = glob.glob(fname)
          files = [Path(f).absolute() for f in sorted(files)]
          if len(files)==1 and n_workers is None: n_workers=0
          # make sure we are inside the notebook folder of the project
          os.chdir(Config().nbs_path)
          results = parallel(_test_one, files, flags=flags, verbose=verbose, n_workers=n_workers)
          passed,times = [r[0] for r in results],[r[1] for r in results]
          if all(passed): print("All tests are passing!")
          else:
              msg = "The following notebooks failed:\n"

josemz · July 23, 2021, 11:43pm

Pete’s version is more than 1 year old, however just wrapping that in Path solves the issue. I’ve used it successfully in a project for some time now, here’s the file if you’re interested. It works very well, I was able to even include codecov in that project using Pete’s version, big shoutout to him.

yegeniy · July 27, 2021, 9:21pm

Has anyone done mocking (unittest.mock) in Jupyter notebooks? I’m running into issues porting some code into nbdev.

There are some tests in the original codebase that leverage the @mock.patch and unittest.main(). They are simple enough to re-write without mocking. That would probably be a better approach anyways, but it got me wondering.

What is everyone’s experience so far with the use mocking in nbdev / Jupyter notebook? Haven’t seen much online.

P.S. let me know if it’d be better etiquette to start a separate post on this.

doyu · February 1, 2023, 8:07pm

@josemz Your link seems to be broken. Do you still have your fixed file?

Alternatively how nowadays does people deal with unit test coverage under nbdev?

josemz · February 3, 2023, 2:36am

Hi. I don’t know of a way to do it with nbdev2, but @muellerzr did something that could be very useful here. Exporting the tests to a file or directory would allow you to use the regular pytest commands to compute the coverage.

doyu · February 6, 2023, 1:07pm

I tried to port @muellerzr 's 01_export.ipynb to the latest nbdev and it seems that, the latest NBProcessor() doesn’t take preprocs= anymore as its param with the following @jeremy 's commit:

How should I fix without preprocs with the latest nbdev? Any hint? The error I got is:

jeremy · February 6, 2023, 8:54pm

You can use class-based preprocessors instead. Have a look at api/10_processors.ipynb and search for def begin to see examples.