HowTo: add formating and linting to nbdev (black, isort, mypy, flake8, pylint)

nbdev
1

Introduction

nbdev is a great tool.

However some of you may not be comfortable with the default fastai codestyle, or maybe your team is already using a different one.

This post will show you how to integrate your nbdev workflows with industry standard tools like:

  • black , The uncompromising code formatter
  • isort , A Python utility / library to sort imports.
  • mypy , Optional static typing for Python
  • pre-commit , A framework for managing and maintaining multi-language pre-commit hooks.

An astute reader may notice that one can
enable black_formatting = True in settings.ini

However,

  • this will only format the code in the module, not the code in the notebook.
  • this does not allow you to use other tools like isort or mypy

nbqa

use nbqa to format and verify code in your notebooks

nbqa is a tool that allows you to run any tool (formatter/linter) on your notebooks, without having to worry about the details.

black

Black is a python code formatter that is becoming the de-facto standard

pip install nbqa

nbqa black nbs/

isort and mypy

isort will sort your imports, and mypy will check your type annotations

pip install isort mypy

nbqa isort nbs/
nbqa mypy nbs/

Automating your flows with Makefile

But you are already using nbdev command-line arguments like nbdev_export and nbdev_test, who wants to remember more commands?

Make is a great tool for automating your workflows, usually comes pre-installed on most unix systems.

Create a Makefile in the root of your project and add the following lines:

# Makefile
format:
    nbqa black nbs/
    nbqa isort nbs/
    nbqa mypy nbs/

Now you can run make format to format your notebooks.

Let’s add more commands to our Makefile:

format:
    nbqa black nbs/
    nbqa isort nbs/
    nbqa mypy nbs/

export:
	nbdev_export

mypy:
	nbqa mypy nbs/ --ignore-missing-imports --check-untyped-defs

test:
	nbdev_test --n_workers 4

prepare:
	# Export, test, and clean notebooks, and render README if needed
	nbdev_prepare

docs:
	nbdev_docs

And now the cherry on top: Makefile supports dependecies, so we can chain commands together.

add this to your Makefile:

all: format prepare mypy

Running make all will format your notebooks, export them, run mypy, run tests and update README.md

pre-commit

Want to automate your formatting even more?

Force your team to format their notebooks before committing?

Use pre-commit

install and configure pre-commit

pip install pre-commit # install 

pre-commit install # enable for this repo

create .pre-commit-config.yaml

repos:
  - repo: https://github.com/fastai/nbdev
    rev: 2.2.10
    hooks:
      - id: nbdev_clean
      - id: nbdev_export

Add nbqa hooks

  - repo: https://github.com/nbQA-dev/nbQA
    rev: 1.5.3
    hooks:
      - id: nbqa-mypy
        args: ["--ignore-missing-imports", "--check-untyped-defs"]
        additional_dependencies: ["mypy", "types-toml", "types-requests"]
      - id: nbqa-black
      - id: nbqa-isort

control your hooks behaviour

  - repo: https://github.com/nbQA-dev/nbQA
    rev: 1.5.3
    hooks:
      - id: nbqa-mypy
        args: ["--ignore-missing-imports", "--check-untyped-defs"] # pass args to mypy
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting
        additional_dependencies: ["mypy", "types-toml", "types-requests"] # install additional dependencies
      - id: nbqa-black
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting
      - id: nbqa-isort
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting

reproducible environments with Pipenv/Poetry/virtualenv

Sometimes you may run into reproducibility issues, where your notebook works on your machine, but not on your team’s.

The best way to avoid it is to use a dependency manager, such as:

But how do you integrate it with nbdev, since it uses settings.ini to manage dependencies?

In this post I show an example of how to use Pipenv with nbdev.

Concusion

I hope you find this post useful.

These changes may make nbdev more friendly to a team environment.

Please don’t hesitate to post any questions or comments below.

7 Likes
2

Love this! Great work!!!

1 Like