Nbdev : A few things/tricks I learned by trial & error

I read all (most) of the documentation a few times, but couldn’t see clearly how all works out or what translates to what.

I have never built a python module before using nbdev, so there is a knowledge gap I need to fill.

So here are my thoughts / experiences / lessons to self.


Running the notebook exports Python modules because of this cell.

Yes and no.

If you run that cell in the notebook, the corresponding module (and all other modules in that project) will be recreated, BUT only if the notebook file has changed. And with changed I mean ‘you change it and save the file’.

  1. Do changes
  2. Save file
  3. run the cell
  4. .py module file is recreated

I guess it applies

if the file hasn't changed from last time:
    do nothing
else:
   recreate modules

If you want to see the notebook and the corresponding module_name.py side by side in Jupyter.

notebooks in:  
project-name/nbs/00_module1.ipynb

module files in:  
project-name/project_name/module1.py

[Jupyter Lab]
An update in module1.py file does not update what you see on the screen (right side) automatically.

You need to use this “Reload python file from disk”


Another option is to “Open in new browser tab” and press F5 to refresh it.


2 Likes

If you make changes in the Notebook that don’t make sense, running this from the Notebook won’t alert you that something is wrong.


Q: Will the module_name.py still be recreated?
A: Yes, even if there is something wrong in it.

Q: How do I know what is wrong?
A: One way is to run, on the console/terminal nbdev_prepare while you are inside project-name directory.

Q: Is there a way to get notified within the Notebook?
A: Yes, run nbdev_prepare from it.
WARNING! Read below why this is a BAD TERRIBLE IDEA.

1 Like

On the image you can see ‘what happens if’ for some cases.


Q: Do I need to use #| export on import statements?
A: Yes


dumb vs no-dumb ( left: notebook, right: module.py )

image


  • Do NOT assume that nbdev will figure out something for you.
  • If you want it IN the module use #|export or #|exporti

Example: import statements


This doesn’t make sense unless you also export / exporti the internal_function.

#| export
var_from_internal_function=internal_function()

More dumb stuff yet to come …

2 Likes


I have been trying to figure out why nbdev would eat all the RAM I can give to it…

It turns out I left that line with
! nbdev_prepare
in the test notebook, which I added just for the photo and question above.

And whenever I run on the Terminal
nbdev_prepare
it just never ends (eating all the RAM until the machine crashes).

Troubleshooting
Running one by one all the pieces:

  • nbdev_export
  • nbdev_test
  • nbdev_clean
  • nbdev_readme

I found out that nbdev_test was the problem.

Then I used
# nbdev_test --do_print --n_workers 1
to find out on which Notebook it was … forever.

Starting /home/jupyter/p/n1.ipynb
- Completed /home/jupyter/p/n1.ipynb
Starting /home/jupyter/p/n2.ipynb
- Completed /home/jupyter/p/n2.ipynb
...

Why?
It runs all the cells in the Notebook, being the last one (here is the dumb idea) a call to itself (i.e. do it again, and again, and again …).

Lesson:
Don’t add the line nbdev_prepare OR nbdev_test in the notebook.

4 Likes

I created this little table to help me separate the concepts of ‘what goes where’. Comments/corrections welcome.


---- DOCUMENTATION Visibility ----	cell/line/code/output/etc. 
#|hide
#|hide_line
#|filter_stream <keyword> ...

#|echo: <true|false>
#|output: <true|false|asis>
#|code-fold: <show|true>

#|exec_doc

---- Python MODULE specification/details ----
#|default_exp <name>
#|export
#|export <some.thing>
#|exporti
#|exports

---- TESTING specification/details ----
#|eval: <true|false>
2 Likes

Don’t put anything after
#| export
as
#| export # Goes to module

or you will get a

TypeError: _export_() takes from 2 to 3 positional arguments but 15 were given

when running
nbdev_export

I guess it because of this other export
#|export <some.thing>
which has a parameter.

1 Like

import statements:

  • When generating Docs: Make the cell that contains it, to Run ALWAYS.
  • When generating Modules: Appear in the module only if you used #| export*

+Info on Mixing of imports and statements ok or not?

I don’t believe that’s correct. You do indeed have to save the notebook for the export to see changes, but there’s no conditional of the type you mentioned.

1 Like

Although not specifically mentioned in the nbdev documentation.

The directives must be at the top of the cell.

This is specified in the Quarto documentation.

So

# Below is my function
#| export

# No, this cell will NOT get exported.

and

#| export
# Below is my function

# YES, this cell will get exported.

If you have ‘issues’ with the use of #| hide in Markdown cells, this may help.
https://forums.fast.ai/t/have-a-complete-toc-when-editing-notebook/

3 Likes

To ensure synchronization between modules and notebooks updates, you could use this in a top cell, e.g. in the 1st one, if possible:

#| hide
%load_ext autoreload
%autoreload 2

To use the @patch

# Import 
from fastcore.foundation import patch  # @patch

In one cell you can now

#| export
@dataclass 
class Number:
    "A number."
    num: int

Display that it works in another cell

#For example, here is the number 5:
Number(5)

> Number(num=5)

And in other cells, continue working on the class.

#| export
@patch
def __add__(self:Number, other):
    "Sum of this and `other`."
    return Number(self.num + other.num)

Example of add method

# Addition:
Number(3) + Number(4)

> Number(num=7)

This is highly beneficial when the class is very large and/or you want to test stuff as you write it.



Here is a demonstration of how you can mix development and explanations using show_doc or @patch

2 Likes