Nbdev : A few things/tricks I learned by trial & error

rgarcia · November 23, 2022, 7:24pm

I read all (most) of the documentation a few times, but couldn’t see clearly how all works out or what translates to what.

I have never built a python module before using nbdev, so there is a knowledge gap I need to fill.

So here are my thoughts / experiences / lessons to self.

Running the notebook exports Python modules because of this cell.

Yes and no.

If you run that cell in the notebook, the corresponding module (and all other modules in that project) will be recreated, BUT only if the notebook file has changed. And with changed I mean ‘you change it and save the file’.

Do changes
Save file
run the cell
.py module file is recreated

I guess it applies

if the file hasn't changed from last time:
    do nothing
else:
   recreate modules

If you want to see the notebook and the corresponding module_name.py side by side in Jupyter.

notebooks in:  
project-name/nbs/00_module1.ipynb

module files in:  
project-name/project_name/module1.py

[Jupyter Lab]
An update in module1.py file does not update what you see on the screen (right side) automatically.

You need to use this “Reload python file from disk”

Another option is to “Open in new browser tab” and press F5 to refresh it.

rgarcia · November 23, 2022, 7:45pm

If you make changes in the Notebook that don’t make sense, running this from the Notebook won’t alert you that something is wrong.

Q: Will the module_name.py still be recreated?
A: Yes, even if there is something wrong in it.

Q: How do I know what is wrong?
A: One way is to run, on the console/terminal nbdev_prepare while you are inside project-name directory.

Q: Is there a way to get notified within the Notebook?
A: Yes, run nbdev_prepare from it.
WARNING! Read below why this is a ~~BAD~~ TERRIBLE IDEA.

rgarcia · November 23, 2022, 8:08pm

On the image you can see ‘what happens if’ for some cases.

Q: Do I need to use #| export on import statements?
A: Yes

dumb vs no-dumb ( left: notebook, right: module.py )

Do NOT assume that nbdev will figure out something for you.
If you want it IN the module use #|export or #|exporti

Example: import statements

This doesn’t make sense unless you also export / exporti the internal_function.

#| export
var_from_internal_function=internal_function()

More dumb stuff yet to come …

rgarcia · November 24, 2022, 1:54am

I have been trying to figure out why nbdev would eat all the RAM I can give to it…

It turns out I left that line with
! nbdev_prepare
in the test notebook, which I added just for the photo and question above.

And whenever I run on the Terminal
nbdev_prepare
it just never ends (eating all the RAM until the machine crashes).

Troubleshooting
Running one by one all the pieces:

nbdev_export
nbdev_test
nbdev_clean
nbdev_readme

I found out that nbdev_test was the problem.

Then I used
# nbdev_test --do_print --n_workers 1
to find out on which Notebook it was … forever.

Starting /home/jupyter/p/n1.ipynb
- Completed /home/jupyter/p/n1.ipynb
Starting /home/jupyter/p/n2.ipynb
- Completed /home/jupyter/p/n2.ipynb
...

Why?
It runs all the cells in the Notebook, being the last one (here is the dumb idea) a call to itself (i.e. do it again, and again, and again …).

Lesson:
Don’t add the line nbdev_prepare OR nbdev_test in the notebook.

rgarcia · November 25, 2022, 11:00am

I created this little table to help me separate the concepts of ‘what goes where’. Comments/corrections welcome.


---- DOCUMENTATION Visibility ----	cell/line/code/output/etc. 
#|hide
#|hide_line
#|filter_stream <keyword> ...

#|echo: <true|false>
#|output: <true|false|asis>
#|code-fold: <show|true>

#|exec_doc

---- Python MODULE specification/details ----
#|default_exp <name>
#|export
#|export <some.thing>
#|exporti
#|exports

---- TESTING specification/details ----
#|eval: <true|false>

rgarcia · November 27, 2022, 4:37pm

Don’t put anything after
#| export
as
#| export # Goes to module

or you will get a

TypeError: _export_() takes from 2 to 3 positional arguments but 15 were given

when running
nbdev_export

I guess it because of this other export
#|export <some.thing>
which has a parameter.

rgarcia · November 28, 2022, 5:32pm

import statements:

When generating Docs: Make the cell that contains it, to Run ALWAYS.
When generating Modules: Appear in the module only if you used #| export*

+Info on Mixing of imports and statements ok or not?

jeremy · December 2, 2022, 3:03am

I don’t believe that’s correct. You do indeed have to save the notebook for the export to see changes, but there’s no conditional of the type you mentioned.

rgarcia · December 17, 2022, 12:10am

Although not specifically mentioned in the nbdev documentation.

The directives must be at the top of the cell.

This is specified in the Quarto documentation.

So

# Below is my function
#| export

# No, this cell will NOT get exported.

and

#| export
# Below is my function

# YES, this cell will get exported.

If you have ‘issues’ with the use of #| hide in Markdown cells, this may help.
https://forums.fast.ai/t/have-a-complete-toc-when-editing-notebook/

Xylem · January 27, 2023, 5:19pm

To ensure synchronization between modules and notebooks updates, you could use this in a top cell, e.g. in the 1st one, if possible:

#| hide
%load_ext autoreload
%autoreload 2