Nbdev : A few things/tricks I learned by trial & error

rgarcia · November 23, 2022, 7:24pm

I read all (most) of the documentation a few times, but couldn’t see clearly how all works out or what translates to what.

I have never built a python module before using nbdev, so there is a knowledge gap I need to fill.

So here are my thoughts / experiences / lessons to self.

Running the notebook exports Python modules because of this cell.

Yes and no.

If you run that cell in the notebook, the corresponding module (and all other modules in that project) will be recreated, BUT only if the notebook file has changed. And with changed I mean ‘you change it and save the file’.

Do changes
Save file
run the cell
.py module file is recreated

I guess it applies

if the file hasn't changed from last time:
    do nothing
else:
   recreate modules

If you want to see the notebook and the corresponding module_name.py side by side in Jupyter.

notebooks in:  
project-name/nbs/00_module1.ipynb

module files in:  
project-name/project_name/module1.py

[Jupyter Lab]
An update in module1.py file does not update what you see on the screen (right side) automatically.

You need to use this “Reload python file from disk”

Another option is to “Open in new browser tab” and press F5 to refresh it.

rgarcia · November 23, 2022, 7:45pm

If you make changes in the Notebook that don’t make sense, running this from the Notebook won’t alert you that something is wrong.

Q: Will the module_name.py still be recreated?
A: Yes, even if there is something wrong in it.

Q: How do I know what is wrong?
A: One way is to run, on the console/terminal nbdev_prepare while you are inside project-name directory.

Q: Is there a way to get notified within the Notebook?
A: Yes, run nbdev_prepare from it.
WARNING! Read below why this is a ~~BAD~~ TERRIBLE IDEA.

rgarcia · November 23, 2022, 8:08pm

On the image you can see ‘what happens if’ for some cases.

Q: Do I need to use #| export on import statements?
A: Yes

dumb vs no-dumb ( left: notebook, right: module.py )

Do NOT assume that nbdev will figure out something for you.
If you want it IN the module use #|export or #|exporti

Example: import statements

This doesn’t make sense unless you also export / exporti the internal_function.

#| export
var_from_internal_function=internal_function()

More dumb stuff yet to come …

rgarcia · November 24, 2022, 1:54am

I have been trying to figure out why nbdev would eat all the RAM I can give to it…

It turns out I left that line with
! nbdev_prepare
in the test notebook, which I added just for the photo and question above.

And whenever I run on the Terminal
nbdev_prepare
it just never ends (eating all the RAM until the machine crashes).

Troubleshooting
Running one by one all the pieces:

nbdev_export
nbdev_test
nbdev_clean
nbdev_readme

I found out that nbdev_test was the problem.

Then I used
# nbdev_test --do_print --n_workers 1
to find out on which Notebook it was … forever.

Starting /home/jupyter/p/n1.ipynb
- Completed /home/jupyter/p/n1.ipynb
Starting /home/jupyter/p/n2.ipynb
- Completed /home/jupyter/p/n2.ipynb
...

Why?
It runs all the cells in the Notebook, being the last one (here is the dumb idea) a call to itself (i.e. do it again, and again, and again …).

Lesson:
Don’t add the line nbdev_prepare OR nbdev_test in the notebook.

rgarcia · November 25, 2022, 11:00am

I created this little table to help me separate the concepts of ‘what goes where’. Comments/corrections welcome.


---- DOCUMENTATION Visibility ----	cell/line/code/output/etc. 
#|hide
#|hide_line
#|filter_stream <keyword> ...

#|echo: <true|false>
#|output: <true|false|asis>
#|code-fold: <show|true>

#|exec_doc

---- Python MODULE specification/details ----
#|default_exp <name>
#|export
#|export <some.thing>
#|exporti
#|exports

---- TESTING specification/details ----
#|eval: <true|false>

rgarcia · November 27, 2022, 4:37pm

Don’t put anything after
#| export
as
#| export # Goes to module

or you will get a

TypeError: _export_() takes from 2 to 3 positional arguments but 15 were given

when running
nbdev_export

I guess it because of this other export
#|export <some.thing>
which has a parameter.

rgarcia · November 28, 2022, 5:32pm

import statements:

When generating Docs: Make the cell that contains it, to Run ALWAYS.
When generating Modules: Appear in the module only if you used #| export*

+Info on Mixing of imports and statements ok or not?

jeremy · December 2, 2022, 3:03am

I don’t believe that’s correct. You do indeed have to save the notebook for the export to see changes, but there’s no conditional of the type you mentioned.

rgarcia · December 17, 2022, 12:10am

Although not specifically mentioned in the nbdev documentation.

The directives must be at the top of the cell.

This is specified in the Quarto documentation.

So

# Below is my function
#| export

# No, this cell will NOT get exported.

and

#| export
# Below is my function

# YES, this cell will get exported.

If you have ‘issues’ with the use of #| hide in Markdown cells, this may help.
https://forums.fast.ai/t/have-a-complete-toc-when-editing-notebook/

Xylem · January 27, 2023, 5:19pm

To ensure synchronization between modules and notebooks updates, you could use this in a top cell, e.g. in the 1st one, if possible:

#| hide
%load_ext autoreload
%autoreload 2

rgarcia · February 20, 2024, 8:45pm

To use the @patch

# Import 
from fastcore.foundation import patch  # @patch

In one cell you can now

#| export
@dataclass 
class Number:
    "A number."
    num: int

Display that it works in another cell

#For example, here is the number 5:
Number(5)

> Number(num=5)

And in other cells, continue working on the class.

#| export
@patch
def __add__(self:Number, other):
    "Sum of this and `other`."
    return Number(self.num + other.num)

Example of add method

# Addition:
Number(3) + Number(4)

> Number(num=7)

This is highly beneficial when the class is very large and/or you want to test stuff as you write it.

Here is a demonstration of how you can mix development and explanations using show_doc or @patch

rgarcia · June 6, 2024, 11:33am

Today GPT tought me something

The issue arises because the @patch decorator expects to work with a callable object (like a function or method), and combining it directly with @property results in a property object, which causes the TypeError.

To work around this, you can use the patch_property from the fastcore library instead. This is designed to handle properties specifically.

As of 2024-06-06

UserWarning: patch_property is deprecated and will be removed;
use patch(as_prop=True) instead

but GPT put me on the right track.

The @patch is really handy when you want to add some functionality to a class, but you leverage partially in the class to create it.

So you can

Create class_A (missing new_func)
Create new_func ( leveraging on existing class_A )
@patch class_A which now will use new_func somehow

Very clean, without using mock implementations. Simply beautiful.

rgarcia · June 9, 2024, 8:06am

This is

@patch(as_prop=True)
def funct_to_add_to_class(self:Language):
     return something(self.language_name)

equivalent to this one

@patch_to(Language, as_prop=True)
def funct_to_add_to_class(self):
     return something(self.language_name)

Here is a nice notebook with the code

patch_to(cls, as_prop=False, cls_method=False):
"Decorator: add `f` to `cls`"

patch(f=None, *, as_prop=False, cls_method=False):
"Decorator: add `f` to the first parameter's class (based on f's type annotations)"

In other words, @patch or patch gets the class name from the type annotation and then calls @patch_to | patch_to with it. They do the same.

rgarcia · October 25, 2024, 10:07am

Q:

Using fastai nbdev you can export cells with

#|export

import logging
import subprocess

Should the cells that only do imports, be exported with #|export or #|exporti and why?

A: Perplexity has the answer below, but I think it doesn’t matter as looking at the module code, I don’t see any library exported when using #|export

In any case, I leave this here so someone else can give some input.

A: (from https://www.perplexity.ai)

When using fastai’s nbdev, the choice between #|export and #|exporti for cells containing only imports depends on how you want those imports to be handled in the generated module. Here’s the explanation:

Export Options

#|export

Using #|export for import cells will include those imports in the generated Python module. This means that when someone imports your module, they will also implicitly import the libraries you’ve specified.

#|exporti

The #|exporti directive, where ‘i’ stands for ‘import’, is specifically designed for handling import statements. When you use #|exporti, the imports are included in the generated module but are not re-exported.

Recommendation

For cells that only contain import statements, it’s generally recommended to use #|exporti rather than #|export. Here’s why:

Cleaner API: Using #|exporti keeps your module’s public API cleaner by not re-exporting third-party libraries.
Explicit imports: It encourages users of your module to explicitly import the libraries they need, which can lead to more maintainable and understandable code.
Avoiding conflicts: It prevents potential naming conflicts that could arise from re-exporting common library names.
Performance: In some cases, it can lead to slightly faster import times for your module, as it doesn’t need to import and re-export unnecessary libraries.

By using #|exporti for your import cells, you ensure that the necessary imports are available for your module’s internal use without affecting the module’s public interface.

[1] nbdev - fast.ai Course Forums