nbdev v2 was launched today. For details, see the launch blog post:
Super exciting stuff! The new tutorial is amazing too. I feel like anyone can write their own libraries effortlessly with nbdev. It’s so magical and makes everyone a magician of making libs.
00:00 - Welcome
01:00 - Getting started and What are those super amazing things that can be done with nbdev?
02:44 - What does Jeremy think of nbdev? What’s Jeremy’s user experience of it?
03:37 - What’s the plan for today’s session?
Turn a github repo into a nbdev repo
04:39 - What’s the first step? Creating a repo
05:34 - Make this repo into a nbdev repo: how to install nbdev, how lightweight is nbdev
06:48 - What are nbdev commands? type nbdev_
How to use them? type nbdev_help
07:25 - How nbdev makes writing docs super easy lazy?
08:17 - How to find out the docs of a nbdev command like nbdev_new
? type nbdev_new -h
08:37 - What does nbdev_new
give us? What is settings.ini
? Why it is so important to help you lazy?
09:56 - What are the stuff generated by nbdev_new
on your cloned repo for you? Which ipynb is your home for docs? (index.ipynb) Which ipynb is your library? (00core.ipynb)
10:54 How should you name ipynbs for your library so that you and people understand how the lib is written? Jeremy showed us how he got started on nbdev with execnb and showed the history of how he explored and invented nbdev
13:36 - Create two ipynbs as two modules card
and deck
14:09 - How to name your module with #|
as in quarto? How and when to hide your cell?
15:32 - What does the heading 1 and heading 2 do? Have a look at execnb as an example
Create a Card module
18:02 - How did Jeremy create a module Card
from scratch in nbdev?
19:01 - Creating suits: How did Jeremy solve the card emoj in python problem live?
21:02 - Creating ranks: based on the codes from the book author
21:23 - Creating a class by starting to create the __init__
function : what is fastai/Jeremy’s coding style?
24:28 - How Jeremy wrote docs of the explorative experiment before starting to build Card class?
26:54 - How to overwrite the __str__
and __repr__
to display the class instance nicely? Also writing docs for the class along the way.
29:03 - How to write docs for input argument (to make docs for the class more clean and short) in the source and how to display docs with show_doc
? Can you even show_doc
other libraries not written in nbdev?
33:13 - Create tests for the class: How to import test
from fastcore
for testing with *
? This way of wild card import require __all__
. nbdev takes care of it and makes it easy.
35:26 - run test_eq?
and test_eq??
or test_eq
or just test_eq( + tab
can give you all the info you need for a class or function
36:26 - How did Jeremy build tests for the Card
class live with different test functions?
37:06 - How to define equality __eq__
for the Class
for the test functions to pass?
38:04 - How to give indication or set the types of the input arguments in the class? def __eq__(self, a: 'Card'): ...
38:32 - How to define a function outside of a class Card
with @patch
from fastcore.utils
in an explorative and experimentative way? Now, you can just def __eq__(self:Card, a:Card): ...
. Also the docs for this function is generated automatically too.
40:08 - Let’s create __lt__
and __gt__
with @patch
and more tests with assert
42:18 - How to export your Card
py file with nbdev_export
and try out your own single module library with pip install -e .
And then import the lib inside index.ipynb
to try out while writing your front page there.
44:37 - to copy the description of the library from setttings.ini
to index.ipynb
and write about the inspiration of this lib.
46:21 - Write about the how to install cell: Note that no code quote is used in the cell. The first middle tricky issue: the lib name can’t have a _
underscore, so it has to be nbdev-Cards
even though the pyfile name and its path name is using nbdev_cards
. Also we need make them clear in settings.ini
file. Jeremy also mentioned how to use the %
in the settings.ini
?
48:33 - How to preview your documentation? nbdev_preview
which also install quarto for you. How Jeremy preview both source file and documentation webpage at the same time to make adjustment and see them in real time.
51:58 - How Jeremy handles functions like __eq__
etc? Do we need to show_doc
them? How to organize these functions?
53:24 - So, you decide which cell in the notebook to show to your readers with #|hide
. For example, you can just show readers one test example and hide the rest without overwhelming them
54:22 - nbdev can also generate advanced webpage like the homepage of nbdev.fast.ai in a single notebook.
55:02 - How to do local test? nbdev_test
to see whether all your tests pass or not. How helpful can nbdev_test
be when a test fails? You can also check out all the functionalities of nbdev_test
by nbdev_test -h
56:52 - How to do debugging in real life in nbdev
with hotkey for reload
and allrun
? in the source notebook, 00
to reload or restart kernel, edit the keyboard shortcut in the notebook for running all cells. Then you can put %debug
in the failed cell to debug what’s wrong.
59:18 - Start writing up the next module to create a deck of cards. Thanks to pip install -e .
your lib will update itself whenever you make changes to it
1:00:51 - Which codes should be exported with #|export
? We can double check the exported pyfile to see what is actually exported with __all__
Creating Deck module
1:02:09 - Write up the Deck class: use ctags -R
and vim
to search and jump through the source codes: /card
to search, ctrl + ]
to jump to the defintion of card
in a different file if needed
1:03:51 - How Jeremy figure out a problem of not exporting a line of import
and test and fix it live?
1:05:24 -
What is the golden rule of making library? Don’t mix import
with other things in a single cell.
1:06:41 - How to overwrite the __len__
and __contains__
and how to use them? len(deck)
and Card(1,1) in deck
1:08:19 - Start creating a pop
method: Who (R) likes and who (quarto) does not like a space between #|
and export
? How to create the function pop
and write docs and default value for arg idx:int=-1): # docs for this arg here
? Then write a test for it right away. And nbdev_export
, nbdev_test
and nbdev_preview
to check the site
1:15:22 - Card
in the markdown cell will be automatically turned into a link to the class Card in the website. This features work for all kinds of libraries.
1:17:00 - Start writing remove
method for the Deck class: copy and write the code, write a test, run tests, check with the preview to see whether exported properly
1:18:48 - Creating a function draw_n
cards: also add another method shuffle
inside the class source and we can show_doc(Deck.shuffle)
at where the shuffle
’s tests are.
1:21:04 - More techniques or features in written tutorials later
Pushing back to github
1:21:50 - Sending to github: first, nbdev_clean
to clean unnecessary metadata; git status
to see what will be going to github; git add -A
to add all the things seen above to be ready to send to github; you can run export, clean and test in one go by nbdev_prepare
; then git commit -am "..."
and git push
to the github.
1:23:45 - What actions does nbdev ask github to do for it? Check them in .github/workflows. Then Jeremy showed how to use gh pages to to display the website and how github running the two actions.
1:25:59 - Now enjoy your library website
1:26:19 - You should be absolutely amazed with what nbdev has done for you.
1:27:21 - run nbdev_docs
to put the homepage inside README.md for you.
Release Your Library
1:28:47 - Use nbdev_pypi
or nbdev_conda
or nbdev_release
to upload. How to deal with the two versions of the pypi library created? use git diff
to see the differences between the two versions; then git commit -am bump
and then git push
1:30:16 - How nbdev makes PR easier for all? and what’s more you can do with nbdev.
Could anyone explain what’s the purpose of bump
in this line of code? Thanks
Try man git-commit
, look for -m
, and tell us what you find!
Thanks Jeremy.
I found out that -m
is an option to allow user to write git commit
message, and there is no option called bump
. (This is where I was confused about)
So bump
is actually an message without quotation marks.
Reading the man git-commit
is very instructive. Thanks!
Thanks Sarada, I finished the longer one and didn’t feel much pause during those 90 minutes. So, I couldn’t believe there are 30 minutes can be cut due to “silence”
I was experimenting fastcore with nbdev. At first, nbdev_preview
works fine. Now I finished some experiments and I can run nbdev_prepare
with success. but when I want to enjoy a nice nbdev_preview
, it throws me the following errors which I have no idea how to fix.
Where should I go to fix it? Thanks
(base) 22:35 ~/Documents/debuggable > nbdev_prepare
Success.
(base) 22:36 ~/Documents/debuggable > nbdev_preview
Preparing to preview
Traceback (most recent call last):
File "/Users/Natsume/mambaforge/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3398, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-1-b58b38ab58c3>", line 2, in <cell line: 2>
NameError: name 'mid' is not defined
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/Natsume/mambaforge/bin/nbdev_filter", line 8, in <module>
sys.exit(nbdev_filter())
File "/Users/Natsume/mambaforge/lib/python3.9/site-packages/fastcore/script.py", line 116, in _f
return tfunc(**merge(args, args_from_prog(func, xtra)))
File "/Users/Natsume/mambaforge/lib/python3.9/site-packages/nbdev/cli.py", line 125, in nbdev_filter
with redirect_stdout(dn): filt.nb_proc(nb).process()
File "/Users/Natsume/mambaforge/lib/python3.9/site-packages/nbdev/process.py", line 168, in process
for cell in self.nb.cells: self._process_cell(cell)
File "/Users/Natsume/mambaforge/lib/python3.9/site-packages/nbdev/process.py", line 149, in _process_cell
cell = opt_set(cell, proc(cell))
File "/Users/Natsume/mambaforge/lib/python3.9/site-packages/nbdev/processors.py", line 145, in __call__
if self.k.exc: raise Exception(f'Error: cell {cell.idx_}:\n{cell.source}') from self.k.exc[1]
Exception: Error: cell 11:
#| echo: false
show_doc(mid)
Is mid
defined in the notebook? Does the cell defining mid
have the correct directive? I think it should be #| export
, but not 100% sure without seeing the notebook.
Details: The NameError
means that nbdev_preview
is trying to access mid
(via show_doc(mid)
), but it isn’t defined in the session. That usually happens when the directives stop the cell that defines mid
from being executed.
Thanks Wasim!
It seems the problem is not with the space between #|
and export
, also Jeremy mentioned that quarto likes no space.
Right now, I get nbdev_preview
to work without error. The way I did is to add #|export
to every function the error reported as “not defined”. I have no idea why I should export those functions even though I don’t intend them to be inside my library. I probably need to read nbdev docs to learn more of how and when to use #|export
.
Another troublesome issue is that when I ctrl + z
the preview and start preview again. I have the following error message in the image below and the only way to solve it for me is to quit and restart browser and terminal. Is there a better solution? Thanks.
So far, what I figured out the use of #|export
:
- export the functions you write specifically for your library purpose.
- export the func so that
show_doc(func)
can run. (this is the place I fixed the error fromnbdev_preview
You shouldn’t be running show_doc on any function you don’t export - since otherwise your users will see documentation for a function that you didn’t actually export for them to use!
ctrl+z suspends a job but doesn’t cancel it - it’s still there in the background (type jobs
to see it). You should use ctrl-c to cancel it.
Thank you Jeremy! This is helpful. Please let me summary what I learnt by experimenting nbdev and from this forum:
On the use of show_doc
in nbdev:
- Only use
show_doc
on functions you export for your library, becauseshow_doc
is intended for users to check the docs of your library functions and classes. - Also, if you didn’t export e.g., func1 and run
show_doc(func1
), nbdev will throw an error. - If for special cases you really want to display
show_doc(func1)
on your preview whenfunc1
is not to be exported to your library, then you should runshow_doc(func1)
, and restart the notebook and comment outshow_doc
. Since you still have things like<nbdev.showdoc.BasicMarkdownRenderer at 0x10fbaea60>
, you can render theshow_doc
effect in nbdev_preview to display.
One more thing to add - you generally don’t need to manually add show_doc
, since it’s automatically called for you for any function or class you #|export
. Have a look at some of the nbdev or fastai source notebooks for examples.
Thanks Jeremy, you are absolutely right, it works the same on my nbdev project too.
I have another quick question about describing the following code on @delegates
in English: Should I say:
func
mid
delegateslow
to decide which args to be included in**kwargs
def low(a, b): pass
@delegates(low)
def mid(c, d, **kwargs): pass
I don’t actually understand what that sentence is trying to say - sorry!
IIUC, correct English is “mid
delegates to low
”.
But you wouldn’t need to say that, because the point of delegates
is that the decorated function signature has kwargs
replaced with to
’s, so the user will already see the effect of that in show_doc
Hi @seem
It seems nbdev only allows headings to extend from level 1 to level 3. Can I customize it to extend to level 4 or more? Where should I look? Thanks