Nbdev discussion

Thank you once again for this wonderful tool!

One question… How can I pass the dependencies of my library to the setup.py file, without editing it directly? Should I use the requirements field in the settings.ini file as a comma separated list? For example, how should I pass torch and torchvision as a requirement?

1 Like

When I’m building docs and use github pages to host the generated .html files I get the following:

The index page is of the format:
https://github_username.github.io/lib_name

Then, to navigate to the test module page for example from the sidebar the link is of the format:
https://github_username.github.io/lib_name/test

Thus, when I follow the table of contents links for a function named test_equal for example, I’m sent to:
https://github_username.github.io/lib_name/test#test_equal

But when I press the hypelink inside the documentation for this function I get redirected to:
https://github_username.github.io/test.html#test_equal

which causes a 404 error. Is it a bug or am I missing something?

I’m trying to get nbdev to ignore some cells during testing.

Here is what the docs say, but I am unclear as to what specifically should I change in my settings.ini:

Everything that is not an exported cell is considered a test, so you should make sure your notebooks can all run smoothly (and fast) if you want to use this functionality as the CLI. You can mark some cells with special flags (like slow) to make sure they are only executed when you authorize it. Those flags should be configured in your settings.ini (separated by a | if you have several of them). You can also apply a flag to one entire notebook by putting # all_flag in one of its cells.

1 Like

Thanks for making this amazing tool!

If anyone has any experience with making a Conda package from this, I’d be really happy to discuss it (and hopefully publish a tutorial for others). pip isn’t getting my dependencies right and with Conda it works much better. However, I’m not sure how to create a Conda package from the output of nbdev; conda skeleton pypi my_pkg fails with

Traceback (most recent call last):
  File "setup.py", line 18, in <module>
    assert o in cfg, "missing expected setting: {}".format(o)
AssertionError: missing expected setting: version
1 Like

Hi guys, thanks for making this amazing tool.

I have installed it, but getting this error when running nbdev_build_docs:

converting: /home/marco/mlserver2/00_core.ipynb
An error occurred while executing the following cell:
------------------
#export
from nbdev.showdoc import show_doc
from mlserver2.core import *
------------------

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-9d176fd58c56> in <module>
      1 #export
----> 2 from nbdev.showdoc import show_doc
      3 from mlserver2.core import *

ModuleNotFoundError: No module named 'nbdev'
ModuleNotFoundError: No module named 'nbdev'

However, I can import nbdev, and nbdev_build_libs works correctly. I can import that code from a python CLI.

I cloned the fastai2 repo and get the same error trying to build the docs there

2 Likes

@dzlob You need to put the flags you want to use separated by | in the variable tst_flags. For instance, fastai2 has the line:

tst_flags = slow|cuda|cpp

(or it will very soon since I realize it’s missing!)

@adamh We will add conda support (and make it as easy as release on pypi for the packages you build with nbdev) very soon

@nextM Do you ahve this problem after doing a dev install or a pip install?

It should be a whitespace separated list, since it’s passed to str.split and then directly to setuptools.setup.

BTW you may find the fastai2 settings.ini useful to see how things work:

1 Like

It’s a bug. We’re looking into it.

Thanks! That’d be a game changer.

Fixed now (with an editable install of nbdev after pulling master).

still working on this error. can anyone decipher this

 ERROR: Command errored out with exit status 1:
     command: /opt/hostedtoolcache/Python/3.6.9/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/runner/work/art-blender/art-blender/setup.py'"'"'; __file__='"'"'/home/runner/work/art-blender/art-blender/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /home/runner/work/art-blender/art-blender/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/runner/work/art-blender/art-blender/setup.py", line 13, in <module>
        for o in expected: assert o in cfg, "missing expected setting: {}".format(o)
    AssertionError: missing expected setting: description
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

You’re missing the description setting in your setting.py.

1 Like

@sgugger From either dev install or pip install, same issue.

I also tried with an earlier version of nbdev, same issue.

I tried installing pip nbdev in a different anaconda environment, and still saw the same issue.

Is it possible when it runs it is using the python environment of the machine?

I think the problem is coming from the #default_export tag because my 00_core.ipynb notebook is empty except for one dummy function tagged with #export (which I can see in the generated code file)

00_core.ipynb looks like:

# default_exp core

#export
def do_twice(func):
    def wrapper_do_twice(*args, **kwargs):
        func(*args, **kwargs)
        # make sure the wrapper function returns the return value of the decorated function
        return func(*args, **kwargs)
    return wrapper_do_twice

These lines are not in my notebook at all, are they added by the #default_export core command?:

from nbdev.showdoc import show_doc
from mlserver2.core import *

I’ve been a fan of literate programming since my first code with tangle and weave. I was always disappointed that it never gained traction. I hope nbdev gets closer to delivering on the promise of literate programming.

To that end, any pointers on how to import existing code and applications into nbdev and re-factoring it? Also any pointers on how to think about literate programming in the nbdev context. For example, when I was using cweb, I wrote the code is if I was writing a book. I started out with the beginning outline and the code segment referred to other fragments later in the narrative. . The beginning section will look something like

=====

This little app takes a set of templates, one to a calendar event and data for each calendar event then expands them using recursive template expansion.

Make ENET monthly meeting blast

<Set up event iteration>
<for each event>
  <create template instance>
  <expand template>
  <generate time key>
  <sort header by skip>

The phrase between the <> characters refers to the section containing the detail and narrative. Every section could have a mixture of code and section references. Using my code base is an example, the first segment would be something like"

set up event iteration:
in order to iterate over the calendar events, gather up a list of event files, baseline template, and create dictionary with common data elements used in template expansion.
— Code section—
events=glob.glob("./*.event")

with open("event_template.rtmpl") as f:
    event_template=f.read()

common_dict = rTemplate.colon_parser("event_common", EOL=False)

event_dict = {}
nonevent_dict = {}

subsequent sections would fill in the detail outlined in the beginning and the tangle command would organize it in a compiler friendly order.

How do I reframe my literate programming experience to fit the nbdev model?

I think this is a thing with my anaconda maybe? I get similar errors when running the notebook 03_export2html from the nbdev repo. It says nbformat not installed, but it definitely is.

It’s likely, yes. Note that the import statements are added to execute all the show_doc cells in the notebook (it is then removed from the html). So it’s logical you have them during the doc building even if they were not in your original notebook.

I’ve only done that on a couple of small projects, by copying and pasting the docs into a notebook, and the modules, and then using “ctrl-shift-hyphen” to split into separate cells as appropriate. Then I move things around so that it makes sense.

It’s actually a very different mindset to cweb. cweb assumes you pretty much know what you want, and you can write it all out in a hierarchical top-down way. nbdev assumes you don’t really know what you’re doing, and you build it up from little pieces step by step.

The best way to understand this mindset is to look at the nbdev and fastai2 source nbs.

I’ll be interested to hear how it works out for you!

First bit of trouble and having is that exploring nbdev is seriously cutting into my sleep. I was awake till 4 AM yesterday playing with it. Damn you, damn you all to hell… :rofl: I apologize for the chaotic nature of these questions. I am asking questions stream of consciousness and I’m clearly missing some very basic knowledge about Jupyter notebooks.

I appreciate your patience and responses.

One thing that tangle/weave did was it allowed you to order the narrative based on its needs, not what the compiler requires. That is really important for writing literate code. However, deviating significantly from the order the compiler needs reduces ability to debug. The debug issue would not exist if one could debug code using the literate form and not the tangled form.

Moving on to specific examples. 00 export creates lots of questions.

Why you have a number prefix on all of the sections (00_export, 00_sync etc.)? I am guessing that ordering is important and the easiest/fastest way of doing this is to use lexical sorting of filenames. The number prefix gives you easy and explicit control independent of a configuration file.

The second and third blocks have some sort of control information and a hidden import. Elaborate please?

Section export to modules/basic foundations is a bit of a mystery. I’m assuming it is in line code executed when the file is loaded/imported. Don’t understand why it’s there or how it gets used.

Show_doc is defined where? Why is there a <IPython.core.display.Markdown object> after the code fragment, apparently hanging out in the block with no apparent connection to anything? Is <IPython.core.display.Markdown object> telling the notebook how to display/publish the documentation extracted by show_doc?

Section reading a notebook is the first bit of code I’ve seen that ends up in the export.py file. Now it starts to make sense and I can see how the blocks end up in the tangled file.

Now I get confused again because I went and looked at the docs directory.

Why are there absolute URLs in index.html,
why are the docs directory contents half HTML but not the same as the live website?

Then there is another disconnect between notebook contents and the matching documentation page. In the notebook, you define read_nb but it’s not defined in the documentation, only reference to the function definition.

Having test code adjacent to the definition makes a lot of sense but in the documentation seems out of context because of missing information.

Yeah there’s a lot of things I don’t understand why they’re visible sometimes and not visible others. For example the regular expression statements are not visible but the block with assertions in it is.

I’m in stop here. I need to get some sleep but if there’s something basic about Jupyter notebooks that I need to know. Tell me where to look for the docs and I will read up before asking more questions

Ok solved by doing this in the nbs directory:

ln -s /home/marco/miniconda3/envs/TEST/lib/python3.7/site-packages/* .

And also modify .gitignore:

# Ignore the symlinks in the nbs folder, but include the notebooks
**/nbs/*
!**/nbs/*.ipynb

But then why does the build libs work? Is build docs called in a different way?

Anyway, for now this will be fine, now I can start building stuff using this awesome tool :smiley: because one of the main reasons I want to use it is because I have never in the past managed to write proper documentation and it ends up all over the place

3 Likes

I don’t understand your question. To build the docs, the library you’re building must be accessible somewhere (at least in the notebook folder with a simlink)