Nbdev discussion

nbdev
64

Thanks a lot for help and nbdev! I will share this with my teammate and I would be more than happy to contribute once we have something that works :slight_smile:

65

Hi,

Iā€™m trying out nbdev and I have the following error when doing: make pypi

First it complained that twine was not installed.
I had to pip install twine, now this.
May be there are some dependencies issues, wrong libraries version or something like that?

adding 'lngen-0.0.1.dist-info/WHEEL'
adding 'lngen-0.0.1.dist-info/METADATA'
adding 'lngen-0.0.1.dist-info/RECORD'
twine upload --repository pypi dist/*
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/keyring/backend.py", line 203, in _load_plugins
    init_func = ep.load()
  File "/usr/local/lib/python3.6/dist-packages/importlib_metadata/__init__.py", line 94, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/keyrings/alt/Windows.py", line 9, in <module>
    from . import file_base
  File "/usr/lib/python3/dist-packages/keyrings/alt/file_base.py", line 13, in <module>
    from keyring.util.escape import escape as escape_for_ini
ModuleNotFoundError: No module named 'keyring.util.escape'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/twine", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/twine/__main__.py", line 25, in main
    return dispatch(sys.argv[1:])
  File "/usr/local/lib/python3.6/dist-packages/twine/cli.py", line 72, in dispatch
    main = registered_commands[args.command].load()
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/dist-packages/twine/commands/upload.py", line 20, in <module>
    from twine import settings
  File "/usr/local/lib/python3.6/dist-packages/twine/settings.py", line 22, in <module>
    from twine import auth
  File "/usr/local/lib/python3.6/dist-packages/twine/auth.py", line 6, in <module>
    import keyring
  File "/usr/local/lib/python3.6/dist-packages/keyring/__init__.py", line 1, in <module>
    from .core import (
  File "/usr/local/lib/python3.6/dist-packages/keyring/core.py", line 192, in <module>
    init_backend()
  File "/usr/local/lib/python3.6/dist-packages/keyring/core.py", line 96, in init_backend
    filter(limit, backend.get_all_keyring()),
  File "/usr/local/lib/python3.6/dist-packages/keyring/util/__init__.py", line 22, in wrapper
    func.always_returns = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/keyring/backend.py", line 216, in get_all_keyring
    _load_plugins()
  File "/usr/local/lib/python3.6/dist-packages/keyring/backend.py", line 207, in _load_plugins
    log.exception("Error initializing plugin %s." % ep)
TypeError: not all arguments converted during string formatting
Makefile:17: recipe for target 'pypi' failed
make: *** [pypi] Error 1

When I tried to pip install keyring, it said that it is skipped:

$ pip install keyring
Keyring is skipped due to an exception: not all arguments converted during string formatting
Requirement already satisfied: keyring in /usr/local/lib/python3.6/dist-packages (20.0.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from keyring) (1.2.0)
Requirement already satisfied: secretstorage; sys_platform == "linux" in /usr/lib/python3/dist-packages (from keyring) (2.3.1)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->keyring) (0.6.0)
Requirement already satisfied: more-itertools in /usr/local/lib/python3.6/dist-packages (from zipp>=0.5->importlib-metadata; python_version < "3.8"->keyring) (8.0.2)
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.

Any suggestion on what to try next?

66

This isnā€™t an nbdev-specific issue, but according to google you can fix it with

pip3 install --upgrade keyrings.alt
1 Like
67

Oh! thank you very much! It resolved the issue.

68

how did you fix this exactly? Iā€™m running into the same thing

I do the first step after the pip install

click here: [nbdev template]

and i get an error after a minute :

Process completed with exit code 1.
69

Thank you once again for this wonderful tool!

One questionā€¦ How can I pass the dependencies of my library to the setup.py file, without editing it directly? Should I use the requirements field in the settings.ini file as a comma separated list? For example, how should I pass torch and torchvision as a requirement?

1 Like
70

When Iā€™m building docs and use github pages to host the generated .html files I get the following:

The index page is of the format:
https://github_username.github.io/lib_name

Then, to navigate to the test module page for example from the sidebar the link is of the format:
https://github_username.github.io/lib_name/test

Thus, when I follow the table of contents links for a function named test_equal for example, Iā€™m sent to:
https://github_username.github.io/lib_name/test#test_equal

But when I press the hypelink inside the documentation for this function I get redirected to:
https://github_username.github.io/test.html#test_equal

which causes a 404 error. Is it a bug or am I missing something?

71

Iā€™m trying to get nbdev to ignore some cells during testing.

Here is what the docs say, but I am unclear as to what specifically should I change in my settings.ini:

Everything that is not an exported cell is considered a test, so you should make sure your notebooks can all run smoothly (and fast) if you want to use this functionality as the CLI. You can mark some cells with special flags (like slow) to make sure they are only executed when you authorize it. Those flags should be configured in your settings.ini (separated by a | if you have several of them). You can also apply a flag to one entire notebook by putting # all_flag in one of its cells.

1 Like
72

Thanks for making this amazing tool!

If anyone has any experience with making a Conda package from this, Iā€™d be really happy to discuss it (and hopefully publish a tutorial for others). pip isnā€™t getting my dependencies right and with Conda it works much better. However, Iā€™m not sure how to create a Conda package from the output of nbdev; conda skeleton pypi my_pkg fails with

Traceback (most recent call last):
  File "setup.py", line 18, in <module>
    assert o in cfg, "missing expected setting: {}".format(o)
AssertionError: missing expected setting: version
1 Like
73

Hi guys, thanks for making this amazing tool.

I have installed it, but getting this error when running nbdev_build_docs:

converting: /home/marco/mlserver2/00_core.ipynb
An error occurred while executing the following cell:
------------------
#export
from nbdev.showdoc import show_doc
from mlserver2.core import *
------------------

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-9d176fd58c56> in <module>
      1 #export
----> 2 from nbdev.showdoc import show_doc
      3 from mlserver2.core import *

ModuleNotFoundError: No module named 'nbdev'
ModuleNotFoundError: No module named 'nbdev'

However, I can import nbdev, and nbdev_build_libs works correctly. I can import that code from a python CLI.

I cloned the fastai2 repo and get the same error trying to build the docs there

2 Likes
74

@dzlob You need to put the flags you want to use separated by | in the variable tst_flags. For instance, fastai2 has the line:

tst_flags = slow|cuda|cpp

(or it will very soon since I realize itā€™s missing!)

@adamh We will add conda support (and make it as easy as release on pypi for the packages you build with nbdev) very soon

@nextM Do you ahve this problem after doing a dev install or a pip install?

75

It should be a whitespace separated list, since itā€™s passed to str.split and then directly to setuptools.setup.

BTW you may find the fastai2 settings.ini useful to see how things work:

1 Like
76

Itā€™s a bug. Weā€™re looking into it.

77

Thanks! Thatā€™d be a game changer.

78

Fixed now (with an editable install of nbdev after pulling master).

79

still working on this error. can anyone decipher this

 ERROR: Command errored out with exit status 1:
     command: /opt/hostedtoolcache/Python/3.6.9/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/runner/work/art-blender/art-blender/setup.py'"'"'; __file__='"'"'/home/runner/work/art-blender/art-blender/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /home/runner/work/art-blender/art-blender/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/runner/work/art-blender/art-blender/setup.py", line 13, in <module>
        for o in expected: assert o in cfg, "missing expected setting: {}".format(o)
    AssertionError: missing expected setting: description
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
80

Youā€™re missing the description setting in your setting.py.

1 Like
81

@sgugger From either dev install or pip install, same issue.

I also tried with an earlier version of nbdev, same issue.

I tried installing pip nbdev in a different anaconda environment, and still saw the same issue.

Is it possible when it runs it is using the python environment of the machine?

I think the problem is coming from the #default_export tag because my 00_core.ipynb notebook is empty except for one dummy function tagged with #export (which I can see in the generated code file)

00_core.ipynb looks like:

# default_exp core

#export
def do_twice(func):
    def wrapper_do_twice(*args, **kwargs):
        func(*args, **kwargs)
        # make sure the wrapper function returns the return value of the decorated function
        return func(*args, **kwargs)
    return wrapper_do_twice

These lines are not in my notebook at all, are they added by the #default_export core command?:

from nbdev.showdoc import show_doc
from mlserver2.core import *
82

Iā€™ve been a fan of literate programming since my first code with tangle and weave. I was always disappointed that it never gained traction. I hope nbdev gets closer to delivering on the promise of literate programming.

To that end, any pointers on how to import existing code and applications into nbdev and re-factoring it? Also any pointers on how to think about literate programming in the nbdev context. For example, when I was using cweb, I wrote the code is if I was writing a book. I started out with the beginning outline and the code segment referred to other fragments later in the narrative. . The beginning section will look something like

=====

This little app takes a set of templates, one to a calendar event and data for each calendar event then expands them using recursive template expansion.

Make ENET monthly meeting blast

<Set up event iteration>
<for each event>
  <create template instance>
  <expand template>
  <generate time key>
  <sort header by skip>

The phrase between the <> characters refers to the section containing the detail and narrative. Every section could have a mixture of code and section references. Using my code base is an example, the first segment would be something like"

set up event iteration:
in order to iterate over the calendar events, gather up a list of event files, baseline template, and create dictionary with common data elements used in template expansion.
ā€” Code sectionā€”
events=glob.glob("./*.event")

with open("event_template.rtmpl") as f:
    event_template=f.read()

common_dict = rTemplate.colon_parser("event_common", EOL=False)

event_dict = {}
nonevent_dict = {}

subsequent sections would fill in the detail outlined in the beginning and the tangle command would organize it in a compiler friendly order.

How do I reframe my literate programming experience to fit the nbdev model?

83

I think this is a thing with my anaconda maybe? I get similar errors when running the notebook 03_export2html from the nbdev repo. It says nbformat not installed, but it definitely is.