Nbdev discussion

Thanks for the answer! Here’s a snippet of the changes I’m seeing in the HTML files:

diff --git a/docs/clean.html b/docs/clean.html
index 071872e..d5882cf 100644
--- a/docs/clean.html
+++ b/docs/clean.html
@@ -159,7 +159,7 @@ summary: "Strip notebooks from superfluous metadata"
 <div class="output_markdown rendered_html output_subarea ">
 <h4 id="clean_nb" class="doc_header"><code>clean_nb</code><a href="https://github.com/fastai/nbdev/tree/master/nbdev/clean.py#L36" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>clean_nb</code>(<strong><code>nb</code></strong>, <strong><code>clear_all</code></strong>=<em><code>False</code></em>)</p>
 </blockquote>
-<p>Clean <code>nb</code> from superfulous metadata, passing <code>clear_all</code> to <a href="/clean#clean_cell"><code>clean_cell</code></a></p>
+<p>Clean <code>nb</code> from superfulous metadata, passing <code>clear_all</code> to <code>clean_cell</code></p>
 
 </div>

For some reason the reference "/clean#clean_cell" is being removed, but curiously if I discard the changes and rerun nbdev_build_docs the diffs do not reappear. It could well be something on my end, so let me try to investigate further.

I’m really enjoying nbdev, thank you for developing it :slight_smile:

I’m trying to add a package to requirements which isnt on pypi but is on github. The package I’m trying to add is https://github.com/shchur/gnn-benchmark and I’ve tried the following (and a bunch of variants to no success)

requirements = networkx numpy matplotlib scipy git+git://github.com/shchur/gnn-benchmark.git

I got this string from https://stackoverflow.com/questions/16584552/how-to-state-in-requirements-txt-a-direct-github-source since the settings.ini file says I should use the same format as setuptools requirements. I get the following error when I try and do pip install . (the CI gives the same when I push to GitHub):

ERROR: Command errored out with exit status 1:
 command: /anaconda3/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/8s/pw80r_bj4tv87rqttkkk90200000gp/T/pip-req-build-joogxlf6/setup.py'"'"'; __file__='"'"'/private/var/folders/8s/pw80r_bj4tv87rqttkkk90200000gp/T/pip-req-build-joogxlf6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/8s/pw80r_bj4tv87rqttkkk90200000gp/T/pip-req-build-joogxlf6/pip-egg-info
     cwd: /private/var/folders/8s/pw80r_bj4tv87rqttkkk90200000gp/T/pip-req-build-joogxlf6/
Complete output (1 lines):
error in grapht setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+git://g'"
----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Would anyone know how to fix this? Cheers!

@lewtun Many Thanks for your prompt response. Works fine now

Quick question:

Say I want to just export an example notebook (without a cfg etc) would this be possible? For exact reasoning, for one of my lessons I want to show how to use #export to help build a library (and convert our ipynb to a .py file). How would I go about this? As I cannot run notebook2script without a Configuration file of sorts. Thanks!

1 Like

hi @hennesseeee, i think you need to add the git repo via the dependency_links argument of setup tools: https://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi

1 Like

There is no script that does that directly, no (though having a config file is a very minimal requirement).

1 Like

Hi @lewtun, thanks for your response and suggestion. I tried adding the following line

dependency_links=[‘https://github.com/shchur/gnn-benchmark/tarball/master#egg=gnnbench-0.1.0’],

in the setup.py along with gnnbench in requirements (I also tried gnnbench==0.1.0) but both give me the following

ERROR: Could not find a version that satisfies the requirement gnnbench (from grapht==0.0.1) (from versions: none)
ERROR: No matching distribution found for gnnbench (from grapht==0.0.1)

As a workaround I’ve changed

requirements = cfg.get(‘requirements’,’’).split()
to
requirements = cfg.get(‘requirements’,’’).split() + [‘gnnbench @ https://github.com/shchur/gnn-benchmark/tarball/master#egg=gnnbench-0.1.0’,]

which seems to work. It looks like with later versions of pip you should be able to have a single string in the requirements which is what my workaround is based (https://stackoverflow.com/a/54216163/2453167). The parser which parses settings.ini seems to have issues if I put gnnbench @ https://github.com/shchur/gnn-benchmark/tarball/master#egg=gnnbench-0.1.0 directly in requirements

Update: this fix doesn’t quite work… its install the base gnnbench package but not the submodules…

Thanks :slight_smile: I figured it out:

create_config('nbdev', user='muellerzr', path='.', cfg_name='settings.ini')
cfg = Config(cfg_name='settings.ini')
from nbdev.export import reset_nbdev_module, notebook2script
reset_nbdev_module()
notebook2script('testingNBDev.ipynb')
2 Likes

Hello! First of all thanks a lot for all the work that went into this. I am really excited about nbdev. But there is one problem I have. It seems the notebook has to get autosaved in order to build the package correctly.
This is annoying because sometimes notebook2script() has no effect if the notebook did not get saved.
You can test this by modifying a nb with an export and an notebook2script() and execute it in short time intervals.

Is there any special configuration I can enable to make sure this compiles better and more immediate ??? Has anybody else noticed the same problem?

2 Likes

Hello all - it’s a fantastic and interesting approach, thanks to all contributors!

Is there a good way to break “big cells” (e.g. classes) into multiple cells so py file and related documentation can still be generated. I am aware of this “hack”

class MyClass():
    def method1(self):
        print("method1")

and in the next cell:

class MyClass(MyClass):
    def method2(self):
        print("method2")

Does anyone have a better idea or approach ? Many thanks!

hey @hennesseeee, i’m not sure whether this will solve your problem but have you tried upgrading setuptools as follows:

pip install setuptools -U

I’ve sometimes found that this was the main source of errors with pip install

@Andreas_Daiminger Well yes, you need to save the notebooks for the changes to take effect, the same way you would need to save your python files. Note that in command mode, you just need to hit S to save.

@tinoka You can use the patch decorator from fastcore to achieve this.

This is remarkable. As an extension we should be able to export few selected cells from a single notebook into a script using #export? It would be wonderful if we could put in notebook2script() as the last cell of a notebook with necessary imports (but without all the other bells and whistles and repo cloning) and we get a .py with only those export cells as output.

1 Like

That’s essentially what happens, we need a configuration file to note where everything is being done to but otherwise that’s exactly what happens :slight_smile:

See my solution a few posts down

1 Like

Tried with your approach but could not get it to work. However, here is a similar effort. I now have the code listed in the link in my project directory and then I call the notebook2script function from the export_nb_cells.py file using from export_nb_cells import notebook2script.

This is a rather neat solution. Will explore this approach a bit more; conceptually, I find the ability to experiment using a notebook and once I figure out how to do things, export the selected cells from the notebook to a script very powerful.

I made sure my setuptools and pip were up to date. I think it may actually be a issue with the package because I can’t seem to install it from command line as I normally do (I have to git clone and then install from the downloaded repo).

Regarding other packages installed from github such as pygsp (which is on pypi but the github version is more up to date) I’ve managed to get it to install with the nbdev package (and pass CI) if I edit setup.py and change the line

requirements = cfg.get(‘requirements’,’’).split()

to

requirements = cfg.get(‘requirements’,’’).split() + [‘pygsp @ https://github.com/epfl-lts2/pygsp/tarball/master’]

I’m not sure if there is a cleaner was to do this by just editing settings.ini.

Thanks for your help!

I’m getting a similar error to Marco. I’m trying to run tests after installing fastai2. Before installing, I delete my fastai2 miniconda directory, restart the terminal, and check pip list and conda list in my base conda environment to confirm nbdev, fastai2, and fastcore aren’t there. I tried a number of combinations (for nbdev, fastai2, and fastcore) of editable and packaged (conda/pip) installs. All tests fail because nbdev_test_nbs can’t find fastai2:

ModuleNotFoundError: No module named 'fastai2'

What I did to check this as cleanly as possible was:

git clone https://github.com/fastai/fastai2
cd fastai2
conda env create -f environment.yml
source activate fastai2
pip install -e '.[dev]'
nbdev_test_nbs

Looks like because of MacOS’s zsh, the .[dev] has to be in quotes.

@sgugger does this seem familiar? I’ll update if I get it working.


update: nbdev_test_nbs works inside the nbdev root folder, after running a dev-install, via:

pip install packaging
git clone https://github.com/fastai/nbdev
pip install -e nbdev

«cd to /nbdev»

nbdev_test_nbs

So… nbdev_test_nbs didn’t work in /fastai2 after the above, and I wasn’t able to run it again in the /nbdev root folder afterwards, though my terminal history seems to point there. It did successfull run in /nbdev/nbdev. Are the tests supposed to only work there?


update: I tried running nbdev_test_nbs in /fastai2 with a fresh git-clone and package-install of everything in a new conda environment – no luck. Maybe… the tests aren’t supposed to be working yet? But that’d mean the nbdev I have isn’t working at all if it can’t find fastai2 … guess I’ll find out next.

No the tests are supposed to work everywhere. It looks like you have a problem in your enviromnent so I’d suggest creating a new one and reinstalling.

So… I got it working, but the way to do it was very weird. I’ll explain it here in case anyone else with a long-term conda system on MacOS has similar problems.


I’ve been reinstalling fresh conda environments before each attempt, before my last post. I later tried deleting and reinstalling Miniconda itself twice. I finally got the tests working (it’s able to import fastai2) by deleting every hidden file/folder I could find relating to jupyter, conda, or ipython; ontop of a fresh conda reinstall.

Now all nbdev_test_nbs tests pass in /nbdev and /fastcore. 17 notebooks fail in /fastai2 → these seem more ’normal’: the result of CUDA tests on a cpu-only system, or maybe import failures from wrong versions (“No module named 'wandb’”, “cannot import name 'PILLOW_VERSION' from 'PIL’”, and “Torch not compiled with CUDA enabled” for example).

This is with a total clean install of miniconda — my base env doesn’t even have Jupyter on it yet. To get it working (along with a bunch of restarts to be on the safe side) what I did was:

  • delete my Miniconda root directory
  • delete every hidden folder and file I could find in my Home directory that seemed relevant:
    • .jupyter; anything with ‘ipython’, ‘conda’, etc.

I already commented out any conda-related lines in my .zshrc (MacOS equivalent of .bashrc or .bash_profile). Then redownload Miniconda, verify, install, and (since it isn’t configured to write to Zsh) copy its conda-initialization lines from ~/.bash_profile to ~/.zshrc. Checking pip list and conda list after restarting Terminal, the base conda env only has a handful of packages.

Then I installed fastai2 to its own env, along with fastcore. I don’t remember if I used packaged or editable, I think I tested both. nbdev_test_nbs worked, with the new errors mentioned above; I dev-installed nbdev and its own tests passed.

This method deletes all Jupyter customizations and conda environments.


Where I think the issue was:
I still had errors even after deleting and reinstalling conda on my system. I noticed in Jupyter there were old ‘named’ kernels available in the dropdown menu. Kernels for deleted environments. This likely meant ipython/ipykernel had a config file pointing there, and was being used by new Jupyter & etc installations. I only had one for “fastai” and a couple unrelated kernels like Scala … so I don’t know how this tripped up nbdev_test_nbs into being unable to import fastai2.

What contributed confusion early in this process was not knowing how “developer”/“editable” pip/conda/python installs work. I’m used to the pre-0.7 course-v1 workflow of freely editing files in a fastai folder; and I wasn’t sure if dev-installs went to the base/system python ‘env’ or to the active env (it’s the active conda env). Also, MacOS’s switch to ZShell as its default from Bash means that the pip install -e .[dev] line doesn’t work → the brackets must be in quotes (double or single): pip install -e '.[dev]', and I didn’t know the significance of “.[dev]” since pip install -e . appaeared to run just fine — so I thought “.[dev]” could’ve been some informal programmer shorthand. Not the case.


Hopefully this helps anyone facing a similar problem.


Note: looks like the new PIL (7.0.0) doesn’t work with Torchvision, as of 3 January, → but the PyTorch devs are aware and plan to update (PyTorch and Torchvision) to fix it this week (the PIL & PILLOW_VERSION error). Until then this: pip install "pillow<7" seems to work.


edit: is fastai2 currently not meant to work on CPU-only systems? After installing a few dependencies (seemingly not taken care of by the environment.yml fastai2 env repo install):

pip install wandb
conda install -c fastai -c pytorch fastai
pip install tensorboard

the only tests that fail, seem to all be CUDA related. Only 6 fail now:

  • 00_torch_core.ipynb
  • 13_learner.ipynb
  • 18_callback.fp16.ipynb
  • 22_tutorial.imagenette.ipynb
  • 32_text.models.awdlstm.ipynb
  • 45_collab.ipynb

Interesting, since Apple has dropped all CUDA support… I wonder if it’s possible to install a cuda-gpu version of PyTorch on MacOS just for code compatibility. I’ll update here if I try it out.

Hi, this seems great so far! I was just wondering:

  1. How do you get the [source] link to show up beside your classes and functions in the docs? I tried comparing my notebook file with the tutorial one but could not find how it was done.
  2. Is there a way to customize the style the way that the nbdev docs are done (the blue lines between headers etc.)

Thanks