Nbdev discussion

[florobax]

Oh ok thanks !

EDIT: I actually still have kind of the same problem. The link to the page is of form user.github.io/module.submodule.html#function while it seems it should be user.github.io/libname/module.submodule#function

EDIT 2: It works with or without the .html but still, the libname appears to be missing in the automated hyperlink for me.

[florobax]

Totally unrelated question: how do you group modules in the table of content of the docs ? For instance, in fastai2, you have all Data modules grouped, but I can’t find how you accomplished this though the notebooks.

[sgugger]

The sidebar is generated from the file sidebar.json in the docs folder. To use a custom one you need to:

• write it (see the fastai2 sidebar.json file for inspiration)
• set custom_sidebar = True in your settings.ini file (so that the file is not erased by the auto-generated one)

As for your first question, the links generated are locals (aka we generate module.submodule.html#function) so they should work with where your site is located.

[florobax]

Nice thanks !

[florobax]

Another question (sorry): by default, when a class subclasses another, it writes the docstring of the parent class in the docs (see fastai2’s DataLoader class). Is there any way to avoid this behaviour ?

[florobax]

Another question (sorry): by default, when a class subclasses another, it writes the docstring of the parent class in the docs (see fastai2’s DataLoader class). Is there any way to avoid this behaviour ?

Found it, just add an empty docstring to the class (or a custom one).

[miko]

Apologies for the extremely vague question. I have the feeling that it would be possible to effectively integrate Nbdev with the newly open sourced metaflow https://medium.com/netflix-techblog/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9
To obtain an environment that allows the development of a fastai based data science project from basic prototyping to production using only Jupyter notebooks.

Sadly, I am not able to think about how that would look like and even if and why it would be a good idea. Anybody has ideas?

[miwojc]

Is nbdev made especially for a library development work (like fastai2) or would it be also recommended for a deep learning project work as well? for example a kaggle competition?

[jeremy (Jeremy Howard)]

I find it helpful to extra out useful functions from notebooks, even just for stuff like kaggle comps. So yes I would say it’s worth trying.

[Yorke (Connor Martin)]

do you mean setup.py, or setting.py? I’m not sure how to go about fixing this. tried setting up new conda environments without luck. /noob

[Behtash]

nbdev is super useful. Thank you ! Is nbdev intended to work only with packages on PyPI? Can we include dependency_links in setup.py or elsewhere? Are there any reservations for non PyPI packages?

[sgugger]

You can change the setup file to suit your needs, but by default it’s pip packages only, yes.

[sachinruk (Sachin)]

Hi,

First of all nbdev has been amazing! and I just published my first pypi repository however I’m running into some trouble.

1. When I run make release I get the error make: *** No rule to make targetrelease’. Stop.. However,make pypi` works fine. I’m guessing they work the same way so it’s not really an issue right now.
2. My side navigation list isn’t working properly as seen on: https://sachinruk.github.io/ProFeTorch/. For instance:
   1. There is only one sub-page named ‘Title’, whereas it was supposed to be called ‘Model’.
   2. There was other pages called ‘Blocks’ for example and can be accessed manually (https://sachinruk.github.io/ProFeTorch/blocks). However again, the top is called ‘Title’ instead of ‘Blocks’ as expected. The associated notebook is shown here on github.

Is the fact that I have my usual (inactive) blog on https://sachinruk.github.io affecting how things are working here? Should I be deleting my blog?

[sgugger]

Hi there! Glad you like nbdev
For 1, it’s probably because you generated your MakeFile before we added that functionality. The MakeFile is generated only once to allow you to customize it to your needs. So delete it, make sure you have the latest nbdev, then run nbdev_build_lib, that will create a new updated MakeFile and you should be good.
For 2, the problem is in your notebook: you need to follow the format:

# Title
> Summary

Here you didn’t put the > in front of the summary.

[sachinruk (Sachin)]

Thanks for that and fixed it. Have some new questions now (sorry). I did upgrade to the latest nbdev.

1. If you go to my docs the source links are pointing to fastai instead of my user tag on github. Did I miss something when creating notebooks?
2. In order to import my model I have to do from profetorch.model.model import Model because my class Model is under profetorch/model/model.py. Now I know that if I put from .model import * in profetorch/model/__init__.py this will ‘fix’ the problem so to speak. But I’m guessing this was not the intended usage when I’m using nbdev. What is the nbdev way of achieving this effect?
3. My equations aren’t rendering when I use \begin{aligned}...\end{aligned}. When I paste this on the katex website it seems it seems to render fine.
4. Minor feature request: Is there a way to remove the fastai logo/ icon and use our own icon on the tabs/ bottom right of page.

Thanks again for your help.

[sgugger]

For 1, you have to set your user in the ‘settings.ini’ file (you might have an older version that is buggy, check that the line for git_url, you should see https://github.com/%(user)s/%(lib_name)s/tree/%(branch)s/).

For 2, if you’re using submodules, it’s logical you have to go that far and it’s standard python (you could just name your model module model instead of model.model). You can of course change the init files to create shorcuts.

For 3, I’m not too sure since I didn’t do the katek integration. It should be surrounded by $$.

For 4, just change the images in docs/images

[bezova]

Please help with ideas where to look for an error: I can not get pass the diff library/notebooks error while I push to git and CI/build run:

Run if [ -n “$(nbdev_diff_nbs)” ]; then echo -e “!!! Detected difference between the notebooks and the library”; false; fi
shell: /bin/bash -e {0}
env: pythonLocation: /opt/hostedtoolcache/Python/3.6.9/x64 !!! Detected difference between the notebooks and the library ##[error] Process completed with exit code 1.

I rerun all notebooks, deleted lib folder, run nbdev_build_lib that created fresh lib folder.
in addition to auto-generated files the lib folder also has my imports.py file that is called by the notebooks, but I’ve seen same structure in fastai libs and do not expect this should create a problem.
When I try to run diff_nb_script() from notebook it returns error that might be Windows specific

[WinError 2] The system cannot find the file specified

and attempt to go line by line stuck on subprocess.run(..) (again looks like just windows).
I also noticed that temporary folders that are being compared contain /.ipynb_checkpoints that are not removed like __pycache__

[sgugger]

I also noticed that temporary folders that are being compared contain /.ipynb_checkpoints

This is weird since the built library should not have notebooks. Behind the scenes this script rebuild the library from the notebooks in a temp dir and compares to your lib folder. I suspect you have a /.ipynb_checkpoints there for some reason.

As for the WinError, it’s a bit hard to fix without seeing more. Do you have a bigger stacktrace?

[jeremy (Jeremy Howard)]

do you mean setup.py, or setting.py

Sorry I meant settings.ini .

[bezova]

here the full error trace, but I have very little idea where to dig further

FileNotFoundError Traceback (most recent call last)
in
1 #hide
2 from nbdev.sync import script2notebook, diff_nb_script
----> 3 diff_nb_script()

c:\users***\nbdev\nbdev\sync.py in diff_nb_script()
139 for d in [d1, d2]:
140 if (Path(d)/‘pycache’).exists(): shutil.rmtree(Path(d)/‘pycache’)
–> 141 res = subprocess.run([‘diff’, ‘-ru’, d1, d2], stdout=subprocess.PIPE)
142 print(res.stdout.decode(‘utf-8’))

~\AppData\Local\Continuum\anaconda3\lib\subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
486 kwargs[‘stderr’] = PIPE
487
–> 488 with Popen(*popenargs, **kwargs) as process:
489 try:
490 stdout, stderr = process.communicate(input, timeout=timeout)

~\AppData\Local\Continuum\anaconda3\lib\subprocess.py in init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
798 c2pread, c2pwrite,
799 errread, errwrite,
–> 800 restore_signals, start_new_session)
801 except:
802 # Cleanup if the child failed starting.

~\AppData\Local\Continuum\anaconda3\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
1205 env,
1206 os.fspath(cwd) if cwd is not None else None,
-> 1207 startupinfo)
1208 finally:
1209 # Child is launched. Close the parent’s copy of those pipe

FileNotFoundError: [WinError 2] The system cannot find the file specified