hey @hennesseeee, i’m not sure whether this will solve your problem but have you tried upgrading setuptools as follows:
pip install setuptools -U
I’ve sometimes found that this was the main source of errors with pip install
@Andreas_Daiminger Well yes, you need to save the notebooks for the changes to take effect, the same way you would need to save your python files. Note that in command mode, you just need to hit S to save.
@tinoka You can use the patch decorator from fastcore to achieve this.
This is remarkable. As an extension we should be able to export few selected cells from a single notebook into a script using #export? It would be wonderful if we could put in notebook2script() as the last cell of a notebook with necessary imports (but without all the other bells and whistles and repo cloning) and we get a .py with only those export cells as output.
1 Like
That’s essentially what happens, we need a configuration file to note where everything is being done to but otherwise that’s exactly what happens 
See my solution a few posts down
1 Like
Tried with your approach but could not get it to work. However, here is a similar effort. I now have the code listed in the link in my project directory and then I call the notebook2script function from the export_nb_cells.py file using from export_nb_cells import notebook2script.
This is a rather neat solution. Will explore this approach a bit more; conceptually, I find the ability to experiment using a notebook and once I figure out how to do things, export the selected cells from the notebook to a script very powerful.
I made sure my setuptools and pip were up to date. I think it may actually be a issue with the package because I can’t seem to install it from command line as I normally do (I have to git clone and then install from the downloaded repo).
Regarding other packages installed from github such as pygsp (which is on pypi but the github version is more up to date) I’ve managed to get it to install with the nbdev package (and pass CI) if I edit setup.py and change the line
requirements = cfg.get(‘requirements’,’’).split()
to
requirements = cfg.get(‘requirements’,’’).split() + [‘pygsp @ https://github.com/epfl-lts2/pygsp/tarball/master’]
I’m not sure if there is a cleaner was to do this by just editing settings.ini.
Thanks for your help!
I’m getting a similar error to Marco. I’m trying to run tests after installing fastai2. Before installing, I delete my fastai2 miniconda directory, restart the terminal, and check pip list and conda list in my base conda environment to confirm nbdev, fastai2, and fastcore aren’t there. I tried a number of combinations (for nbdev, fastai2, and fastcore) of editable and packaged (conda/pip) installs. All tests fail because nbdev_test_nbs can’t find fastai2:
ModuleNotFoundError: No module named 'fastai2'
What I did to check this as cleanly as possible was:
git clone https://github.com/fastai/fastai2
cd fastai2
conda env create -f environment.yml
source activate fastai2
pip install -e '.[dev]'
nbdev_test_nbs
Looks like because of MacOS’s zsh, the .[dev] has to be in quotes.
@sgugger does this seem familiar? I’ll update if I get it working.
update: nbdev_test_nbs works inside the nbdev root folder, after running a dev-install, via:
pip install packaging
git clone https://github.com/fastai/nbdev
pip install -e nbdev
«cd to /nbdev»
nbdev_test_nbs
So… nbdev_test_nbs didn’t work in /fastai2 after the above, and I wasn’t able to run it again in the /nbdev root folder afterwards, though my terminal history seems to point there. It did successfull run in /nbdev/nbdev. Are the tests supposed to only work there?
update: I tried running nbdev_test_nbs in /fastai2 with a fresh git-clone and package-install of everything in a new conda environment – no luck. Maybe… the tests aren’t supposed to be working yet? But that’d mean the nbdev I have isn’t working at all if it can’t find fastai2 … guess I’ll find out next.
No the tests are supposed to work everywhere. It looks like you have a problem in your enviromnent so I’d suggest creating a new one and reinstalling.
So… I got it working, but the way to do it was very weird. I’ll explain it here in case anyone else with a long-term conda system on MacOS has similar problems.
I’ve been reinstalling fresh conda environments before each attempt, before my last post. I later tried deleting and reinstalling Miniconda itself twice. I finally got the tests working (it’s able to import fastai2) by deleting every hidden file/folder I could find relating to jupyter, conda, or ipython; ontop of a fresh conda reinstall.
Now all nbdev_test_nbs tests pass in /nbdev and /fastcore. 17 notebooks fail in /fastai2 → these seem more ’normal’: the result of CUDA tests on a cpu-only system, or maybe import failures from wrong versions (“No module named 'wandb’”, “cannot import name 'PILLOW_VERSION' from 'PIL’”, and “Torch not compiled with CUDA enabled” for example).
This is with a total clean install of miniconda — my base env doesn’t even have Jupyter on it yet. To get it working (along with a bunch of restarts to be on the safe side) what I did was:
- delete my Miniconda root directory
- delete every hidden folder and file I could find in my Home directory that seemed relevant:
- .jupyter; anything with ‘ipython’, ‘conda’, etc.
I already commented out any conda-related lines in my .zshrc (MacOS equivalent of .bashrc or .bash_profile). Then redownload Miniconda, verify, install, and (since it isn’t configured to write to Zsh) copy its conda-initialization lines from ~/.bash_profile to ~/.zshrc. Checking pip list and conda list after restarting Terminal, the base conda env only has a handful of packages.
Then I installed fastai2 to its own env, along with fastcore. I don’t remember if I used packaged or editable, I think I tested both. nbdev_test_nbs worked, with the new errors mentioned above; I dev-installed nbdev and its own tests passed.
This method deletes all Jupyter customizations and conda environments.
Where I think the issue was:
I still had errors even after deleting and reinstalling conda on my system. I noticed in Jupyter there were old ‘named’ kernels available in the dropdown menu. Kernels for deleted environments. This likely meant ipython/ipykernel had a config file pointing there, and was being used by new Jupyter & etc installations. I only had one for “fastai” and a couple unrelated kernels like Scala … so I don’t know how this tripped up nbdev_test_nbs into being unable to import fastai2.
What contributed confusion early in this process was not knowing how “developer”/“editable” pip/conda/python installs work. I’m used to the pre-0.7 course-v1 workflow of freely editing files in a fastai folder; and I wasn’t sure if dev-installs went to the base/system python ‘env’ or to the active env (it’s the active conda env). Also, MacOS’s switch to ZShell as its default from Bash means that the pip install -e .[dev] line doesn’t work → the brackets must be in quotes (double or single): pip install -e '.[dev]', and I didn’t know the significance of “.[dev]” since pip install -e . appaeared to run just fine — so I thought “.[dev]” could’ve been some informal programmer shorthand. Not the case.
Hopefully this helps anyone facing a similar problem.
Note: looks like the new PIL (7.0.0) doesn’t work with Torchvision, as of 3 January, → but the PyTorch devs are aware and plan to update (PyTorch and Torchvision) to fix it this week (the PIL & PILLOW_VERSION error). Until then this: pip install "pillow<7" seems to work.
edit: is fastai2 currently not meant to work on CPU-only systems? After installing a few dependencies (seemingly not taken care of by the environment.yml fastai2 env repo install):
pip install wandb
conda install -c fastai -c pytorch fastai
pip install tensorboard
the only tests that fail, seem to all be CUDA related. Only 6 fail now:
- 00_torch_core.ipynb
- 13_learner.ipynb
- 18_callback.fp16.ipynb
- 22_tutorial.imagenette.ipynb
- 32_text.models.awdlstm.ipynb
- 45_collab.ipynb
Interesting, since Apple has dropped all CUDA support… I wonder if it’s possible to install a cuda-gpu version of PyTorch on MacOS just for code compatibility. I’ll update here if I try it out.
Hi, this seems great so far! I was just wondering:
- How do you get the [source] link to show up beside your classes and functions in the docs? I tried comparing my notebook file with the tutorial one but could not find how it was done.
- Is there a way to customize the style the way that the nbdev docs are done (the blue lines between headers etc.)
Thanks
I’ve been using nbdev since the release on my laptop, and so far it’s been great.
I’ve just pulled my repo on another computer after installing nbdev as well, and it seems the behaviour of nbdev_build_lib has changed.
in my notebooks i have a bunch of import mylib.foo as foo, mylib being the lib_name from settings.ini
on my laptop, (with nbdev installed very early at the release, and not updated since), the generated code is stayed untouch as import mylib.foo as foo and it works great
on the fresh install, the code is modified to import .foo as foo and my python (3.7) is complaining with a SyntaxError: invalid syntax error message. It doesn’t like the .foo part.
Did you guys change how the imports are processed? How can I solve this syntax error?
This problem seems to have been introduced with fix #13
Python doesn’t allow relative imports to be of the form import foo or impor foo as bar. You need to use the syntax from foo import …
The latest version (master) of nbdev will return an error message with this when converting your notebooks.
I have some helper functions which are only used as subroutines to other functions. Is there a way to have these functions export to the built library without them appearing in the docs?
You should keep them private in that case (begin their name with an underscore). Any private class/function is not shown in the docs.
I can also add a special flag #export_noshow if there is a big interest for this.
2 Likes
Quick question: How are imports.py files generated using nbdev?
They are the (only) one we write manually.
I was sure =). Other quick question, can we patch_to class methods?
I realized that we don’t need weave for python. You can use things before they’re defined. I’ve actually started doing that now.
I find it convenient to know what order things are defined in. So if I run the notebooks in that order, each one only depends on the previous notebooks.