Nbdev_clean fails cryptically

nbdev
1

I’m getting this from nbdev_clean (and hence from nbdev_prepare):

…/nbplanetary2 ❯ nbdev_clean
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:110: UserWarning: Failed to clean notebook
  warn(f'{warn_msg}')
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)
/home/ayek72/mambaforge/envs/py310/lib/python3.10/site-packages/nbdev/clean.py:111: UserWarning: clean_ids
  warn(e)

I pushed it up nevertheless and have my GH workflow failing with:

Run if [ -n "$(nbdev_diff_nbs)" ]; then echo -e "!!! Detected difference between the notebooks and the library"; false; fi
  if [ -n "$(nbdev_diff_nbs)" ]; then echo -e "!!! Detected difference between the notebooks and the library"; false; fi
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.13/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.8.13/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.13/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.13/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.13/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.13/x64/lib
!!! Detected difference between the notebooks and the library
Error: Process completed with exit code 1.

My repo is public, if somebody wants to have a look:

2

Hello @michaelaye I was not able to repro the error on your GitHub repo.

I would install the latest of all dependencies from github

  • fastcore
  • ghapi
  • execnb
  • nbdev

echo fastcore ghapi execnb nbdev | xargs -n1 -I {} pip install -Uq git+https://github.com/fastai/{}.git

2 Likes
3

I faced the same problem reported by @michaelaye but @hamelsmu suggested solution fixed for me.

But before using @hamelsmu suggestion, I tried

pip3 install --upgrade --force-reinstall nbdev

and it didn’t make the problem go away. Shouldn’t the command above update nbdev dependencies as well? How would I know which ones (such as fastcore ghapi execnb nbdev) to update in the future?

The same issue is happening at fastai/workflow/nbdev-ci. Example: https://github.com/thigm85/learntorank/runs/8162841471?check_suite_focus=true

4

A regular pip install will grab the latest versions from pypi, and I believe if you have a minimum version that satisfies the dependency spec it wont update those dependencies. The command I showed above forces you to use everything at the bleeding edge, which is recommended as we are rapidly squashing bugs and fixing things at the moment.

When things slow down a bit more, I don’t anticipate you would have to install everything from git – but for right now, I recommend it

5

Good point about CI, @jeremy I think you may have seen this before, perhaps a new release might address this?

6

Yes, the issue is that it also happens on the CI, which has the following code:

  python -m pip install --upgrade pip
  if [ $USE_PRE ]; then
    pip install -U git+https://github.com/fastai/fastcore.git
    pip install -U git+https://github.com/fastai/ghapi.git
    pip install -U git+https://github.com/fastai/execnb.git
    pip install -U git+https://github.com/fastai/nbdev.git
  else
    pip install -U nbdev
  fi

so it should work.

7

Yes by default USE_PRE is set to false, you can turn it on like this but I don’t want you to have to fiddle with that

8

Yes, not ideal but good to know. Thanks.

1 Like
9

Btw, I set USE_PRE to true and got an uninformative notebook test error that I am not able to reproduce locally with nbdev_test

10

Updating the tools as instructed fixed my local nbdev_clean issue. I pushed the changes again, and my workflow still fails with above error message. Not sure what I should check now?

Note that nbdev_test fails with n_workers>1 due to my code using dask.distributed. with n_workers==1 it runs through, but that doesn’t produce any changes that could be missing to explain the error in the workflow. So I’m at a loss what to look at. :frowning:

11

Yes i think we are waiting for a release to be cut for nbdev
In the meantime, you can set clean_ids = True in your settings.ini file

1 Like
12

I was thinking another issue is that i had an old nbdev v2 workflow on GH? So I’m in the process of updating it, but it’s not so easy as my install is a bit more difficult.

On a side note, I wasted a day on the fact that Github workflow booleans are not behaving as real booleans :frowning:

13

even with clean_ids in settings.ini

I continue to have this error message:

$ nbdev_clean
/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/site-packages/nbdev/clean.py:112: UserWarning: Failed to clean notebook
  warn(f'{warn_msg}')
/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/site-packages/nbdev/clean.py:113: UserWarning: clear_all
  warn(e)
/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/site-packages/nbdev/clean.py:113: UserWarning: clear_all
  warn(e)

I use nbdev v2.3.2 but had the same behaviour with v2.2.10

fastcore             1.5.24
ghapi                1.0.3
execnb               0.1.3
nbdev                2.3.2
nbdev-numpy          0.0.292
nbdev-stdlib         0.0.309
14

Try with latest fastcore? I think this was fixed in fastcore#482 in version 1.5.25.

15

Thank you! That was it.
Does it mean nbdev should depend on fastcore>=1.5.25 ?

1 Like
16

That’s probably a good idea - just added that to the latest release.

1 Like