Nbdev with Google Colaboratory: A Tutorial

Hi Everyone! I’ve made a quick video describing two methods for using nbdev along with Google Colab:

The two methods are as follows:

  1. Download your notebooks to your system, and run Linux Subsystem (or whatever you want) to do notebook2script, etc, and push to GitHub

  2. Work entirely out of Colab and never leave a notebook instance to push. This is a bit harder to setup so I have made a template repository here to help you get started :slight_smile:

Two known issues:

  1. Tests will not pass, as it will show a connection error with the notebooks. I’m unsure how to fix this, but if anyone finds a solution while working on it please let me know and put a PR in! :slight_smile:
  2. Updating index.ipynb doesn’t seem to update the readMe. I’m unsure what I’ve missed here but otherwise everything should work as it should.

I hope this helps you get started with using Colab :slight_smile:

One word of caution: Be careful not to share your local directory in your Google Drive, as this will contain your credentials for your GitHub account!

13 Likes

this leaves lots of functionalities out…

For example, can we use nbdev_update_lib within colab?

As far as i understand, and i have just started looking at this tool, we are limited to push. How about the other features? Can we use them within Colab?

You can use any bash command in colab. Simply put a ! in front of it

Thank you for your kind reply. I have tried that, but i got this…

/usr/local/lib/python3.6/dist-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
warn(“IPython.utils.traitlets has moved to a top-level traitlets package.”)
This cell doesn’t have an export destination and was ignored:
e
This cell doesn’t have an export destination and was ignored:
e
This cell doesn’t have an export destination and was ignored:
e
This cell doesn’t have an export destination and was ignored:
e
Converted core.py.
Traceback (most recent call last):
File “/usr/local/bin/nbdev_update_lib”, line 8, in
sys.exit(nbdev_update_lib())
File “/usr/local/lib/python3.6/dist-packages/fastscript/core.py”, line 73, in _f
func(**args.dict)
File “/usr/local/lib/python3.6/dist-packages/nbdev/cli.py”, line 27, in nbdev_update_lib
script2notebook(fname=fname)
File “/usr/local/lib/python3.6/dist-packages/nbdev/sync.py”, line 126, in script2notebook
[ _script2notebook(f, dic, silent=silent) for f in files]
File “/usr/local/lib/python3.6/dist-packages/nbdev/sync.py”, line 126, in
[ _script2notebook(f, dic, silent=silent) for f in files]
File “/usr/local/lib/python3.6/dist-packages/nbdev/sync.py”, line 97, in _script2notebook
splits = _split(code)
File “/usr/local/lib/python3.6/dist-packages/nbdev/sync.py”, line 50, in _split
default_nb = re.search(f’File to edit: {prefix}(\S+)\s+’, lines[0]).groups()[0]
AttributeError: ‘NoneType’ object has no attribute ‘groups’

When trying out the following -

!pip install nbdev
from nbdev.showdoc import *

I get the following error -

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-11-e0e43e5288cc> in <module>
----> 1 from nbdev.showdoc import *

/opt/conda/lib/python3.7/site-packages/nbdev/__init__.py in <module>
      4 
      5 if IN_IPYTHON:
----> 6     from .flags import *
      7     from .showdoc import show_doc
      8     #from .export import notebook2script

/opt/conda/lib/python3.7/site-packages/nbdev/flags.py in <module>
    114            nbdev_collapse_input, nbdev_collapse_output, needs_local_scope(nbdev_add2all)]
    115     for fn in fns: register_line_magic(fn)
--> 116     for flag in Config().get('tst_flags', '').split('|'): _new_test_flag_fn(flag)

/opt/conda/lib/python3.7/site-packages/nbdev/imports.py in __init__(self, cfg_name)
     41         while cfg_path != cfg_path.parent and not (cfg_path/cfg_name).exists(): cfg_path = cfg_path.parent
     42         self.config_file = cfg_path/cfg_name
---> 43         assert self.config_file.exists(), "Use `create_config` to create settings.ini for the first time"
     44         self.d = read_config_file(self.config_file)['DEFAULT']
     45         add_new_defaults(self.d, self.config_file)

AssertionError: Use `create_config` to create settings.ini for the first time

You need to be inside of the directory you’re working in for that to work, as it’s tied to a nbdev library

While following this readme file, I do not find any change in the directory. Where should I change the directory to for the first time I set up the nbdev?

Those notebooks work along with the video, so please watch it. I cleaned them up inside the repo to get rid of anything that is not needed (and also a potential security vulnerability), so some parts are missing.

1 Like

Thanks. Was able to work through it after following the video.

1 Like

Hi @muellerzr, @jeremy , everyone (o:

Might it be better to have a single py package to provide colab support (rather than a different template to clone)? Borrowing a lot from Zacharys project, i’ve created nbdev_colab_helper which is really just

  • a py package
    • that makes it easy to to setup colab for running nbdev and github commands (with externalized passwords etc)
    • that could be made part of nbdev in future?
  • docs, a colab helper notebook
  • and a re-write of the tutorial to use only github, colab and google drive in the browser

There are quite a few TODOs but hopefully this is a step in the right direction.

What do you think?

4 Likes

@pete88b I’ve done that for fastai: https://course.fast.ai/start_colab.html

Does it make sense to combine ideas from them?

@pete88b just finished going thru all that. It looks terrific. I think personally it makes most sense to have it as a separate project, which we could link to. What do you think?

If you look at the Makefile for any fast.ai project you can see how we deploy pypi and conda packages .

Sounds good - I’ll take a proper look. I can see fastbook/__init__.py via fastbook? but where is the source in github?

I’d be happy to handle this as a separate project - even happier if @muellerzr or anyone else would like to collaborate (o:

@pete88b source repo is fastai/course20

1 Like

@hamelsmu might this way of working help to workaround your dependency issue?

Hope you don’t mind the mention - but it’d be great to get your feedback on this way of working.

Hi @pete88b Nice to meet you! I was able to follow your tutorial and think its great. Definitely think you should do a PR to add a section to https://docs.fast.ai/dev/! @jeremy what do you think?

@hamelsmu my thought above ^^^

1 Like

I have been working with nbdev in Google Colab using @pete88b 's great colab_helper notebook. It worked perfectly until about a week ago. Now, two of the functions crash with the same error message. The functions are: nbdev_build_docs() and nbdev_test_nbs(). The error message is:

TypeError                                 Traceback (most recent call last)
<ipython-input-18-e9e09cc99e52> in <module>()
  1 from nbdev.cli import nbdev_build_docs
----> 2 nbdev_build_docs()

4 frames
/usr/local/lib/python3.6/dist-packages/fastcore/script.py in _f(*args, **kwargs)
 66     def _f(*args, **kwargs):
 67         mod = inspect.getmodule(inspect.currentframe().f_back)
---> 68         if not mod: return func(*args, **kwargs)
 69 
 70         p = anno_parser(func)

/usr/local/lib/python3.6/dist-packages/nbdev/cli.py in nbdev_build_docs(fname, force_all, mk_readme, n_workers, pause)
100                      pause:Param("Pause time (in secs) between notebooks to avoid race conditions", float)=0.5):
101     "Build the documentation by converting notebooks mathing `fname` to html"
--> 102     notebook2html(fname=fname, force_all=force_all, n_workers=n_workers, pause=pause)
103     if fname is None: make_sidebar()
104     if mk_readme: make_readme()

/usr/local/lib/python3.6/dist-packages/nbdev/export2html.py in notebook2html(fname, force_all, n_workers, cls, template_file, exporter, dest, pause, execute)
582     else:
583         passed = parallel(_notebook2html, files, n_workers=n_workers, cls=cls,
--> 584                           template_file=template_file, exporter=exporter, dest=dest, pause=pause, execute=execute)
585         if not all(passed):
586             msg = "Conversion failed on the following:\n"

/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in parallel(f, items, n_workers, total, progress, pause, timeout, chunksize, *args, **kwargs)
710     "Applies `func` in parallel to `items`, using `n_workers`"
711     if progress is None: progress = progress_bar is not None
--> 712     with ProcessPoolExecutor(n_workers, pause=pause) as ex:
713         r = ex.map(f,items, *args, timeout=timeout, chunksize=chunksize, **kwargs)
714         if progress:

/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in __init__(self, max_workers, on_exc, pause, mp_context, initializer, initargs)
691         self.not_parallel = max_workers==0
692         if self.not_parallel: max_workers=1
--> 693         super().__init__(max_workers, mp_context=mp_context, initializer=initializer, initargs=initargs)
694 
695     def map(self, f, items, timeout=None, chunksize=1, *args, **kwargs):

TypeError: __init__() got an unexpected keyword argument 'mp_context'

Furthermore, !nbdev_fix_merge index.ipynb fails with the following message

Traceback (most recent call last):
  File "/usr/local/bin/nbdev_fix_merge", line 8, in <module>
    sys.exit(nbdev_fix_merge())
  File "/usr/local/lib/python3.6/dist-packages/fastcore/script.py", line 77, in _f
    func(**args.__dict__)
  File "/usr/local/lib/python3.6/dist-packages/nbdev/cli.py", line 157, in nbdev_fix_merge
    fix_conflicts(fname, fast=fast, trust_us=trust_us)
  File "/usr/local/lib/python3.6/dist-packages/nbdev/merge.py", line 96, in fix_conflicts
    start,cells,end = extract_cells(raw_text)
  File "/usr/local/lib/python3.6/dist-packages/nbdev/merge.py", line 14, in extract_cells
    while not lines[i].startswith(' "cells"'): i+=1
IndexError: list index out of range

Any help will be appreciated. The nebdev tool working in Google Colab, thanks to @pete88b, has been a great aid in development.

Try updating both nbdev and fastcore

Hi everyone,

I’ve pinned nbdev, fastcore and fastscript versions in nbdev_colab_helper - I think this makes sense for now as these libs are seeing lots of changes.

@joseadolfo feel free to at-mention me if this doesn’t fix your problem.

Pete

1 Like