Run_notebook & notebook2script issue in notebook vs command line

Part 2 (2019)
1

I noticed something odd about using notebook2script.py and run_notebook.py on the command line vs. in a notebook. I’m not 100% sure if this isn’t due to my env or not.

I can run run_notebook.py from the command line only if I use a registered ipykernel in the target notebook. However, within a notebook I can only run notebook2script.py if I am using an auto-generated kernel.

I’m using auto-generated ipykernels (I think with nb_conda?) – so any conda env you have will show up as a kernel, without requiring you to manually register each one with:

(envname) $ python -m ipykernel install --name "envname"

Now, if I set the kernel as the auto-gen’d one, I can’t run run_notebook.py on command line because it can’t find conda-env-fastai-py, the name of the auto-gen’d kernel. This kind of makes sense, and from what I read seems like a current Jupyter limitation.

The confusing bit, is that I can’t run a cell containing

!python notebook2script.py 01_matmul-Copy1.ipynb

in my 01_matmul-Copy1 notebook without using the auto-gen’d kernel. If I use my registered fastai kernel, I get:

Traceback (most recent call last):
  File "notebook2script.py", line 3, in <module>
    import json,fire,re
ModuleNotFoundError: No module named 'fire'

My guess is that the #!/usr/bin/env python line in run_notebook.py is pointing to my base Python (which doesn’t have fire), but when it’s run with the auto-gen’d kernel, something somewhere is ‘tricked’ into treating my fastai env as base Python, and thus: no errors.

NOTE: If I find a solution I’ll post here; the running solution is to use a registered kernel when running from the command line, and an nb_conda auto-gen’d kernel when running from a cell in a notebook.

Early command-line testing before I discovered the in-notebook failure case:

notebook : 00_exports-Copy1.ipynb

$ notebook2script.py 00_exports-Copy1.ipynb

$ python run_notebook.py 00_exports-Copy1.ipynb

env : Python [conda env:fastai]

  • run_notebook.py — FAIL
  • notebook2script.py — PASS

env : fastai

  • run_notebook.py — PASS
  • notebook2script.py — PASS

notebook : 01_matmul-Copy1.ipynb

env : fastai

  • run_notebook.py — PASS
  • notebook2script.py — PASS
2

I’m guessing you didn’t activated Jupyter virtualenv first

!python notebook2script.py 01_matmul-Copy1.ipynb

run system python interpreter which is not virtualenv interpreter
I had this problem on windows basically you need to activate virtual env and inside it your code
are you using Windows ?

3

I’m on macOS, using conda for envs.

4

Sorry don’t know to much about MacOS

image

when you start terminal do you see name in (brackets) can you try run
conda activate
conda deactivate and run your script to check is it Virtual ENV problem

5

@Borz @Michal_w

Trying to make this run locally via anaconda with fast ai installed.

First I activate my fastai env

Blockquoteconda acitivate fastai

Then I CD into my directory with fast ai code.
There I try to run

Blockquotepython run_notebook.py 00_exports.ipnyb

get the following error:

Traceback (most recent call last):
File “run_notebook.py”, line 12, in
if name == ‘main’: fire.Fire(run_notebook)
File “C:\Users\c17999\AppData\Local\Continuum\anaconda3\envs\fastai\lib\site-packages\fire\core.py”, line 138, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File “C:\Users\c17999\AppData\Local\Continuum\anaconda3\envs\fastai\lib\site-packages\fire\core.py”, line 471, in _Fire
target=component.name)
File “C:\Users\c17999\AppData\Local\Continuum\anaconda3\envs\fastai\lib\site-packages\fire\core.py”, line 675, in CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File “run_notebook.py”, line 8, in run_notebook
nb = nbformat.read(open(path), as_version=nbformat.current_nbformat, encoding=‘utf-8’)
File "C:\Users\c17999\AppData\Local\Continuum\anaconda3\envs\fastai\lib\site-packages\nbformat_init.py", line 141, in read
return reads(fp.read(), as_version, **kwargs)
File “C:\Users\c17999\AppData\Local\Continuum\anaconda3\envs\fastai\lib\encodings\cp1252.py”, line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: ‘charmap’ codec can’t decode byte 0x9d in position 18436: character maps to

Looked around couldnt find anyone providing resolutions or facing same issues