Jupyter Notebook Enhancements [Discussion]

Part 1 (2018)
1

This thread is dedicated for discussing Jupyter Notebook Enhancements.

If you have an issue with one of the recipes posted here please discuss it here.

If you have a complete recipe to contribute please post it here.

Thank you.

2

These are pretty amazing set of tips. Thanks a lot for this. I will share mine but this worked on only one of my remote connections and I haven’t figured out why that happened.

So when you are doing a large computation using jupyter and there is loss of connection. There is no way to get the output of the cell back. This https://github.com/ipython/ipython/issues/4140 issue describes it in more detail. jeanpijon and lucasb-eyer solution are decent solutions for this.

Would be interested if someone found any different hack.

3 Likes
3

Thanks! works perfectly! Do you know if there is a way to freeze the view of a cell? Kind of like the Excel Freeze Pane function. It is useful for referencing some earlier cell without scrolling.

image

4

Btw the shortkey seems does not work in Firefox, but work fines in Chrome. Is there an easy way to activate the custom function instead of usingn shortkey to activate everytime? Thank you.

5

Yes, I switched to using chrome for jupyter long time ago, since some things like some short keys don’t seem to work in firefox.

6

For some reason, I cannot install Chrome on my laptop, it crashes everytime I start a browser.

7

if it’s a unix system, use gdb’s core dumping or strace to see where it fails and google the outcome?

usually moving the old config folder away solves this kind of problems:

mv ~/.config/google-chrome ~/.config/google-chrome-last

any luck with chromium?

1 Like
8

Thanks for your help, but I am in Window… no luck so far, have been searching for quite some time before I switch to Firefox

9

Awesome tips all.
One question, is it possible to update value of some variable, while the cell is running?
Use-case is that say, I have set a high value for number of epochs and so the model.fit() function is executing, but then I decide to end the training prematurely. If I interrupt the execution, the function won’t be returning so cannot use to plot graphs later. Thanks.

10

I have really taken to Jeremy’s approach of experimenting with notebooks, so I try different things, copy cells, save previous output to compare to the next run, etc. My nb’s often get messy and it gets hard to track which cell I just executed, etc. I’d like a way to clear the input prompt numbers without clearing outputs. I’ve searched the web and can’t find a way - I suspect because most folks don’t use the nb’s the way Jeremy and I do! Anybody know how to do this?

Clarification: I’d be happy to be able to clear one input prompt number at a time, but it would be nice to be able to clear all at once.

11

Never tried this package but it may be what you’re looking for:

12

This sounds great, but I guess I need clarification! I’ve been testing this by running the individual cell and it doesn’t work - does this only work if you’re running the entire nb?

13

Thanks, but I don’t want to strip or clear the outputs; I want to clear the input prompt numbers, meaning the 4 in this cell:

image

14

Hmm, have you tried this command from your command line

nbstripout --keep-output FILE.ipynb

See if that works.

15

…actually, I just tried on a test nb and it doesn’t work running the entire nb either - meaning, doing Restart & Run All. I’m on GCP; does that matter?

16

Is there a way to convert notebook to script back and forth? I would like to convert notebook to script as it is easier to refactor in IDE, but then is there an easy way to convert back to notebook.

17

I am looking for the answer of this question too…

As far as I remember, Pycharm can run Jupyter notebooks as it is without converting it to script. But then, unfortunately we cannot use pycharm to debug the Jupyter notebook, which make it not useful. This has been a 4 years old feature request.
But maybe this hack will work to debug jupyter notebooks with pycharm… I haven’t tried it yet.
Here is the blog post of the hack described:

See this fastai forum thread for a similar request.

Here is a way to debug a notebook with Pycharm, which I am not convinced to be a good solution.

This is a promising Visual Python Debugger for Jupyter Notebooks project, that did not work for me well. Not always the breakpoints break and seems still buggy.

1 Like
18

I searched a bit about your question.

I found Jupytext a promising project that syncs Jupyter notebooks as plain text documents or as .py files. (It is made by Marc Wouts, the same author of the debug jupyter notebooks with pycharm method I mentioned in my previous post)

JupyterPyCharm_75pct
JupyterPyCharm_75pct.gif1323×645 3.22 MB

Jupytext can save Jupyter notebooks as

  • Markdown and R Markdown documents,
  • Julia, Python, R, Bash, Scheme and C++ scripts.

Jupyter will save your notebook to your favorite format(s), from .ipynb to .py , .R , .jl , .md , .Rmd … The text representation can be edited outside of Jupyter. Simply refresh the notebook in Jupyter to get the latest input cells from the script or Markdown document. When refreshing, kernel variables are unaffected, and output cells are reloaded from the traditionnal .ipynb if present. You can also delete your .ipynb notebook entirely if you don’t need to save output cells.

Collaborating on Jupyter Notebooks

With Jupytext, collaborating on Jupyter notebooks with Git becomes as easy as collaborating on text files.

The setup is straightforward:

  • Open your favorite notebook in Jupyter notebook
  • Associate a .py representation (for instance) to that notebook
  • Save the notebook, and put the Python script under Git control. Sharing the .ipynb file is possible, but not required.

Collaborating then works as follows:

  • Your collaborator pulls your script. The script opens as a notebook in Jupyter, with no outputs.
  • They run the notebook and save it. Outputs are regenerated, and a local .ipynb file is created.
  • They change the notebook, and push their updated script. The diff is nothing else than a standard diff on a Python script.
  • You pull the changed script, and refresh your browser. Input cells are updated. The outputs from cells that were changed are removed. Your variables are untouched, so you have the option to run only the modified cells to get the new outputs.

@jeremy I remember you were working on a better version control of Jupyter notebooks, where we can exclude cell outputs. Maybe Jupytext is an ideal solution that can do that, and on top of that, it allows us to sync Jupter notebooks to our favourite IDE. Affectively we can use the good of both worlds. Nice debugging in IDE and nice inline outputs in Jupyter notebooks.

1 Like
19

Debugging notebooks with PyCharm is working perfectly for me. See the method at the end of this thread:
https://forums.fast.ai/t/pycharm-setup-help/12197/13

It does not solve the issue of importing a notebook, refactoring, and exporting, but tracing, breakpoints, and code navigation are all available.

1 Like
20

Thanks. I’ll give it a try.

I think both methods have their own advantages. Your method allows to debug without running the code again in pycharm.

While jupytext allows you to refactor and use the powerful version control of pycharm and any other tools of a proper IDE. But then running the code again in pycharm sometimes it is resource heavy. Save and load models can minimize that downside though.

I will check your method and report back.
I guess, you cannot put a breakpoint on a line that does not belong to fastai library?