Jupyter Notebook Enhancements [Discussion]

(Haider Alwasiti) #18

I searched a bit about your question.

I found Jupytext a promising project that syncs Jupyter notebooks as plain text documents or as .py files. (It is made by Marc Wouts, the same author of the debug jupyter notebooks with pycharm method I mentioned in my previous post)

Jupytext can save Jupyter notebooks as

  • Markdown and R Markdown documents,
  • Julia, Python, R, Bash, Scheme and C++ scripts.

Jupyter will save your notebook to your favorite format(s), from .ipynb to .py , .R , .jl , .md , .Rmd … The text representation can be edited outside of Jupyter. Simply refresh the notebook in Jupyter to get the latest input cells from the script or Markdown document. When refreshing, kernel variables are unaffected, and output cells are reloaded from the traditionnal .ipynb if present. You can also delete your .ipynb notebook entirely if you don’t need to save output cells.

Collaborating on Jupyter Notebooks

With Jupytext, collaborating on Jupyter notebooks with Git becomes as easy as collaborating on text files.

The setup is straightforward:

  • Open your favorite notebook in Jupyter notebook
  • Associate a .py representation (for instance) to that notebook
  • Save the notebook, and put the Python script under Git control. Sharing the .ipynb file is possible, but not required.

Collaborating then works as follows:

  • Your collaborator pulls your script. The script opens as a notebook in Jupyter, with no outputs.
  • They run the notebook and save it. Outputs are regenerated, and a local .ipynb file is created.
  • They change the notebook, and push their updated script. The diff is nothing else than a standard diff on a Python script.
  • You pull the changed script, and refresh your browser. Input cells are updated. The outputs from cells that were changed are removed. Your variables are untouched, so you have the option to run only the modified cells to get the new outputs.

@jeremy I remember you were working on a better version control of Jupyter notebooks, where we can exclude cell outputs. Maybe Jupytext is an ideal solution that can do that, and on top of that, it allows us to sync Jupter notebooks to our favourite IDE. Affectively we can use the good of both worlds. Nice debugging in IDE and nice inline outputs in Jupyter notebooks.

1 Like

Pycharm can not open .ipynb file
(Malcolm McLean) #19

Debugging notebooks with PyCharm is working perfectly for me. See the method at the end of this thread:

It does not solve the issue of importing a notebook, refactoring, and exporting, but tracing, breakpoints, and code navigation are all available.

1 Like

(Haider Alwasiti) #20

Thanks. I’ll give it a try.

I think both methods have their own advantages. Your method allows to debug without running the code again in pycharm.

While jupytext allows you to refactor and use the powerful version control of pycharm and any other tools of a proper IDE. But then running the code again in pycharm sometimes it is resource heavy. Save and load models can minimize that downside though.

I will check your method and report back.
I guess, you cannot put a breakpoint on a line that does not belong to fastai library?


(Malcolm McLean) #21

You can put a breakpoint in any file in the fastai conda environment. But it needs to be a line in a file. There might be some trick for breaking into Pycharm directly from the notebook, but I suspect you’d end up tracing the ipython interpreter. I’ll try it later.


(Haider Alwasiti) #22

I think that does make sense.
Maybe we can import a dummy python library just to make it break inside it. There shouldn’t be any difference than breaking a fastai library.


(Stas Bekman) #23

Please note, I gathered all the discussion about this topic and moved it to this newly created thread dedicated for discussions and problem resolution. The original thread is to compile complete working solutions and no discussions there to make it easy for people to actually find recipes w/o needing to scroll forever.


(Rick N) #24

In case anyone else wants this - I cross posted in Part 1 v3 and did get the answer. See https://forums.fast.ai/t/general-course-chat/24987/333?u=ricknta


(nok) #25

How does this part work? How to open script in note book and getting back all the cells layout ?


(Haider Alwasiti) #26

It is shown in the demo :slight_smile:
Demo time

Introducing Jupytext PyParis Binder

Looking for a demo?


(nok) #27

That could be a solution for my usecase, thank you! I will check this out soon.


(nok) #28

Hi, I have experiment with Jupytext a bit over the last week.

I think it suggests using .py as the source, while notebook are served as caching the output for more nicely formatted presentation.

It also comes with the ability to sync .ipynb and py where everytime you save the notebook, it updates the script (not the reverse way, you will need to run a command to do make changes in .py reflected in notebook).

So there is really 2 approaches to work:

  1. Use .py mainly, you can just open .py in notebook and use it like a notebook, save a .ipynb when you need to present your output.

  2. Use .ipynb mainly, keep a .py script always sync with the notebook. The .py keep code diff more nicely.

Which approach is better in your opinion?


(Haider Alwasiti) #29

It is up to your use case and taste. For me I don’t need the diff too much, and mostly want to debug notebooks in pycharm. So I prefer this approach. See this for an example.

One of the major issue for me using Jupytext is that you have run 2 scipts at the same time. Which is resource intesive when the scripts are DL stuff. So I switched to only debugging using pycharm.


(nok) #30

Looking for putting the “Go to current running cell” into an extension.
Also I would like to create a functionality where I am jump to the “recent” / last selected cell, so I can jump back and forth between two cell.

I am not sure if this is doable with extension or this need to be done within the jupyter library directly.

If someone experience in creating extension and famliar with Jupyter, could you give me some pointers where should I focus on. I am not too familiar with JS, but I think there is quite some code snippets floating around, my job is to gather them and put into a right place.

1 Like

(nok) #31

I like this function so much that I have build an jupyter extension. Unfortunately the PR process is rather slow. Here is my PR. The functionality was similar as what @stas built, I add a button and with an dedicate extension you can easily configure the shortcut with the nbextension configurator.


(Stas Bekman) #32

Great idea, @nok.

Unfortunately, it will not always work due to this bug in the notebook software:

So go to current running cell will take you to the first stuck cell instead forever.

I think more people need to chime in that bug report with words and not just +1s so that someone at jupyter dev will actually do something about it.

And I hope you also integrate the autoscroll with your extension too (except it has the same problem because of that bug).


(nok) #33

Interesting, I do experience some lag of * sometimes but have not observe it would stuck forever, will check out your notebook later.

Yes, I have also implemented the autoscroll function with a different default hotkey to activate the function( or you can just check the box in nbconfigurator)

1 Like

(Stas Bekman) #34

Interesting, I do experience some lag of * sometimes but have not observe it would stuck forever, will check out your notebook later.

I didn’t say forever, I said for the duration of the notebook run - i.e. as long as there is at least one cell running, those earlier cells report being busy. And since you’re going after the active cell, it will be the wrong cell - it’ll be the first stuck cell.

Do let me know if you can reproduce it on your side - thank you!

The only culprit with this approach is that the cells much be uncollapsed - if they are collapsed you can’t jump to the current cell.

And when your PRs are approved and the new extensions become available please post a little how-to here Jupyter Notebook Enhancements, Tips And Tricks

1 Like

Jupyter Notebook Enhancements, Tips And Tricks
(nok) #35

I am able to reproduce the bug. Although 1 out of many trials, the cell state seems behave correctly.


(Stas Bekman) #36

Thank you for doing that. Now you can see how it’d impact your extensions. It started happening specifically with python-3.6.0.


(nok) #37

I have no clue why it is associated with python version, I would expect it is from jupyter side. I have done some more testing, but let’s keep the discussion in the issue to draw some attention from jupyter dev hopefully. :

Some minor progress, I have found a weird way to produce a working environment, but root cause is unknown.
see https://github.com/jupyter/notebook/issues/4334#issuecomment-464433137