Debugging notebooks with PyCharm is working perfectly for me. See the method at the end of this thread:
https://forums.fast.ai/t/pycharm-setup-help/12197/13
It does not solve the issue of importing a notebook, refactoring, and exporting, but tracing, breakpoints, and code navigation are all available.
Thanks. I’ll give it a try.
I think both methods have their own advantages. Your method allows to debug without running the code again in pycharm.
While jupytext allows you to refactor and use the powerful version control of pycharm and any other tools of a proper IDE. But then running the code again in pycharm sometimes it is resource heavy. Save and load models can minimize that downside though.
I will check your method and report back.
I guess, you cannot put a breakpoint on a line that does not belong to fastai library?
You can put a breakpoint in any file in the fastai conda environment. But it needs to be a line in a file. There might be some trick for breaking into Pycharm directly from the notebook, but I suspect you’d end up tracing the ipython interpreter. I’ll try it later.
I think that does make sense.
Maybe we can import a dummy python library just to make it break inside it. There shouldn’t be any difference than breaking a fastai library.
Please note, I gathered all the discussion about this topic and moved it to this newly created thread dedicated for discussions and problem resolution. The original thread is to compile complete working solutions and no discussions there to make it easy for people to actually find recipes w/o needing to scroll forever.
In case anyone else wants this - I cross posted in Part 1 v3 and did get the answer. See https://forums.fast.ai/t/general-course-chat/24987/333?u=ricknta
How does this part work? How to open script in note book and getting back all the cells layout ?
It is shown in the demo 
Demo time
Looking for a demo?
That could be a solution for my usecase, thank you! I will check this out soon.
Hi, I have experiment with Jupytext a bit over the last week.
I think it suggests using .py as the source, while notebook are served as caching the output for more nicely formatted presentation.
It also comes with the ability to sync .ipynb and py where everytime you save the notebook, it updates the script (not the reverse way, you will need to run a command to do make changes in .py reflected in notebook).
So there is really 2 approaches to work:
Use .py mainly, you can just open .py in notebook and use it like a notebook, save a .ipynb when you need to present your output.
Use .ipynb mainly, keep a .py script always sync with the notebook. The .py keep code diff more nicely.
Which approach is better in your opinion?
It is up to your use case and taste. For me I don’t need the diff too much, and mostly want to debug notebooks in pycharm. So I prefer this approach. See this for an example.
One of the major issue for me using Jupytext is that you have run 2 scipts at the same time. Which is resource intesive when the scripts are DL stuff. So I switched to only debugging using pycharm.
Looking for putting the “Go to current running cell” into an extension.
Also I would like to create a functionality where I am jump to the “recent” / last selected cell, so I can jump back and forth between two cell.
I am not sure if this is doable with extension or this need to be done within the jupyter library directly.
If someone experience in creating extension and famliar with Jupyter, could you give me some pointers where should I focus on. I am not too familiar with JS, but I think there is quite some code snippets floating around, my job is to gather them and put into a right place.
I like this function so much that I have build an jupyter extension. Unfortunately the PR process is rather slow. Here is my PR. The functionality was similar as what @stas built, I add a button and with an dedicate extension you can easily configure the shortcut with the nbextension configurator.
Great idea, @nok.
Unfortunately, it will not always work due to this bug in the notebook software:
So go to current running cell will take you to the first stuck cell instead forever.
I think more people need to chime in that bug report with words and not just +1s so that someone at jupyter dev will actually do something about it.
And I hope you also integrate the autoscroll with your extension too (except it has the same problem because of that bug).
Interesting, I do experience some lag of * sometimes but have not observe it would stuck forever, will check out your notebook later.
Yes, I have also implemented the autoscroll function with a different default hotkey to activate the function( or you can just check the box in nbconfigurator)
Interesting, I do experience some lag of * sometimes but have not observe it would stuck forever, will check out your notebook later.
I didn’t say forever, I said for the duration of the notebook run - i.e. as long as there is at least one cell running, those earlier cells report being busy. And since you’re going after the active cell, it will be the wrong cell - it’ll be the first stuck cell.
Do let me know if you can reproduce it on your side - thank you!
The only culprit with this approach is that the cells much be uncollapsed - if they are collapsed you can’t jump to the current cell.
And when your PRs are approved and the new extensions become available please post a little how-to here Jupyter Notebook Enhancements, Tips And Tricks
I am able to reproduce the bug. Although 1 out of many trials, the cell state seems behave correctly.
Thank you for doing that. Now you can see how it’d impact your extensions. It started happening specifically with python-3.6.0.
I have no clue why it is associated with python version, I would expect it is from jupyter side. I have done some more testing, but let’s keep the discussion in the issue to draw some attention from jupyter dev hopefully. :
Some minor progress, I have found a weird way to produce a working environment, but root cause is unknown.
see https://github.com/jupyter/notebook/issues/4334#issuecomment-464433137
Thank you for helping me to find out what broke it. I’m going to try to reproduce your progress, @nok. And yes, let’s continue it in the github thread so perhaps we get some attention from the devs.