Jupyter Notebook Enhancements, Tips And Tricks


(Stas Bekman) #1

This thread is dedicated to Jupyter Notebook enhancements and related goodies.

Please contribute your tips and improvements that make our lives easier.

Please only post complete recipes in this thread, so that it’s easy to benefit from.

For questions, problems and discussions please use this thread.

Thank you!


Jupyter Notebook Enhancements [Discussion]
Jupyter Notebook Enhancements [Discussion]
Fast.ai v3 2019课程中文版笔记
Jupyter Notebook Enhancements [Discussion]
Jupyter Notebook Enhancements [Discussion]
(Stas Bekman) #2

I will start:

Go To The Current Running Cell Keyboard Shortcut

I often find myself scrolling through the running notebook, trying to find the currently running cell.

I wrote a keyboard shortcut that takes me there directly using Alt-I.

If you’d like to the same functionality add the following code to ~/.jupyter/custom/custom.js (you may need to create the folder and the file if you don’t have them already):

// Go to Running cell shortcut
Jupyter.keyboard_manager.command_shortcuts.add_shortcut('Alt-I', {
    help : 'Go to Running cell',
    help_index : 'zz',
    handler : function (event) {
        setTimeout(function() {
            // Find running cell and click the first one
            if ($('.running').length > 0) {
                //alert("found running cell");
                $('.running')[0].scrollIntoView();
            }}, 250);
        return false;
    }
});

You can experiment with this code also by creating a code cell in your notebook and adding:

%%javascript

on the first line, and then pasting the above code on the following lines, and running the cell. it will affect only the current notebook.

I got the idea from https://stackoverflow.com/questions/44273643/how-can-i-jump-to-the-cell-currently-being-run-in-a-jupyter-notebook

wrt implementation I’m not sure if Alt-I is the best choice - suggestions are welcome.

Next, it’d be nice to have a special mode where the notebook automatically re-focuses the view on the currently running cell, so one could easily follow Run All Cells hands off. I don’t know yet enough about jupyter innards to know how to code that one, so suggestions are welcome.

Pretty Print All Cell’s Outputs (and not just the last output of the cell)

Normally only the last output in the cell gets pretty printed - the rest you have to manually add print() which is not very convenient. So here is how to change that:

At the top of the notebook add:

from IPython.core.interactiveshell import InteractiveShell

# pretty print all cell's output and not just the last one
InteractiveShell.ast_node_interactivity = "all"

Examples:

Normal behavior: only one output is printed:

In  [1]: 4+5
         5+6

Out [1]: 11

After the new setting is activated both outputs get printed:

In  [1]: 4+5
         5+6

Out [1]: 9
Out [1]: 11

To restore the original behavior add:

from IPython.core.interactiveshell import InteractiveShell

# pretty print only the last output of the cell
InteractiveShell.ast_node_interactivity = "last_expr"

note: you have to run the setting change in a separate cell for it to take effect for the next cell run.

To make this behavior consistent across all notebooks edit: ~/.ipython/profile_default/ipython_config.py:

c = get_config()

# Run all nodes interactively
c.InteractiveShell.ast_node_interactivity = "all"

The tip was found here: https://stackoverflow.com/a/36835741/9201239

Suppressing Output

The side-effect of this change is that some libraries will start spewing a lot of noise (e.g. matplotlib). To fix that add ; at the end of the lines whose output you don’t want to be displayed.

Example:

fig, axes = plt.subplots(2, 1, figsize=(20, 20))
axes[0].set_ylim(-5, 5);

Without using ; it outputs:

Out[53]: (-5, 5)

However the hack doesn’t seem to work when the code line that outputs something is part of some loop, in which case the workaround of assigning to _ does the trick:

fig, axes = plt.subplots(2, 1, figsize=(20, 20))
for i in [1,2]:
    _ = axes[i].set_ylim(-5, 5)

Developer chat
Jupyter Notebook Enhancements [Discussion]
(Stas Bekman) #3

Found a collection of tips here:


(Stas Bekman) #4

To Skip A Cell From Running (e.g. work in progress)

Add at the top of the cell:

%%script false

some multiline code 
that you want to skip for a time being 
(e.g. work in progress) 
without commenting out / deleting cell 
goes here

(Stas Bekman) #5

To Include Markdown in Your Code’s Output (Colors, Bold, etc.)

Just using print() often makes it difficult to have certain outputs standout in the sea of outputs.

https://stackoverflow.com/a/46934204/9201239 suggests a way to fix that and be able to include markdown in your output - here is a reduced version on the original post:

from IPython.display import Markdown, display
def printmd(string):
    display(Markdown(string))

printmd("**bold text**")

You can add color using html:

def printmd(string, color=None):
    colorstr = "<span style='color:{}'>{}</span>".format(color, string)
    display(Markdown(colorstr))

printmd("**bold and blue**", color="blue")

I currently started using this for printing scores - it stands out nicely from the rest of the noise.


(Stas Bekman) split this topic #6

21 posts were merged into an existing topic: Jupyter Notebook Enhancements [Discussion]


(Stas Bekman) #7

Restart and Run All fix

My browser is too slow and “Restart and Run All” only manages to restart, but fails to “Run All”. This adds the missing delay and as a bonus a shortcut Meta-F (to complement Alt-F=Run All ignoring errors). Runs from ‘Command Mode’ (i.e. may need to hit Esc first).

// Meta-F: "Restart and Run All" slow delay fix + shortcut [Command mode]
Jupyter.keyboard_manager.command_shortcuts.add_shortcut('Meta-F', {
    help: 'Restart And Run All',
    help_index: 'zz',
    handler: function (event) {
      Jupyter.notebook.kernel.restart();
      restartTime = 2000 // decrease this if you have a fast computer
      setTimeout(function(){ Jupyter.notebook.execute_all_cells(); }, restartTime);
      return false;
    }
});

see earlier posts for where to add this code.


(Stas Bekman) #8

Control notebook resource allocation

Sometimes notebooks take a way more resources than available, causing undesired behavior, like a hanging system.

There are various ways to limit system resource usage, I decided to try the modern cgroups to limit memory usage in this case.

sudo cgcreate -a stas:stas -t stas:stas -g memory:JupyterGroup

replace stas with your setup’s user:group (usually yourusername:yourusername).

Then I decided to give each notebook max of 15GB RAM and 25GB Total (RAM+SWAP):

sudo echo $(( 15000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/JupyterGroup/memory.limit_in_bytes
sudo echo $(( 10000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/JupyterGroup/memory.kmem.max_usage_in_bytes

Now I start the notebook as:

cgexec -g memory:JupyterGroup jupyter notebook

a greedy kernel now gets terminated if it consumes more than what I allocated, without crashing/overloading my system.

Note that filenames under /sys/fs/cgroup/memory/JupyterGroup/ might be slightly different on your system. You may have to check the documentation.

I also needed to install the following packages, before I was able to run this:

sudo apt-get install cgroup-bin cgroup-lite cgroup-tools cgroupfs-mount libcgroup1


(Stas Bekman) #9

Following the currently executing cell mode

Won’t it be nice to be able to watch the progress of the notebook run hands off? Now you can:

Add the following in ~/.jupyter/custom/custom.js and reload the notebooks you’re running:

/*
 In Command mode Meta-[ toggles Follow Exec Cell mode, Meta-] turns it off.

 To adjust the behavior you can adjust the arguments:
 * behavior: One of "auto", "instant", or "smooth". Defaults to "auto". Defines the transition animation.
 * block:    One of "start", "center", "end", or "nearest". Defaults to "center".
 * inline:   One of "start", "center", "end", or "nearest". Defaults to "nearest".
 https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollIntoView
*/
function scrollIntoRunningCell(evt, data) {
    $('.running')[0].scrollIntoView({behavior: 'smooth', inline: 'center'});
}

Jupyter.keyboard_manager.command_shortcuts.add_shortcut('Meta-[', {
    help: 'Follow Executing Cell On',
    help_index: 'zz',
    handler: function (event) {
        Jupyter.notebook.events.on('finished_execute.CodeCell', scrollIntoRunningCell);
        //console.log("Follow Executing Cell On")
        return false;
    }
});

Jupyter.keyboard_manager.command_shortcuts.add_shortcut('Meta-]', {
    help: 'Follow Executing Cell Off',
    help_index: 'zz',
    handler: function (event) {
        Jupyter.notebook.events.off('finished_execute.CodeCell', scrollIntoRunningCell);
        //console.log("Follow Executing Cell Off")
        return false;
    }
});

Now in Command Mode (when cell in focus has a blue box around it and not green, or hit Esc to toggle mode), hit Meta-[ to get the currently run cell stay in the middle of the screen, hit Meta-] to return to normal behavior.

If this is not working, debug this setup by uncommenting console.log() calls and watch your browser Developer Tools’ Console to check that custom.js got loaded without errors and that the shortcuts got registered and the handler is activated. Sometimes you need to restart jupyter notebook, but most of the time tab-reload works.

If you just want to jump once to the current executing cell use Alt-I after you add the following to ~/.jupyter/custom/custom.js and reload the notebooks you’re running:

// Alt-I: Go to Running cell shortcut [Command mode]
Jupyter.keyboard_manager.command_shortcuts.add_shortcut('Alt-I', {
    help : 'Go to Running cell',
    help_index : 'zz',
    handler : function (event) {
        setTimeout(function() {
            // Find running cell and click the first one
            if ($('.running').length > 0) {
                //alert("found running cell");
                $('.running')[0].scrollIntoView();
            }}, 250);
        return false;
    }
});

Caveat: for it to work - the sections should all be uncollapsed - otherwise it won’t know to go into a collapsed section.

You can adjust the activation shortcut keys to your liking.

Remember that all 3 shortcuts will only work in the Command mode (see above for figuring that out).

This has been tested to work with jupyter notebook 5.6.0 with python 3.6.6.


Jupyter Notebook Enhancements [Discussion]
Developer chat
(Stas Bekman) #16

nbdime: Selective Diff/Merge Tool for jupyter notebooks

I also need to share very useful tips about nbdime which is very useful for working with jupyter notebooks.

Install it first:

pip install -U nbdime

it should automatically configure it for jupyter notebook. If something doesn’t work, see installation.

Then put the following into ~/.jupyter/nbdime_config.json:

{

  "Extension": {
    "source": true,
    "details": false,
    "outputs": false,
    "metadata": false
  },

  "NbDiff": {
    "source": true,
    "details": false,
    "outputs": false,
    "metadata": false
  },

  "NbDiffDriver": {
    "source": true,
    "details": false,
    "outputs": false,
    "metadata": false
  },

  "NbMergeDriver": {
    "source": true,
    "details": false,
    "outputs": false,
    "metadata": false
  },

  "dummy": {}
}

Change outputs value to true if you care to see outputs diffs too.

Now when you do:

git diff

Instead of getting a very noisy and hard to parse normal diff:

--- a/examples/tabular.ipynb
+++ b/examples/tabular.ipynb
@@ -2,12 +2,12 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from fastai import *          # Quick access to most common functionality\n",
-    "from fastai.tabular import *  # Quick access to tabular functionality\n",
+    "from fastai import *          # Very Quick access to most common functionality\n",
+    "from fastai.tabular import *  # Very Quick access to tabular functionality\n",
     "from fastai.docs import *     # Access to example data provided with fastai"
    ]
   },

You will get this sweetness:

snap5

The second feature I like is that now your notebook has a button: [nbdiff] along all the tools and it’ll show you the diff right in your browser!

snap6

And did I say that it does notebook merging too! Whoah! Use the setting above and it’ll ignore any noise when merging (i.e. metadata, execution_count, etc.), and only merge code cells! Of course you can adjust the configuration to suite your needs.

For the full docs see its website.


(nirant) #17

Hey @stas, thanks for this. This is amazing!

I took what I learned here, combined with what I learnt through fast ai and daily use of Jupyter notebook into this one place: https://github.com/NirantK/best-of-jupyter

Hope you find this useful!


(nirant) #18

For those coming here for the first time, here are some direct links to what you do better in your Jupyter usage:

Contents


(Stas Bekman) #19

Tell the notebook to save itself now

juputer notebook by default autosaves itself every 5 min or so if you haven’t changed the defaults.

But if you want to make sure the notebook is saved at the end of the run, you can just insert a new cell at the end of your notebook and make sure you run it:

%%javascript
IPython.notebook.save_notebook()

now your notebook will be always saved as soon as it’s done running.

This is useful if you’re then immediately needing to commit the change to git.

credit: the idea came from here.


(Haider Alwasiti) #21

Python (not Jupyter) trick, but still useful for our long running Jupyter Notebooks:

When the code takes extremely long to run and I don’t want to be staring at it all the time but want to know when it is done.

How can I make the (Python) code sort of sound an “alarm” when it is done?

The following answer works only on server side. So it is useful only when you have a local server:

On Windows

import winsound
duration = 1000  # millisecond
freq = 440  # Hz
winsound.Beep(freq, duration)

Where freq is the frequency in Hz and the duration is in milliseconds.

On Linux (and Mac)

import os
duration = 1  # second
freq = 440  # Hz
os.system('play --no-show-progress --null --channels 1 synth %s sine %f' % (duration, freq))

In order to use this example, you must install sox .

On Debian/Ubuntu/LinuxMint you need to run in your terminal:

sudo apt install sox

Here is the macports way of doing that…run this is your terminal:

sudo port install sox

Speech on Mac

And something really cool if you’re using a mac in terminal, maybe can do the same in windows, but I only know for sure for mac, this will tell you it’s done:

import os
os.system('say "your program has finished"')

Speech on Linux

import os
os.system('spd-say "your program has finished"')

You need to install the speech-dispatcher package in Ubuntu (or the corresponding package on other distributions):

sudo apt install speech-dispatcher

Source:
https://stackoverflow.com/questions/16573051/sound-alarm-when-code-finishes/16573339#16573339


(Stas Bekman) #22

Make Esc always switch to Command mode

update: this problem should be fixed in the recent version of navigation-hotkeys.

If you use a very handy jupyter_contrib_nbextensions/nbextensions/navigation-hotkeys it introduces a very bad idea of changing the behavior of Esc to always switch to the opposite mode, making Esc unreliable as a precursor for a command mode shortcut call. Which leads to a lot of unpredictable behavior. So the following takes the goodness of the navigation-hotkeys extension and avoids the undesirable.

  1. disable navigation-hotkeys
  2. add to ~/.jupyter/custom/custom.js the following:
/* start a custom version of jupyter_contrib_nbextensions/nbextensions/navigation-hotkeys to remove Esc=> Edit mode toggle as it interferes with the workflow - Esc needs to be deterministic - switches to Command mode */

var add_command_shortcuts = {
        'home' : {
            help    : 'go to top',
            help_index : 'ga',
            handler : function() {
                IPython.notebook.select(0);
                IPython.notebook.scroll_to_top();
                return false;
            }
        },

        'end' : {
            help    : 'go to bottom',
            help_index : 'ga',
            handler : function() {
                var ncells = IPython.notebook.ncells();
                IPython.notebook.select(ncells-1);
                IPython.notebook.scroll_to_bottom();
                return false;
            }
        },

        'pageup' : {
            help    : 'page up',
            help_index : 'aa',
            handler : function() {
            var wh = 0.6 * $(window).height();
            var cell = IPython.notebook.get_selected_cell();
            var h = 0;
            /* loop until we have enough cells to span the size of the notebook window (= one page) */
            do {
                h += cell.element.height();
                IPython.notebook.select_prev();
                cell = IPython.notebook.get_selected_cell();
            } while ( h < wh );
            var cp = cell.element.position();
            var sp = $('body').scrollTop();
            if ( cp.top < sp) {
                IPython.notebook.scroll_to_cell(IPython.notebook.get_selected_index(), 0);
            }
            cell.focus_cell();
            return false;
            }
        },

        'pagedown' : {
            help    : 'page down',
            help_index : 'aa',
            handler : function() {

            /* jump to bottom if we are already in the last cell */
            var ncells = IPython.notebook.ncells();
            if ( IPython.notebook.get_selected_index()+1 == ncells) {
                IPython.notebook.scroll_to_bottom();
                return false;
            }

            var wh = 0.6*$(window).height();
            var cell = IPython.notebook.get_selected_cell();
            var h = 0;

            /* loop until we have enough cells to span the size of the notebook window (= one page) */
            do {
                h += cell.element.height();
                IPython.notebook.select_next();
                cell = IPython.notebook.get_selected_cell();
            } while ( h < wh );
            cell.focus_cell();
            return false;
            }
        }

    };

var add_edit_shortcuts = {
        'alt-add' : {
            help    : 'split cell',
            help_index : 'eb',
            handler : function() {
                IPython.notebook.split_cell();
                IPython.notebook.edit_mode();
                return false;
            }
        },
        'alt-subtract' : {
            help    : 'merge cell',
            help_index : 'eb',
            handler : function() {
                var i = IPython.notebook.get_selected_index();
                if (i > 0) {
                    var l = IPython.notebook.get_cell(i-1).code_mirror.lineCount();
                    IPython.notebook.merge_cell_above();
                    IPython.notebook.get_selected_cell().code_mirror.setCursor(l,0);
                    }
            }
        },
        'shift-enter' : {
            help    : 'run cell, select next codecell',
            help_index : 'bb',
            handler : function() {
                IPython.notebook.execute_cell_and_select_below();
                var rendered = IPython.notebook.get_selected_cell().rendered;
                var ccell = IPython.notebook.get_selected_cell().cell_type;
                if (rendered === false || ccell === 'code') IPython.notebook.edit_mode();
                return false;
            }
        },
        'ctrl-enter' : {
            help    : 'run cell',
            help_index : 'bb',
            handler : function() {
                var cell = IPython.notebook.get_selected_cell();
                var mode = cell.mode;
                cell.execute();
                if (mode === "edit") IPython.notebook.edit_mode();
                return false;
            }
        },
        'alt-n' : {
            help    : 'toggle line numbers',
            help_index : 'xy',
            handler : function() {
                var cell = IPython.notebook.get_selected_cell();
                cell.toggle_line_numbers();
                return false;
            }
        },
        'pagedown' : {
            help    : 'page down',
            help_index : 'xy',
            handler : function() {

                var ic = IPython.notebook.get_selected_index();
                var cells = IPython.notebook.get_cells();
                var i, h=0;
                for (i=0; i < ic; i ++) {
                    h += cells[i].element.height();
                    }
                var cur = cells[ic].code_mirror.getCursor();
                h += cells[ic].code_mirror.defaultTextHeight() * cur.line;
                IPython.notebook.element.animate({scrollTop:h}, 0);
                return false;
            }
        },
        'pageup' : {
            help    : 'page down',
            help_index : 'xy',
            handler : function() {

                var ic = IPython.notebook.get_selected_index();
                var cells = IPython.notebook.get_cells();
                var i, h=0;
                for (i=0; i < ic; i ++) {
                    h += cells[i].element.height();
                    }
                var cur =cells[ic].code_mirror.getCursor();
                h += cells[ic].code_mirror.defaultTextHeight() * cur.line;
                IPython.notebook.element.animate({scrollTop:h}, 0);
                return false;
            }
        },
        'ctrl-y' : {
            help : 'toggle markdown/code',
            handler : function() {
                var cell = IPython.notebook.get_selected_cell();
                var cur = cell.code_mirror.getCursor();
                if (cell.cell_type == 'code') {
                    IPython.notebook.command_mode();
                    IPython.notebook.to_markdown();
                    IPython.notebook.edit_mode();
                    cell = IPython.notebook.get_selected_cell();
                    cell.code_mirror.setCursor(cur);
                } else if (cell.cell_type == 'markdown') {
                    IPython.notebook.command_mode();
                    IPython.notebook.to_code();
                    IPython.notebook.edit_mode();
                    cell = IPython.notebook.get_selected_cell();
                    cell.code_mirror.setCursor(cur);
                }
                return false;
            }
        }
    };

Jupyter.keyboard_manager.edit_shortcuts.add_shortcuts(add_edit_shortcuts);
Jupyter.keyboard_manager.command_shortcuts.add_shortcuts(add_command_shortcuts);

/* end a custom version of jupyter_contrib_nbextensions/nbextensions/navigation-hotkeys */

(Haider Alwasiti) #23

Restoring Jupyter client’s GUI (scores and graphs…etc) after client disconnects for long time

Requirements:

  1. we should use tmux to keep the jupyter server running, even after ssh disconnects. Without tmux, jupyter server will be killed after ssh connection disconnects.
  2. The client (e.g., your pc/laptop showing the jupyter notebook) may go to hibernation or sleep without problem, but the tab of the browser showing the jupyter clients should not be closed. Only that tab that began running the notebook will be able to restore the whole running GUI elements of the notebook.
  3. Increase 2 parameters in the jupyter config file:

nano /home/<username>/.jupyter/jupyter_notebook_config.py

if config file is not available, we should generate it by:

jupyter notebook --generate-config

ctrl+w: search for c.NotebookApp.iopub_data_rate_limit

Uncomment and change these values into something very large:

c.NotebookApp.iopub_msg_rate_limit = 100000000

c.NotebookApp.iopub_data_rate_limit = 10000000000


(Stas Bekman) #24

Automatic Notebook Halt / Restart cell

When included at the bottom of a notebook, it will stop the notebook at the end of Run All, and pop up a modal dialog letting the user know that the notebook was auto-stopped at the end of the run. The dialog offers a “Kernel Restart” button which restarts the notebook.

%%javascript
require(
    ["base/js/dialog"], 
    function(dialog) {
        dialog.modal({

                title: 'Notebook Halted',
                body: 'This notebook is no longer running; the kernel has been halted. Close the browser tab, or, to continue working, restart the kernel.',
                buttons: {
                    'Kernel restart': { click: function(){ Jupyter.notebook.session.restart(); } }
                }
        });
    }
);
Jupyter.notebook.session.delete();

source.


(Haider Alwasiti) #25

Use plotly instead of matplotlib to get better-looking and interactive chart

For those used to matplotlib , all we have to do is add one more letter ( iplot instead of plot ) and we get a much better-looking and interactive chart! We can click on the data to get more details, zoom into sections of the plot, and as we’ll see later, select different categories to highlight.

  1. One-line charts for rapid exploration
  2. Interactive elements for subsetting/investigating data
  3. Option to dig into details as needed
  4. Easy customization for final presentation

That’s just as simple:

df['claps'].iplot(kind='hist', xTitle='claps', yTitle='count', title='Claps Distribution')

Try it now with nbviewer live demo:

The plotly Python package is an open-source library built on plotly.js which in turn is built on d3.js . This code is using a wrapper on plotly called cufflinks designed to work with Pandas dataframes. So, our entire stack is cufflinks > plotly > plotly.js > d3.js which means we get the efficiency of coding in Python with the incredible interactive graphics capabilities of d3.

source:

P.S.: For jupyter Lab you should install also:
jupyter labextension install @jupyterlab/plotly-extension


(Shital Shah) #26
  • Shift + Tab cycles through help info like docstring
  • Ctrl + Shift + - splits the current cell from cursor
  • Ctrl + / comments out selected lines in the cell

(深度碎片) #27

How to add a keyboard shortcut to insert a code snippet

Inspired by https://stackoverflow.com/a/51719689/9201239
and some help from @stas

If you first enter a cell with either code mode or markdown mode, like the following

Then you press Ctrl + Shift + M, you will get the following

To make this happening, you just need to

  1. go to custom.js whose path can be found by running
echo $(jupyter --config-dir)/custom/custom.js
  1. add the following codes into custom.js and save.
Jupyter.keyboard_manager.edit_shortcuts.add_shortcut('Ctrl-Shift-M', {
                              help : 'add details drop',
                              help_index : 'zz',
                              handler : function (event) {
                              var target = Jupyter.notebook.get_selected_cell()
                              var cursor = target.code_mirror.getCursor()
                              var before = target.get_pre_cursor()
                              var after = target.get_post_cursor()
                              target.set_text(before + '[/details][details=""]' + after)
                              cursor.ch += 20 // where to put your cursor
                              target.code_mirror.setCursor(cursor)
                              return false;
                              }}
                                                        );

  1. finally, go back to Jupyter Notebook and refresh your page, and you are ready to go.

Jupyter Notebook Enhancements [Discussion]
Fast.ai v3 2019课程中文版笔记