HALP! Cannot save Jupyter Notebooks after running cells. Weird issue!

Hey everyone. Back here asking for help again :slight_smile:

I’m using Gradient notebooks and seeing a peculiar issue. After running a notebook that has some plotting elements, maybe a matplotlib graph, maybe the output of show_batch(), I am unable to save the notebook. I can keep executing other notebook cells and active terminals continue to operate fine but the Jupyter UI itself become non-responsive. Autosave/checkpointing also stops working.

I’ve been monitoring disk performance using iostat and cpu using htop, there isn’t any high resource utilisation or anything weird. For instance, running 02_production.ipynb from the fastbook repo works fine until the cell with dls.valid.show_batch(max_n=4, nrows=1). Up until this point, I can keep saving the notebook. After this runs, Ctrl+S does nothing. If I reload the Jupyter file explorer, it doesn’t respond. The only way to recover is to clear all cookies from my browser (or login from another browser) and reload Jupyter. Obviously, this also wipes out all unsaved changes from the notebooks.

I’m tearing my hair out over this. Been trying to troubleshoot this for over a month to no avail. Paperspace support has also been unable to help. You’re my last resort guys.

Edit: actually I’m not even sure what the trigger is. E.g. after simply running the first cell in this notebook, which is just imports, the whole thing gets stuck.

I can keep on running the other cells in the notebook, any terminals opened separately continue to work just fine. It’s only Jupyter specific stuff like navigating through directories, saving notebooks, etc. that get stuck. Issue seems to be in both JupyterLab and Classic Notebook interfaces. :frowning:

Hey folks. Bumping this thread again hoping it gets some attention. Thanks!

Sounds like the jupyter kernel is becoming unresponsive on the Paperspace server. What happens if you just try a simple matplotlib plot like

import matplotlib.pyplot as plt
import numpy as np
plt.plot(np.arange(10)

Does it show anything? If you keep running into this problem I’d switch to colab, or AWS, or something else. It’s not worth a month of your time, especially if Paperspace can’t help you with their own product.

I ran the code, doesn’t seem to trigger a problem. This issue is so sporadic and weird that I just can’t seem to figure out a solution.

The reason I stuck with Paperspace is because it has a free tier and I still have the Fast.ai credits. But the way things are progressing, their platform is actually preventing me from getting my work done :frowning:

So I tried the following on an EC2 instance:

git clone https://github.com/fastai/fastsetup.git
cd fastsetup
./setup-conda.sh
source ~/.bashrc
conda install -yq mamba
mamba install -y fastbook
cd ..
git clone https://github.com/fastai/fastbook

Post these steps, I’m still getting the same issue. Can’t save the notebook after I run through all the cells. A lot of UI stuff which was not working in Paperspace is working fine now; it’s just this one crucial thing that just doesn’t seem to get fixed!

When all else fails, turn on logging and see if you can find something broken to fix.

%logstart

Checked the logs as well; didn’t find anything of note.