Hello, I just started using gradient to do the 1st chapter of the course. I’ve noticed that saving a notebook doesn’t persist the outputs and cell runs once the notebook has been closed. Is anyone else seeing this? Is it expected?
I am also using Gradient. I guess it’s expected as Server (Machine) will be stopped after sometime and in that case outputs will not be persisted in the notebook. You need to run the cells every time you starts the machine/server of Gradient. But you can save the notebook to save all the codes and export the model.
Turns out that I had a pretty fundamental misunderstanding of how Jupyter notebooks work, since I haven’t used them in quite awhile. You cannot persist python variables in a notebook, unless they’re pickleable and you have to do that explicitly.
I’m note sure why Jupyter notebooks don’t automatically persist all pickle-able objects by default and if there’s a config/setting that can be tweaked to make them do this.
I don’t believe this is possible except doing it explicitly yourself like you said. If you were running the Jupyter notebooks on a dedicated server then the state is preserved in memory until you close the notebook, restart the notebook kernel or reboot the server/container. It is definitely handy to be able to maintain state between working sessions, but not a necessity.
As far as the outputs go, this is something that even a dedicated server may not handle exactly how you would expect it to. While your browser is disconnected from the server, the server will buffer some notebook state (ex: training loop status bars/stats) but the buffer is limited and if too much data is added to the buffer before your browser reconnects to the server, some of the data that returns to your notebook in the browser will be lost. The way I work around this is whenever I am training a model where I expect the buffer might be overrun if I disconnect from it, I have the browser run on a machine that will stay online and connected to the server.