For a given machine learning problem, say Kaggle competition to predict X, we often have to experiment with a lot of data samplers, models, and hyperparameters. I’m working on AWS p2.xlarge and sometimes training takes longer than I care to sit in front of my workflow is usually:
- ssh tunnel to jupyter notebook on AWS
- experiment with small batch to see if training works
- run full training with a python script on a screen and save trained weights
- go back to jupyter notebook to see results
I’m sure there are more efficient ways of doing things so I’d like to hear from the forum. One thing I’d really love is to be able to “run the notebook on a screen” the same way as running a python script. I tried running the jupyter notebook server on a screen but once I close the window, everything seems to be wiped out.