Is it possible for 2 engineers editing the same ipynb file at the same time ?
For example, engineer 1 edit the first half, and engineer 2 edit the second half.
During half way of training, could it be saved and later continue ?
Suppose I am editing http://server:8888/lesson1.ipynb, the jupyter is running on team’s sharing server, I am accessing from my mac book in office.
My training(fit) is still running, and suppose already 50% in progress, while I need to leave office.
Once I got home, connect my mac book to WIFI, could I still continue the on-going training ? looks like if I refresh that browser tab, I need to re-train again. What’s the best practice here ?
If you were editing at the same time how would you resolve conflicts? If you really want to do this I suggest using jupytext which keeps a parallel .py file. Then you can resolve conflicts using git merge on the .py file.
If you want to pause training then you can set checkpoints e.g. save every epoch. This will allow you to restart.
like google doc, collabedit ?
set checkpoints e.g. save every epoch --> this means save in code like
jupyter has more clever support for this requirement ?
not jupyter itself but fastai has a callback at the end of every epoch so you could save the weights for the last epoch completed or for each epoch if you want.