Training and lost notebook sessions

As you might have experienced (see here also for a description:
when training a large model that might take days to finish, if something happens to your jupyter connection and is lost, but the kernel is alive and the training is still running, there is no way to resume the notebook and see the process of your training (epochs, accuracy, etc.)
I was wondering if there is another way (through command line? or through another ui) that you can easily run and get your progress?

1 Like