How do you manage your time when model is training?

Whenever I am experimenting with a deep learning model, I find it difficult to get into the flow state and maintain it, due to the unavoidable interruptions caused by model training time.

If the training time is around 10-20 mins, as is the case when fine-tuning dream booth, I use the training time to do some reading or other tangentially related study. Unfortunately, whenever the training finishes I find that the context switch decreases my ability to think deeply about the model I am training.

I remember Jeremy made a passing remark in one of the lectures that learning to train models overnight is an important skill. I have found this to be the case and sometimes I’d write a bash script defining multiple training strategies and then I’d run it overnight.

I am curious about what strategies others employ for this problem.

1 Like
5 Likes

I almost marked it as the solution but this is more of a visual depiction of the problem :laughing: