Whenever I am experimenting with a deep learning model, I find it difficult to get into the flow state and maintain it, due to the unavoidable interruptions caused by model training time.
If the training time is around 10-20 mins, as is the case when fine-tuning dream booth, I use the training time to do some reading or other tangentially related study. Unfortunately, whenever the training finishes I find that the context switch decreases my ability to think deeply about the model I am training.
I remember Jeremy made a passing remark in one of the lectures that learning to train models overnight is an important skill. I have found this to be the case and sometimes I’d write a bash script defining multiple training strategies and then I’d run it overnight.
I am curious about what strategies others employ for this problem.