I am just curious as to what everyone does while their model is training? I usually have a bunch of things I want to try but I have to wait for the results of the current training to decide. Sometimes unexpected errors come up. But if the process is straightforward, I tend to find other interesting kernels or spend time on the forums. I might leave my laptop and get food as well. What about others? What do y’all do?
2 Likes
Browse the forum, see if there’s something I can answer, or play a YouTube video or movie if it’s long enough. If it’s so long (2+ hours), I’ll leave my laptop on and go do errands!
2 Likes
Sometimes I just watch the process bars go and cheer when the accuracy improves more than expected.
3 Likes
Fun question. In a way the harder question is what do I do when the model is NOT training? During those times, my GPU is getting obsolete, my brain is getting older, and I’m not on the verge of discovering something new and cool 
1 Like
Well I am usually living my life LOL! Typically the models I run train for like 8-9 hours so I am typically doing other stuff. If I am training on Google Colab I check my progress about once every hour. If I am using Kaggle Kernels I check if there’s any error and the kernel finished, otherwise I let it run.
1 Like
My preferred way is to rewrite my previous code so that it is more readable/ reuasable.
1 Like
woah, I’ve never trained a network that takes this long to train. I’ve done one or two hour things. How do you run a lot of tests on such a project? Like suppose you want to try different values for weight decay or tune something else? I’m guessing even a subset of the data would take too long.
Small subset, purposefully picking out similar classes. So for imagenet maybe 5 species of fish, 1 cat, bird,and three other separate classes. Also want to look at dataset distribution of classes and probably pick whichever is the most occurring.
1 Like
Play a ranked match in Overwatch and contemplate my life decisions 
2 Likes
I run it on a data sample (very small) so I will get most errors this way. Then I will possibly let in run when I’m sleeping or when it takes some time to train like 10+ minutes I play with my cats.
Currently I’m still doing part 1, so while model is learning I can watch lesson video.
1 Like