I run all my model training on a server with 4 GPUs. Ideally, I’d like to just be able to do everything on the terminal and make my workflow really streamlined / efficient. Of course, this requires knowing terminal/git/tmux/etc pretty well.
However, I find myself having to push the repo from the server to github, then pull it on my local machine and run it on my computer for two reasons:
Plotting: It seems like there really isn’t a way around it since plotting requires a GUI (unless there’s some interface that allows me to do the heavy lifting on the server, but render plots on my computer simultaneously).
Debugging: It seems like the usual python debuggers (e.g. pdb) have a lot of missing features that would be really nice for deep learning workflows in particular. I find using pdb to be a drag to debug code on the server, and instead find it much easier to just use jupyter notebooks.
Am I missing something? What is everyone’s workflow as far as training on servers but doing other stuff on computers with displays?