Workflow when working on server via ssh: tools for debugging? plotting?


I run all my model training on a server with 4 GPUs. Ideally, I’d like to just be able to do everything on the terminal and make my workflow really streamlined / efficient. Of course, this requires knowing terminal/git/tmux/etc pretty well.

However, I find myself having to push the repo from the server to github, then pull it on my local machine and run it on my computer for two reasons:

  • plotting
  • debugging

Plotting: It seems like there really isn’t a way around it since plotting requires a GUI (unless there’s some interface that allows me to do the heavy lifting on the server, but render plots on my computer simultaneously).

Debugging: It seems like the usual python debuggers (e.g. pdb) have a lot of missing features that would be really nice for deep learning workflows in particular. I find using pdb to be a drag to debug code on the server, and instead find it much easier to just use jupyter notebooks.

Am I missing something? What is everyone’s workflow as far as training on servers but doing other stuff on computers with displays?

I use jupyter notebooks using port forwarding.
In my .ssh/config file I add
LocalForward 8889
for the proper host

I use port 8889 locally, because I still want to be able to use the default 8888 port for my local jupyter notebooks.

that way I can just access the jupyter notebooks remotely in my local Chrome, and do plotting as well.

What about using sshfs?

Use X11 forwarding if you want to do pyplots over SSH