Workflow when working on server via ssh: tools for debugging? plotting?

Deep Learning
1

Hi!

I run all my model training on a server with 4 GPUs. Ideally, I’d like to just be able to do everything on the terminal and make my workflow really streamlined / efficient. Of course, this requires knowing terminal/git/tmux/etc pretty well.

However, I find myself having to push the repo from the server to github, then pull it on my local machine and run it on my computer for two reasons:

  • plotting
  • debugging

Plotting: It seems like there really isn’t a way around it since plotting requires a GUI (unless there’s some interface that allows me to do the heavy lifting on the server, but render plots on my computer simultaneously).

Debugging: It seems like the usual python debuggers (e.g. pdb) have a lot of missing features that would be really nice for deep learning workflows in particular. I find using pdb to be a drag to debug code on the server, and instead find it much easier to just use jupyter notebooks.

Am I missing something? What is everyone’s workflow as far as training on servers but doing other stuff on computers with displays?

2

I use jupyter notebooks using port forwarding.
In my .ssh/config file I add
LocalForward 8889 127.0.0.1:8888
for the proper host

I use port 8889 locally, because I still want to be able to use the default 8888 port for my local jupyter notebooks.

that way I can just access the jupyter notebooks remotely in my local Chrome, and do plotting as well.

1 Like
3

What about using sshfs?

4

Use X11 forwarding if you want to do pyplots over SSH