What's your development setup?

I thought I’d share my setup (how I’ve been developing with IPython), and see if others are doing the same or would like to.

I have a Windows desktop at home, running a 1080Ti.

I setup a VPN (I use OpenVPN) on my router, and use dynamic DNS to a subdomain (of a domain I own).

I host the jupyter notebook as a server (which is super low bandwidth to use and run the cells- I sometimes do it on my phone!)

I also use RDP if I ever need to do Dev ops stuff.

Security Note: I do not forward the port of the IPython notebook. It’s only accessible through my VPN.

1 Like

It would also be interesting to know how folks edit and browse source. Personally I use vim (with ctags for creating the symbol library) with a very small number of customizations in vimrc, in a tmux session on my server. If I have a slow connection (or no connection) I’ll work locally, using either gvim or vscode.

I normally use ssh port forwarding for connecting to the jupyter notebook, although for the university server they make us use a rather flakey VPN (GlobalProtect).

I use port forwarding for jupyter notebook as well. To browse source code in fastai I use pycharm since I am using intellij at work so the interface is quite familiar.

Are battlestation pics allowed? :stuck_out_tongue:

I use this script I wrote to connect and port forward my AWS sessions. I run Windows 10 locally, with the Ubuntu on Windows shell to connect, usually running tmux. I use vim (or rather often ??[insert class here] in Jupyter notebook) to view the fastai source code. I might switch to vscode, but my profession (robotics / embedded engineering) demands I use vim rather often, so I feel that I should embrace the learning curve as opposed to using vscode or a similar IDE.

I would like to upgrade to a 1050Ti or better and stop using AWS, but currently my student budget says I should wait a few months.

Looking forward to seeing other setups, I’m always looking for ways to improve mine!

4 Likes

I used to use vim exclusively and had a really custom setup. I eventually switched over to IntelliJ / jetbrains ecosystem for Intellisense, code generation, plugins, and how well tags/indexing and refactoring works.

I now use vim for cleaning / altering / generating text data. If anyone has never used “recording” (macros) in vim, try it out.

You can make a little function, which manipulates some target text, and repeat it as much as you want. (so chaining searches and modifications and repeating them for every line (or match) that matches the search, etc.)

I probably explained that horribly, but basically search and replace for things you can’t describe with a Regex.

Open VPN is something I plan to setup as well.

I have setup pfsense as a firewall for all traffic and want to setup VNC for two boxes on the network so that I can login to my GPU box and run desktop apps on it (eg Pycharm) from other machines on the network. I only need to run VNC over the local network.

Any tips for using Open VPN?

For general source code browsing I’m using Pycharm. Normally I use Eclipse+pydev for python work as latency on interactions is lower than PyCharm (in my experience), but sometimes browsing nested packages needs extra setup, and in these cases often easier to use PyCharm. I am actually trying out using PyCharm as full replacement for Jupyter Notebook - get notebook + ide all in one.

PS I’ve never been able to remember any Vim commands, if I have to use an editor I use nano :slight_smile:

I tunnel port 8888 over ssh and use noip.com and port forwarding for accessing a box behind a NAT with ever changing public IP.

I use the same set up as Jeremy (vim + tmux). I also use tmuxinator which is really neat for bringing a tmux session up.

Locally, I use i3 which is quite easy to learn and makes laptops - even with smaller screens - much more user friendly. You (nearly) never have to use the mouse. Can be achieved with other windows managers as well but I find i3 to fit my needs perfectly.

And that is the crappy laptop that I use. Though would probably have very little use for something with beefier specs up to the point where it would have a GPU. But those things remain outside my budget for now.

Having said the above, if I were to come across this thread when I was just starting out with programming (which was very recently), I would be completely intimidated.

If one is able to reliably access jupyter notebook with a GPU, that is nearly all that matters.

Despite being a die hard fan of vim + tmux + i3, I am not sure I would have gone this route if I were to do it all again. Maybe yes, maybe not, but probably not.

Also, nearly universally, regardless of the field, the tools are secondary or ternary at best even for fields one would think are very technical, such as photography. There was one professional snooker player (whose name I forgot) who played with a < 10$ cue. And many, many other examples could be mentioned.

5 Likes

My router is an Asus RT-Arch 13. I got it for a good deal a while ago on Newegg.

It natively supports OpenVPN, just had to go to the admin page (usually accessible by going to your router/gateway ip). Which is usually .1- often 192.168.1.1, but doesn’t have to be. 10.0.0.1 is also popular.

If yours doesn’t natively support it- check to see if you can flash ddwrt. That’s a bit involved and you can brick your router, so unless you have done something like that before, I’d probably recommend against it.

Not sure if that’s the only way (at the router level) but it’s how mine’s set up.

I personally would never port forward because I’m not confident in my ability to properly setup a public facing firewall and I’m concerned about security.

bots will find open ports.

I use ssh tunnels to access jupyter notebooks remotely and then mount the remote home directory with sshfs so that I can use a local editor on my MacBook. As a local editor for Python I have used PyCharm for a while now. For my research I also have to edit Fortran, Matlab, R files and Shell scripts regularly, for which I have started to use Visual Studio Code.

After hearing about the possibilities of jumping around in code in lesson 8 (I just always used plain search -.-), I will probably make a full switch to VSC, so that I only have to memorize one set of shortcuts.

I have to switch computing environments often (AWS, different super-computers, …) which is why I use screen instead of tmux which isn’t installed in many environments I use and I don’t have root permission to install it.

Knowing enough Vim to navigate and edit a file has also turned out to be super helpful in many cases where I couldn’t mount my remote directories.

I recently got a Surface Pro. I am still in the process of setting up things to work as they do on my Mac. I currently use Putty to ssh with port tunnels and win-sshfs for remote mounting. Unfortunately, one of the super-computers I am working with has two-factor authentication which doesn’t seem to work with win-sshfs, but that’s a minor issue.

A minor but super important productivity boost for me is having a convenient alias for every ssh(fs) command and having ssh-keys set up everywhere. Watching Jeremy’s videos really made me realize that I have to make everything as convenient as possible for myself. Strangely enough I tortured my self with copy/pasting and typing long commands for years before!

1 Like

Git Bash on Windows is actually really good. I highly recommend it.

I’ve had a much better experience than with PuTTY and other bash emulators.

Thanks for the tip. I will try it out :slight_smile:

I’m using my isp provided router and directing all traffic through a low power server running pfsense. I am thinking of upgrading the router to a Sophos XG series but am pretty time constrained and know setting this up will take more time than I want.

Setting up Open VPN on pfsense looks pretty straightforward and on my todo list…

I use TurboVNC at work to run 3d graphics applications on remote machines, rendering is really good and mouse latency and interaction pretty good too. I expect even better performance on my network.

If you want to stick with what you know:

https://github.com/kasecato/vscode-intellij-idea-keybindings (i presume intellij bindings are consistent with pycharm).

(I use eclipse bindings in pycharm).

If pydev author ports the debugger to vs code will be great as its a pleasure to use:

http://www.pydev.org/vscode/

Yeah it’s pretty amazing at that…

I am on google cloud + ubuntu for this course (not building own machine), and a very beginner to coding. I would like to find a way to browse the codes that Jeremy shows.

I have probably missed it somehow. I have installed pycharm (community edition) and VS Code on my macbook, trying to find a way to open the codes, not sure how.

I tried following the instructions for VS Code here, but the command not sure what does the keystroke Control+Shift+P refer to as I have tried the Mac equivalent but it didn’t seem to work.
https://marketplace.visualstudio.com/items?itemName=donjayamanne.jupyter

For pycharm, I understand that the professional version is required, so I wasn’t able to connect, although I tried but the SSH options are not available.

If any kind soul could do a step by step write up on how to access the codes, it would be great :smiley:

For pycharm on your laptop you can use the free version just fine.

I presume youve installed git, then cloned the fastai repo to your laptop (git clone https://github.com/fastai/fastai.git).

To browse code only these 2 steps are optional: 1)Download minconda3 from conda.io and install it.
2)Go into the fastai folder on laptop and open the readme.md file and follow the instructions if you want to be able to run code locally (eg on cpu)

On laptop now open pycharm then open the fastai direcory (file->open).

If you installed conda and fastai (id recommend doing so so that you can jump from the fastai code base into pytorch and dig around in there too): Then go to file->settings->project. Fastai should be already the project name. Then select project interpreter. Select the little cog buttton on top right, add local, select existing environment button. find the conda path in your home directory, go into the conda env directory select fastai/bin/python3.6

If not, pycharm will just pickup the default python install (fine for browsing but will show red underscores indicating it cant find libraries)

1 Like

thank you so much @adrian!

I am running fastai notebooks in a docker container on a headless linux box at home. I access jupyter notebook/lab, container contents and the server via ssh and tunnel aliases. I also tunnel in through my home’s router. Nano for simple remote text manipulation. VS Code ( via sshfs ) and Jupyter for Fastai code exploration. Trying Pixiedust visual debugging magic in Jupyter.

Rich code exploration along with visual debugging support for fastai notebooks running in a remote docker container seems to almost be available.
(wow!)
options that seem to be under active development at this time:

  • MS seems to be working on a VS Code solution with Jupyter.
  • PyDev seems may be working toward a VSCode/Jupyter solution as well.
  • Am I missing an option?

A Portable CPU (Gaming laptop) running Ubuntu 18.04 (Yes, I got CUDA Up on it. I’m very proud of that)

Here is a little fun idea that I found:

If I leave the port forwarding open, I can access jupyter nb using my Mobile phone during college lectures.

(reason: My college doesn’t allow us to use Laptops during lectures and secondly, The gaming laptop is too heavy to carry around)

So I set the hyper parameters in different cells and set a Notification alert using Telepyth-it sends notification to me via Telegram and I can check the errors or loss values, then I just login using my mobile browser and set execution on a different cell.

I do interactive coding when at home, but when away I leave a few play ideas open that I can run using my mobile phone combination.
This is the only Time I use the Play button in Jupyter notebook over shift+enter but it is a really great idea-keeping your machine busy even when you are travelling.