Outlined here is my current ideal workflow for this course and machine learning projects in general (small caveat: I’m new to ml but not to software). I like it because it allows developers to use their finely-tuned local text editors and supports git/GitHub for version control, which feels like a necessity for me for non-trivial projects, without losing the benefits of Jupyter. Here it is:
- Pick a favorite text editor with support for Jupyter notebooks. Right now, choices appear to be Atom with Hydrogen and Visual Studio Code with the Jupyter plugin. Vim keybindings should be available for both as well for those inclined.
- Establish a folder and repo for your project using either
git clonefrom a GitHub repo.
- Work on regular Python (
.py) files using your aforementioned favorite text editor, committing and pushing changes as you make them with git.
- Whenever you need to know the result of a computationally-expensive block of Python code, leverage a remote server with a GPU thanks to the Hydrogen/Jupyter plugins.**
- To communicate your results clearly, write them up in a top-level Jupyter notebook (
.ipynb) file. I think this format is well-suited to top-level communications because of its ability to combine figures, code, and nicely-formatted written explanations. But, as described above, I’d rather work with basic Python files with version control for development/iteration, particularly for the details of a project.
- Finally, because of the use of git/GitHub, it’d be fairly straightforward to
git pullthe repo onto a remote server and write the top-level there if any code snippets it needs to execute for demonstrations or results would benefit from the use of a GPU (currently, I don’t think it is possible to run snippets of a local
.ipynbfile remotely like it is for local
What do you guys think?
**This part is the key. Although I initially had trouble trying to connect to Crestle in this way when I first posted a few hours ago, I have since verified it works using a friend’s computer with ngrok to tunnel to the friend’s
localhost and thereby run snippets of my code on their Jupyter instance. The same should be possible to set up on a generic remote server with better hardware, provided sufficient privileges.