During the lesson today somebody mentioned that having a debugger was the one missing piece for Jupyter Notebooks. So just wanted to mention available alternatives that I’ve actually used
The cheapest/easiest is Visual Studio Code (VSCode). It’s free, works well, and has most of the features that I wanted out of Jupyter Notebooks that I can’t get without installing extensions. Also, it runs Jupyter automatically for notebooks without you having to run the Jupyter instance via terminal. You do have to have Jupyter installed though …
More info can be found here:
The other option that I’ve personally tried (which provide easy debugging) is DataSpell from JetBrains:
This is a paid product but they do have a 30-day evaluation version. And they do provide a free open source license to qualifying developers. I found VSCode to be faster/easier for my personal use but the general impression I got was that DataSpell has a lot more features. But it feels a little slow and has a slightly higher learning curve.
Just wanted to share something I had noticed, not sure if it makes a difference to others.
For a classification task where it’s one hot encoded. the cross entropy loss becomes -log(predication), if you graph -log(x) it makes a really nice curve for gradient decent to move prediction close to 1.
Yeah, the intellisense and markdown folding were the features I love the most via VSCode But there’s a bunch of small things which makes the overall experience so much better under VSCode! The only trouble is that the features are spread across several different menus and so you have to hunt around a bit till you get used to it …
If you’re hungry for another run at derivatives, the chain rule, and backprop, I enjoyed working through Andrej KArpathy’s “micrograd” videos. It zooms in more to how single “neurons” interact and digs a bit deeper into building a Pythonic autograd class hierarchy. Really great stuff that complements JH’s material. The spelled-out intro to neural networks and backpropagation: building micrograd - YouTube
I agree. I have to do most code development on a remote host and have tried most of the better known and a few lesser known IDEs. When I need to debug in this environment %debug and pdb are the tools I use to solve the majority of my problems since they are so easy and %debug in particular gets you straight to the problem.
The debate about which IDE is going to last forever. Me experience is:
Pycharm: Excellent for writing code with many advanced features but needs the paid pro version for remove host and has in my opinion a fundamental flaw in that the code has to reside locally and pycharm then syncs to the remote machine to run the code when required. If you also run notebooks on the remote host over ssh then this can often mess up the syncing and causes problems. For this reason I stopped using it. Debugging code worked very well with pycharm, possibly the best of all of them and very fast.
Wing Pro: Less well known and again you need the pro license to work on a remote host. Support for writing code not as good as pycharm or vs code but very good for remote debug sessions with probably the fastest and most robust use.
VS Code: Great for writing code as mentioned by @Fahim but I find very slow and flaky when I try to use it in debug mode over a remote host
The one thing that messes all of them up is if you have to debug something that has multiple threads such as pytorch dataloaders, but usually this can be fixed by setting num_workers to 1 or something similar.
I’d be interested in the opinion of others on the course that probably have more experience than me.
I found this to be excellent also. Implementing backprop “by hand” made it all click for me. Showing it with regular ole scalars helps focus the core concepts. It’s just one screen of code! After that, I felt like I could implement anything from scratch.
I just caught up on all the lessons! So pumped that we are doing everything from scratch. Looking forward to following along and creating notebooks to train on my own datasets!
We saw the log rules for the quotient and multiplication: ln(x/y)=ln(x)−ln(y) ln(x*y)=ln(x)+ln(y)
just a reminder that ln(x) / ln(y) and ln(x) * ln(y) which look similar have no simplification rules.
A tip that helps me remember this: x * y can lead to large numbers and applying log to it converts the multiplication to a summation, which is smaller than the multiplication.
Hope this helps someone
This was the first of the fastai lessons so far where I got REALLY lost, probably because I’ve never studied calculus and because things move faster around that.
I’m going through everything in this week’s notebook really slowly and expanding my explanations to myself in notes as and where something feels like a step or progression is compressed. Same with some of the new terms / shorthands for things we’d been doing in part 1 but that were never referred to with those terms (like ‘back propagation’ etc). Will get there in the end, I hope, and will try not get dissuaded by the forward march of the lessons!
I don’t think there is a substitute for debugging in the exact environment the code is running in and which it was developed for. Anything else is a compromise.
