Sharing my notes for this lesson.
Collaborative filtering:
(Pretty similar to the lesson notebook, but with some added screenshots/explanations)
Gradient descent/optimization techniques:
I wasn’t sure how much of the optimization techniques were going to be covered in future lessons, so I ended up doing quite a bit of background study on my own (never a bad thing of course).
I found the Excel method of working through the algorithms works very well for getting an understanding. Though I did have one question that I was never able to resolve (Question about graddesc.xlsm).
Here are some links I found useful for some of the “lower level” stuff.
Derivatives of multivariable functions (talks about what partial derivatives are, the gradient, has some GREAT visualizations of showing the gradient as slope of steepest ascent, the Jacobian, etc.
This course is REALLY good – it’s done by the same guy who did this very popular neural network series on YouTube: https://www.youtube.com/watch?v=aircAruvnKk&t=2s.
Anyway, the course is:
It seems like a lot but the videos are fairly short. This lesson in particular is helpful:
Backprop (calculating derivative) as a circuit: (video version of Stanford notes linked above)
RMS prop:
Adam:
Weight decay/L2 regularization: