Reading of 'The Matrix Calculus you need for Deep Learning'

Our study group has just finished reading the chapters of fastbook, and are now moving into The Matrix Calculus you need for Deep Learning. We meet Wednesdays 6-9pm PST on the official discord server: Discord / live coding details (under ‘fastai study groups’)

Prerequisites:
While it seems difficult, most the the prerequisite math will be covered in the next study group. You will be ahead of the game if you know what ‘partial derivative’ means. Knowing what a ‘derivative’ is should suffice. I would suggest trying to look up the meaning of derivative, I have heard Khan University is good. If you are overwhelmed at the next meeting, I can hold a follow up session to catch you up.

Details:
We expect to get through sections 1-4 next week, reading from 6:15-8:30, and everyone is asked to come up with three questions as they read. In the last 30 minutes the group will go round robin asking their a single one of the three questions, and letting the group try to answer. If there is time remaining we will try to answer any remaining questions.
Link to the paper: https://arxiv.org/abs/1802.01528

Up Next: After the above paper we will be reading Ian Goodfellow’s “Deep Learning Book” (https://www.deeplearningbook.org/)

You may find out more about what we have done previously here: [Unofficial, Virtual] SF/Bay Area Study Group

@butchland @tyoc213

1 Like

You can also find a copy of the paper as blog post here:

@tyoc213 @marii - I’ve started reading it and I think I need 2-3 days of advanced reading in order to prepare to cover parts 1-4 (especially 4) – 2-3 hours won’t be enough time if you (like me) haven’t read through and understood it the first time around (especially me, since I think I last tackled calculus/matrices way way back long ago in college).

PS. Its a good refresher though, I’ve started taking notes using Latex again so I can capture the math symbols accurately.

Best regards,
Butch

@butchland Okay sounds good. What do you think about 1-3 on Wednesday then?

hi @marii,

I think 1-3 doesn’t really cover much
1 - Intro
2 - Review of scalar derivative rules
3 - Intro to vector calculus and partial derivatives

So I think we can cover 1-3 pretty easily.

However the section 4 - is the start of the meat of the discussion and I’m not sure we can finish it in 1 session - it consists of several “heavy” subsections :
4a - generalization of the Jacobian
4b - derivatives of vector element-wise binary operators
4c - derivatives involving scalar expansion
4d - chain rules

So I think targeting 1-4 for the next session is fine, I’m just not sure we can finish section 4.
Lets just see when we get to it. (although I’m preparing for it by reading ahead).

Best regards,
Butch
cc @tyoc213

1 Like

Next week we will continue on sections 4 and 5, as we were not able to finish 4 this week. So if you are joining, please work through 1-3 and ~1/2 of 4.

Here’s a link to my notes:

(Not yet completed)

I found these videos from TWIML covering the same topics

Part 1: https://www.youtube.com/watch?v=pQ5HT8LylZs&t=4s
Part 2: https://www.youtube.com/watch?v=rWRb8K-hcTo

I’ve only watched part of Part 1 but it seems to be useful…

Best regards,
Butch

cc: @tyoc213 @marii