Is this the right way to perform backpropagation?

Part 2 (2019)
1

Hi everyone,

I had troubles understanding the math of backpropagation, thus I decided to get more information on the subject. I stumbled upon this lesson the CS231n Winter 2016: Lecture 4 video. In this lesson, they made Computer Graphs that they used to calculate the forward and backward pass by hand. To see if I understood the material I implemented this for the model used in Lesson 8 (Linear–>Relu–>Linear–>MSE). However, I am not sure if I did it right. So my question is: can someone check this for me? Especially, the ReLU part was somewhat confusing.

Here is the assignment:

backprop_question
backprop_question.JPG1332×767 96.4 KB

Thanks a lot for your time!

Mees Molenaar

2

Hi,
at first look it seems ok except that those formulas are valid for a scalar. When you have matrix and vectors (the actual case in deep learning) than you have to consider Transposes and multiplication may not always so obviously defined.
Some good material here https://explained.ai/matrix-calculus/index.html

2 Likes
3

You are right, this exercise was performed with scalars. I have to look into the case of matrices!
Thanks a lot for your response and link to the material.

Mees