I had exactly the same question when I was watching these videos. I think there is something wrong with the math, at least if you are doing conventional matrix multiplication (i.e. rank 2).
As the OP mentions, the first matrix (vector) he shows is 3 rows (RGB) and one column (single pixel) so it’s 3 x 1. The matrix that he multiplies this vector by is 3 x 5. So the “inner numbers” are 1 and 3, i.e. they do not match. I assume that this is similar to the zero-rank array problem I had earlier, where the shape of the pixel vector is actually (3,) not (3, 1). But I don’t have a heuristic to understand how these zero rank arrays work (i.e. why is the 3 treated as the “inner” number, not the “outer” number, and what is the presumed “outer” number?).
In the Coursera course, the 3 x 5 weight matrix would have been transposed to 5 x 3 and placed in front of the activation matrix of 3 x 1. You would then get a 5 x 1 vector, as shown. It’s true that Jeremy’s way of doing it gets the same result, but it’s not marix multiplication as I’m familiar with it (but I’m no mathematcian!).
Later on, Jeremy uses an excel spreadsheet and does use conventional matrix multiplication where the inner numbers disappear, so I assume he is using some sort of unconventional shorthand in the OP’s example. But if I’m missing something then I’d love have someone explain it to me. I found it very confusing.
This is a great course, Jeremy is brilliant, it’s all free – I’m not discounting any of that. But I will say that for simpler minds such as my own, these unconventional and unexplained math variances make it much more difficult and time-consuming for me to follow because I always feel like there’s something that I don’t understand.
Welcome anyone’s insight to set me straight.