Effect of Second layer on first layer results in simple neural net

Part 1 (2020)
1

Hi,

In Ch4. Under the Hood : Training a Digit Classifier.
In Simple net

simple_net = nn.Sequential( nn.Linear(28*28,30),
nn.ReLU(),
nn.Linear(30,1)
)

trying to understand effect of Second Layer on the result of first layer…
First layer will output predictions, now these predictions are feed as input to second layer

Will Second Layer pics the highest probability ?

Thanks

2

Hello,

nn.Linear(28*28,30) receives a data sample (i.e., a handwritten digit) and linearly transforms it to obtain a vector of dimension 30 (30 is an arbitrary figure, and there are myriad other choices, e.g., 90, 60, 20, or 10. However, it is best to experiment and discover which delivers the best accuracy). Next, its result is run through a non-linear layer, i.e., ReLU. Finally, the output of ReLU is passed to nn.Linear(30,1), which yields the score of the input being a 3, with a higher value meaning the model is more confident it was a 3.

Does that make sense?

3

Thank You @BobMcDear for your reply
Yes it does make sense and I understand this.

which yields the score of the input being a 3, with a higher value meaning the model is more confident it was a 3.

So second layer purpose is to yield higher value prediction from predictions of first layer. Right?

Thank You

4

Hello,

You’re welcome.

Sorry, I don’t quite understand your statement - are you asking whether the purpose of the second linear layer is to make predictions given the output of the ReLU, in which case you would be correct (although the story is more nuanced and requires further elaboration), or am I misinterpreting your meaning?

Thank you.

5

Yes, you understand me correctly and that’s what I am asking.

Let me elaborate this more.

Output of first layer is vector of dimension(30), goes through nn.ReLU() layer, After going through nn.ReLU(), it [vector of dimension (30)] goes through second layer which is matrix (30,1). So I am interpreting that

That is what I think is happening…

may be I am wrong or missing something here. if so, kindly point me in right direction

Thank You!

6

Hello,

You are correct in your understanding.

Please let me know if you have further questions or concerns.

7

Thank You
It’s help me to move forward

Thank You

1 Like