Understanding LSTMs - Regarding the 3 sigmoids in Chris Olah's blog

anshbansal · July 18, 2017, 12:15pm

I was reading Chris Olah’s blog at http://colah.github.io/posts/2015-08-Understanding-LSTMs/

I was thinking that given the same input the sigmoid will always give the same output. So why would we have 3 different sigmoids in the cells? Is it just to make the diagram clear or am I missing something?

niazangels · July 18, 2017, 2:38pm

You are probably making one of the two following assumptions, just as I did

The input is fed directly to a sigmoid function
There is a matrix multiplication involved before piping it through the sigmoid function, but the weights and biases are the same for all three cases.

If you look closel, the input and the state remain the same for all three, there is a matrix multiplication involved and the output of this matrix multiplication is what is passed on to the sigmoid function. If the weights and biases involved in these operations are different for the three cases, we can expect the output of the sigmoid to be completely different.

And sure enough…