Using Skip connections for the fully connected layer


(geethasaikrishna) #1

I am new to the course &I completed till lesson 4.

When I used the learn.summary, it has showed two FC layers connected one after the other till the output sofmax. I am wondering if we could do a skip connection even for the FC layers. Doing so can we increase the number of FC layers & get accuracy gains?

If anyone has any experiments done with adding more FC layers or FC layers with skip connections it would be really helpful to know your experience.


(marc) #2

It’s definitely possible to use skip connection with FC layers. However it will be useful only if your network has lots of layers. With only a few layers, the gardients flow well enough and the skip connections don’t really make a difference.
You may want to read http://arxiv.org/abs/1708.05123
they use FC with skip connections.


(geethasaikrishna) #3

@marcemile thanks for sharing this. It is helpful…:slight_smile: