Can I have feedback on article about batchnorm please?

(Alvaro Durán Tovar) #1

Hi! I just wrote this article about batchnorm and I’d really appreciate if I can have some feedback on missing important bits or if I said something incorrect.

Just starting writing articles, it’s quite fun!

Thanks!

0 Likes

(Morgan McGuire) #2

Looks great @alvarodt! I wouldn’t be the best to comment on the technicalities, but I appreciate that it is short and easy to understand and gives some tips (e.g. the transfer learning tips)!

Maybe you could include some more fastai code for any recommendations? e.g for the below:

It’s a good idea to unfreeze the BatchNorm layers contained within the frozen layers to allow the network to recalculate the moving averages for you own data.

You could show an example of how to do that in fastai :slight_smile:

Otherwise, well done, look forward to reading more!

`

0 Likes

(Alvaro Durán Tovar) #3

Thanks! My intention for this blog is that, explain deep learning concepts as simple and practical as possible. I’m amazed how some people is able to show something as very complex while it being simple to use.

You are 100% correct! I’ll add the code when have time for it.

Also about the transfer learning trick I’m sure I have hear it from Jeremy in some video but can’t find it, I would like to link to it.

1 Like

(tal) #4

One of the unknown secrets about batchnorm is that it is very memory intensive.
for Resnet models, for example, batchnorm layers are accounted for ~40% of the total GPU memory consumption.

you can consider adding to the tips section using " In-Place Activated BatchNorm", which really solves the memory consumption problem:

I tried it, and was able to almost double my batch size for resnet models.

1 Like