Correct Order of BatchNorm and ReLU

Part 1 (2019)
1

In “lesson7-resnet-mnist.ipynb” , “Basic CNN with batchnorm” has order — Conv2d - BatchNorm - ReLU
But fast.ai method ‘conv_layer’ has order as ---- Conv2d - ReLU - BatchNorm

What is the correct order for these layers?

1 Like
2

There have been massive discussions on where to put the BatchNorm…
Intuitively, doing BatchNorm - ReLU seems a little odd, because you will throw away all your negative values from your freshly normalized batch, which (I think) is the reason why its ReLU - BatchNorm in fastai. Most other implementations do it the intuitively odd way (and so does the original paper I think). In the end it does not make an immense difference.