Correct Order of BatchNorm and ReLU

In “lesson7-resnet-mnist.ipynb” , “Basic CNN with batchnorm” has order — Conv2d - BatchNorm - ReLU
But method ‘conv_layer’ has order as ---- Conv2d - ReLU - BatchNorm

What is the correct order for these layers?

1 Like

There have been massive discussions on where to put the BatchNorm
Intuitively, doing BatchNorm - ReLU seems a little odd, because you will throw away all your negative values from your freshly normalized batch, which (I think) is the reason why its ReLU - BatchNorm in fastai. Most other implementations do it the intuitively odd way (and so does the original paper I think). In the end it does not make an immense difference.