Should Batch Norm statistics be updated during Transfer Learning?

Part 2 (2019)
1

I’m confused about whether the batch norm statistics should be kept frozen during transfer learning.

As another user has pointed out, in part 2 of FastAi 2019 deep learning course, we updated the batch norm statistics. Why are BatchNorm layers set to trainable in a frozen model

However, reading a 2020 guide from Keras documentation tells me to not update the batch norm statistics. https://keras.io/guides/transfer_learning/

  • How should I choose whether to update the batch norm statistics or not?.
  • Is it something very dataset specific?
1 Like
2

I guess that is the case.

This publication has outlined some interesting approaches:

BN ImageNet Transfer

Copying over the batch norm parameters from ImageNet.

BN ImageNet Mean Var

Resampling the values of all batch norm parameters according to the ImageNet means and variances.

BN Identity Init

For the Mean Var Init all of the batch norm parameters are initialised to the identity:

gamma & sigma = 1
mu & beta = 0

1 Like
3

Thank you for the article!