Batchnorm transfer


I am finishing my first round on Part2 course and I am quite surprised by the batch norm issue that is discussed around 1:02h. It seems that, when freeze, the batchnorm layers should not be updated. I more or less understand the logic behind but, in practice, I am not sure if this is actually what default (X)ResNets are doing in fastai library.

Could anybody tell me if I should add:

def set_grad(m, b):
        if isinstance(m, (nn.Linear,nn.BatchNorm2d)): return
        if hasattr(m, 'weight'):
            for p in m.parameters(): p.requires_grad_(b)

apply_mod(learn.model, partial(set_grad, b=False))

to my code if I want to unfreeze?


P.S.: I see that @xnet did a similar question before but nobody answers

1 Like

When something is frozen, it’s not updated. I think what Jeremy is saying is: don’t every freeze the batchnorm parameters. This means that the batchnorm parameters should always be updated, regardless of which layer they are from. Your code snippet freezes all the parameters in learn.model except those from a nn.Linear or a nn.BatchNorm2d module.

Thanks for the reply! So when doing learn.freeze() this should also be added to the code right? Sorry for the naive question again :slight_smile:

Does the Learner class as it is defined in Part 2 have a freeze method?