Batchnorm transfer

(Joan Gibert Fernandez) #1

Hi!

I am finishing my first round on Part2 course and I am quite surprised by the batch norm issue that is discussed around 1:02h. It seems that, when freeze, the batchnorm layers should not be updated. I more or less understand the logic behind but, in practice, I am not sure if this is actually what default (X)ResNets are doing in fastai library.

Could anybody tell me if I should add:

def set_grad(m, b):
        if isinstance(m, (nn.Linear,nn.BatchNorm2d)): return
        if hasattr(m, 'weight'):
            for p in m.parameters(): p.requires_grad_(b)

apply_mod(learn.model, partial(set_grad, b=False))

to my code if I want to unfreeze?

Thanks!

P.S.: I see that @xnet did a similar question before but nobody answers

1 Like