Here is the code I am referencing, inside the
init method of the dynamic unet, we have this two sections:
middle_conv = nn.Sequential(ConvLayer(ni, ni*2, act_cls=act_cls, norm_type=norm_type, **kwargs),
ConvLayer(ni*2, ni, act_cls=act_cls, norm_type=norm_type, **kwargs)).eval()
unet_block = UnetBlock(up_in_c, x_in_c, self.sfs[i], final_div=not_final, blur=do_blur, self_attention=sa,
act_cls=act_cls, init=init, norm_type=norm_type, **kwargs).eval()
So I was digging through the code and was confused by this. Why are we calling the
.eval() method after creating them?
Don’t we want them in training mode?
I believe this is because dummy data is being passed through the model at the time it is initialized to determine input/output shapes the layers and calling
.eval() prevents any updates to batch norm stats during this process.
This file has been truncated.
"! [ -e /content ] && pip install -Uqq fastai # upgrade fastai on colab"
That seems like a plausible reason. However I can’t see the code that switches it back to training mode. Or will this happen automatically during the training loop?
I am assuming training on eval mode will not enable us to update the batchnorm which is undesirable behavior.
I believe that switch is automated during the training loop because eval mode should be turned on as well when your validation set is evaluated every epoch.
Awesome. Thanks for the help!