In the Dynamic UNET, why are we creating the middle conv and unet block in eval mode?

Part 1 (2020) Non-beginner
1

Here is the code I am referencing, inside the init method of the dynamic unet, we have this two sections:

middle_conv = nn.Sequential(ConvLayer(ni, ni*2, act_cls=act_cls, norm_type=norm_type, **kwargs),
                                    ConvLayer(ni*2, ni, act_cls=act_cls, norm_type=norm_type, **kwargs)).eval()

and

unet_block = UnetBlock(up_in_c, x_in_c, self.sfs[i], final_div=not_final, blur=do_blur, self_attention=sa,
                                   act_cls=act_cls, init=init, norm_type=norm_type, **kwargs).eval()

So I was digging through the code and was confused by this. Why are we calling the .eval() method after creating them?
Don’t we want them in training mode?

2

I believe this is because dummy data is being passed through the model at the time it is initialized to determine input/output shapes the layers and calling .eval() prevents any updates to batch norm stats during this process.

https://pytorch.org/docs/stable/notes/autograd.html#evaluation-mode-nn-module-eval

1 Like
3

Hello :wave:
That seems like a plausible reason. However I can’t see the code that switches it back to training mode. Or will this happen automatically during the training loop?

I am assuming training on eval mode will not enable us to update the batchnorm which is undesirable behavior.

4

I believe that switch is automated during the training loop because eval mode should be turned on as well when your validation set is evaluated every epoch.

1 Like
5

Awesome. Thanks for the help!