That’s a good question !
When I implemented my home-cooked Adam optimizer class I naturally initialized it outside of the loop but did not compare it to initial Adam, I just went on implementing nn.Linear.
And when looking at the source code of Pytorch’s Adam I don’t see anything special that would justify it.
https://pytorch.org/docs/stable/_modules/torch/optim/adam.html#Adam
So it might be a mistake…