Jeremy is answering this now.
1 Like
That’s what Jeremy is showing. But unless you have dampening, momentum doesn’t have the 0.1. Check the PyTorch source code 
3 Likes
You’re not doubling it since your old contributions have 0.9**i after i iterations, and that get to 0 pretty quickly.
1 Like
what about the bias correction of the exp moving averaging
Jeremy is talking about it now.
Is this why I saw opt.lr overshooting my target lr the other day?
Edit: actually I bet it’s becaue of the opposite… since momentum goes down the dampening does too so the amount added increases compared to the amount coming in from previous iterations 
I don’t know what optimizer you were using, but if it was the default Adam, the previous conversation isn’t applicable.
1 Like
Noooooooooooooooooooooooooooooooooooooooo, Jeremy you are breaking it!!!
16 Likes
Is eps still in the wrong place? The divide will happen first.
Edit: It matches up with the forumla, though. So it must be correct.
3 Likes
Don’t listen to Jeremy, in Adam, the epsilon goes outside the square root. He just undid a fix I pushed a week ago where our Adam wasn’t working.
23 Likes
Jeremy just said you can put the epsilon inside the sqrt or not, but you’re saying inside it’s breaking Adam? Do you know why it is breaking it?
2 Likes
Is the smell test is to check if everything in the denom could go to zero and the epsilon is the safe term?
The real answer is that if you put it inside, you would need a default to 1e-10 or 1e-14, so that it matches the defaut of Adam of 1e-5 or 1e-7 you have in Adam with epsilon outside of the square root.
This is what Jeremy explained when he showed the square roots are different.
2 Likes
epsilon as a hyperparam ?
you said norm is sum of squares but the code says mean of squares (for r1)
Yes, you can even schedule it
3 Likes
Norm is the mean of squares, Jeremy misspoke.
So lamb works well for language models… should it work well for other domains as well? Are there places it wouldn’t be as good as Adam?
1 Like
I thought L2 Norm was square root of sum of squares: http://mathworld.wolfram.com/VectorNorm.html
1 Like
It’s for big batch size training mostly. I’m not sure the difference is going to be visible for small batch sizes, but we didn’t have time to experiment a lot yet.
2 Likes