Hey all,
I was trying to submit a PR for QHAdam optimizer (in 12_optimizer.ipynb) but I noticed the last test for average_grad is failing. I have been trying to fix it so that I can submit, but I’m blocked. Any help would be super appreciated!
This is the offending test:
…
test_eq(state[‘grad_avg’], (0.1 * 0.9+0.1) * p.grad)
However when I dig into it I can’t for the life of me figure out why its failing as the 2 tensors compared appear to be identical
AssertionError: ==:
tensor([[0.7600, 0.9500, 1.1400]])
tensor([[0.7600, 0.9500, 1.1400]])
I have traced it back to torch.equal() but I have hit a brick wall. Manually checking that both tensors are equal returns True, as expected:
torch.equal(tensor([[0.7600, 0.9500, 1.1400]]),tensor([[0.7600, 0.9500, 1.1400]]))
True
To add to my confusion, the other tests work without any issues…
p = tst_param([1,2,3], [4,5,6])
state = {}
state = average_grad(state, p, mom=0.9)
test_eq(state[‘grad_avg’], p.grad)
state = average_grad(state, p, mom=0.9)
test_eq(state[‘grad_avg’], p.grad * 1.9)
#Test dampening
state = {}
state = average_grad(state, p, mom=0.9, dampening=True)
test_eq(state[‘grad_avg’], 0.1 * p.grad)
state = average_grad(state, p, mom=0.9, dampening=True)
test_eq(state[‘grad_avg’], (0.1 * 0.9+0.1) * p.grad)
Is this some hidden pytoch or python voodoo that is tripping me up?