I always thought thought
a -= b is exactly same as
a = a-b , however I found out if I do something like this
for l in model.layers: if hasattr(l,'weight'): l.weight = l.weight - lr * l.weight.grad
I will get an error
TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)"
In my code,
l is a
l.weight.grad * lr is
torch.nn so I understand where this error comes from
What surprises me is if we rewrite it with
-= (like below) - it works fine!
for l in model.layers: if hasattr(l,'weight'): l.weight -= lr * l.weight.grad
I want to dig in a little bit deeper to understand what cause this difference however I don’t know where to start; according to Python doc these two things should be exactly the same; if anyone know anything, I would really curious to know