What's the difference between` a -= b` and `a = a-b`?

Part 2 (2019)
1

I always thought thought a -= b is exactly same as a = a-b , however I found out if I do something like this

for l in model.layers:
                if hasattr(l,'weight'):
                    l.weight = l.weight - lr * l.weight.grad

I will get an error
"TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)"
In my code, l is a nn.module and l.weight is torch.nn.Parameter and l.weight.grad * lr is torch.nn so I understand where this error comes from

What surprises me is if we rewrite it with -= (like below) - it works fine!

for l in model.layers:
                if hasattr(l,'weight'):
                    l.weight -= lr * l.weight.grad

I want to dig in a little bit deeper to understand what cause this difference however I don’t know where to start; according to Python doc these two things should be exactly the same; if anyone know anything, I would really curious to know

Thanks

1 Like
2

Try writing it like this:

l.weight = l.weight - (lr * l.weight.grad)
3

tried and it is the same -

Out of curiosity, why would you think it will make a difference? I always thought a - b * c would be same as a - (b*c)

4

I guess they are the same, is it possible to share a small code that reproduces this error. It becomes a lot easier to debug.

5

sure - I created an small code example below, basically when we are trying to update the weight attribute of an nn.module object, it will throw a TypeError if the new value is not an nn.Parameter; for some reason we can get around this when we do -= or +=


import torch
m = torch.nn.Linear(10,1)

# this won't work - will get a TypeError, weight attribte of an ann.Module has to be a Parameter 
with torch.no_grad():
    m.weight = m.weight + torch.randn(m.weight.shape)

# However when we use += we can get around with the TypeError, below code works fine
with torch.no_grad():
    m.weight += torch.randn(m.weight.shape)
6

Basically the error mentions that the weight has to be of type nn.Parameter not some FloatTensor. Converting the output to a type of Parameter makes the code a+b work. It has to be of type Parameter as it will also store gradients. The below code works.

with torch.no_grad():
    m.weight = nn.Parameter(m.weight + torch.randn(m.weight.shape))

Why this works?

with torch.no_grad():
    m.weight += torch.randn(m.weight.shape)

My guess is, this function is overridden somewhere in the torch code to do the auto casting. I could not find the source code for the same. If I can get it, I will update it here.

1 Like
7

Most generally the difference is a += b is in-place operation while a = a+b is out of place operation. So translating to pytorch the former is a.add_(b) and latter a.add(b). I would dig in that direction to understand the underlying error.

3 Likes
8

Thanks everyone, so here is what I’ve learnt so far:

  1. a += b is NOT syntax sugar of a = a + b : one mutate the object in-place while the other makes a copy; this is not just unique for Pytorch, you can find the same for python list
  2. Normally when you add nn.Parameter and a regular torch.tensor together you will get a regular tensor; However with in-place operation, you only modify the .data of the tensor so the type of object will not change
a = nn.Parameter(torch.tensor([1.,2.]))
b = torch.tensor([3.,4.]) 
a = a+b

a is a torch.tensor in above code

a = nn.Parameter(torch.tensor([1.,2.]))
b = torch.tensor([3.,4.]) 
a += b

type(a) returns nn.parameter

2 Likes
9

one more thing I just learnt that’s relevant - when you update an attribute in-place, it will not triggle the __setattr__ method at all; therefore even the types aren’t correct, you will not get an exception