Hello!
Here’s the rundown. I have included what I deem to be the necessary for understanding.
I have a linear model using PyTorch’s nn.Linear class.
linear_model = nn.Linear(28*28, 1)
I have a custom optimizer class.
class Optimizer:
"""A simple attempt to model an optimizer."""
def __init__(self, parameters, learning_rate):
"""Initialize optimizer attributes."""
self.parameters = list(parameters)
self.learning_rate = learning_rate
def step(self):
"""Update the parameters."""
for parameter in self.parameters:
parameter.data -= parameter.data.grad * self.learning_rate
def zero_gradient(self):
"""Reset the gradient of each parameter."""
for parameter in self.parameters:
parameter.grad = None
And have instantiated this class as such:
optimizer = Optimizer(linear_model.parameters(), 1)
The gradients of this model is calculated by the following function.
def calculate_gradients(x_batch, y_batch, model):
predictions = model(x_batch)
loss = l1_norm(predictions, y_batch)
loss.backward()
l1_norm is defined as such.
def l1_norm(predictions, targets):
predictions = predictions.sigmoid()
return torch.where(targets==1, 1-predictions, predictions).mean()
When I run the model, on the very first epoch a runtime error is thrown:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Input In [17], in <cell line: 1>()
----> 1 train_model(linear_model, 20)
Input In [16], in train_model(model, epochs)
1 def train_model(model, epochs):
2 for epoch in range(epochs):
----> 3 train_epoch(model)
4 print(valid_epoch(model), end=', ')
Input In [15], in train_epoch(model)
2 for x_batch, y_batch in train_dataloader:
3 calculate_gradients(x_batch, y_batch, model)
----> 4 optimizer.step()
5 optimizer.zero_gradient()
Input In [9], in Optimizer.step(self)
10 """Update the parameters."""
11 for parameter in self.parameters:
---> 12 parameter.data -= parameter.data.grad * self.learning_rate
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
What I’ve gathered is that, despite loss.backward being run in calculate_gradients, parameter.data.grad remains of None type and hence cannot be multiplied with self.learning_rate.
Does anyone have an idea as to why loss.backward is keeping parameter.data.grad as None? nn.Linear does make sure requires_grad is set to True so I’m not quite sure what’s going wrong.
I would really appreciate any help! Please let me know if I’m still ambiguous or if I should supply any more information.