Trouble with a simple cross recognizer

(S_s) #1

After doing the simple SGD demo, I wanted to get a better grasp of the backprop so decided to
implement a simple network which will recognize a ‘X’ sign in a 3x3 array.

ie if its fed the ‘X’ (below) as input, it should output 1 for all the rest 0.

tensor([[1., 0., 1.],
        [0., 1., 0.],
        [1., 0., 1.]])

I am finding that the network does not converge to expected outcome.

I was expecting the learned weights to be like below, but the code does not converge to that.
Appreciate any help or pointers to what needs tweaking? Thanks!

tensor([[1., -1., 1.],
        [-1., 1.,-1.],
        [1., -1., 1.]])

Code below:

#'X' represented as 3x3
x = torch.FloatTensor([[1,0,1],[0,1,0],[1,0,1]])
#count of training samples
sample_count = 10
ratio_of_crosses_in_input = 0.33

#Create sample_count no of samples of 3x3
input_ = torch.rand(sample_count, 3, 3)
#Create a bias element to append to end of image data
bias = torch.ones(sample_count,1)
#reshape the image sample from 3x3 => 9x1 to make it a linear 1D array
input_1x = input_.reshape(sample_count,9)

#Add bias as last element, ie 10th element
input_with_bias = torch.cat((input_1x,bias), 1)
#Make % of the samples of type 'X'
input_with_bias[int(sample_count*(1-ratio_of_crosses_in_input)):,:] = torch.cat((x.reshape(1,9)[0],     torch.ones(1)), 0)

#Create expected result tensor with all outputs corresponding to 'X' set as 1
expected_y = torch.zeros(sample_count, 1)
expected_y[int(sample_count*(1-ratio_of_crosses_in_input)):] = 1

#Create single layer nn, initialized to random
layer1 = torch.rand(10) - 0.5
#MSE
def mse(y_hat, y): return ((y_hat -y)**2).mean()

lr = 0.05
sig = torch.nn.Sigmoid()
layer1 = nn.Parameter(layer1)

def update():
    y_hat = input_with_bias@layer1
    y_hat = sig(y_hat)
    loss = mse( y_hat, expected_y)
    loss.backward()
    if i % 5000 == 0:
      print("Loss:")
      print (loss)    
  
    with torch.no_grad():
      layer1.sub_(lr * layer1.grad)
      layer1.grad.zero_()

#Update the weights 
for i in range(50000):
  update()
0 Likes