Lesson 6 RNN Model:Dimension propagation in forward method of the model


(Irshad Muhammad) #1

Hi
I am trying to figure the dimension of the matrixes while going through the forward method, I am unable to get it right. I have written my understanding of dimension in comments. (complete code below)

#assume bs=512,  n_hidden=256, 
    def forward(self, *cs):
        bs = cs[0].size(0)  # bs = 512
        h = V(torch.zeros(bs, n_hidden).cuda()) # h  dim =  512 x256
        for c in cs:
            inp = F.relu(self.l_in(self.e(c))) #inp dim =256 x 1, since n_hidden=256
            h = F.tanh(self.l_hidden(h+inp)) #how can we add h(dim=512 x 256) to inp(dim= 256 x 1 )

Can anyone tell me where I am tripping.

Here is the complete code of the model for reference

class CharLoopModel(nn.Module):
    # This is an RNN!
    def __init__(self, vocab_size, n_fac):
        super().__init__()
        self.e = nn.Embedding(vocab_size, n_fac)
        self.l_in = nn.Linear(n_fac, n_hidden)
        self.l_hidden = nn.Linear(n_hidden, n_hidden)
        self.l_out = nn.Linear(n_hidden, vocab_size)
        
    def forward(self, *cs):
        bs = cs[0].size(0)
        h = V(torch.zeros(bs, n_hidden).cuda())
        for c in cs:
            inp = F.relu(self.l_in(self.e(c)))
            h = F.tanh(self.l_hidden(h+inp))
        
        return F.log_softmax(self.l_out(h), dim=-1)