Adapting RNN to work with tabular time series data

Part 1 (2019)
1

Hello! This is my first post on fastai. Loving the videos so far.

I have completed video 7 and I am trying to adapt the RNN model to predict on a different type of data rather than NLP. My data is tabular time series data, so each column represents a time and each row represents an item. To try to do this I walked through the lesson 7 RNN notebook and I have been making the appropriate changes.

I have successfully transformed the first versions of the RNN to work with the tabular time series data, but I’m struggling to include hidden RNN layers. I will post here the function that I transformed successfully and the one I am struggling with. If anyone has any advice on how to make the second function work that would be greatly appreciated.

==
1st function, successfully adapted (note that the variable names are slightly different):

FASTAI VERSION FOR NLP:

class Model_0(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(nv, nh)
        self.ln_hidden = nn.Linear(nh, nh)
        self.ln_output = nn.Linear(nh, nv)
        self.bn = nn.BatchNorm1d(nh)

    def forward(self, x):
        h = torch.zeros(x.shape[0], nh)
        for i in range(x.shape[1]):
            h += self.embed(x[:,i])
            h = self.bn(F.relu(self.ln_hidden(h)))
        print(self.ln_output(h).shape)
        return self.ln_output(h)

MY VERSION FOR TABULAR TIME SERIES:

class Model_1(nn.Module):
    def __init__(self):
        super().__init__()
        self.l_in = nn.Linear(nv, nh)
        self.l_hidden = nn.Linear(nh, nh)
        self.l_out_1 = nn.Linear(nh, 2)
        self.bn = nn.BatchNorm1d(nh)
        self.bno = nn.BatchNorm1d(2)
        self.classes = [0,1]

    def forward(self, x0, x):
        t1 = self.l_in(x[:,0].unsqueeze(1))
        hi = self.bn(F.relu(self.l_hidden(t1)))
        for i in range(1, x.shape[1]):
            hi += self.l_in(x[:,i].unsqueeze(1))
            hi = self.bn(F.relu(self.l_hidden(hi)))
        tout = self.bno(F.relu(self.l_out_1(hi)))
        return tout.squeeze(1)

==
2nd function, not working

FASTAI VERSION

class Model_5(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(nv, nh)
        self.rnn = nn.GRU(nh, nh, 2, batch_first=True)
        self.ln_output = nn.Linear(nh, nv)  
        self.bn = BatchNorm1dFlat(nh)
        self.h = torch.zeros(2, x.shape[0], nh)
        
    def forward(self, x):
        res, h = self.rnn(self.embed(x), self.h)
        self.h = h.detach()
        return self.ln_output(self.bn(res))

MY VERSION

out = 1
layers = 2

class Model_2(nn.Module):
    def __init__(self):
        super().__init__()
        self.l_in = nn.Linear(nv, nh)
        self.l_hidden = nn.GRU(nh, nh, layers, batch_first=True)
        self.l_out_1 = nn.Linear(nh, out)
        self.bn = nn.BatchNorm1d(nh)
        self.bno = BatchNorm1dFlat(out)
        self.classes = [0,1]
        
    def forward(self, x0, x):
        hi = torch.zeros(layers, x.shape[0], nh)
        t = self.l_in(x.unsqueeze(2))
        res, hi = self.l_hidden(t, hi)
        squeezed = self.l_out_1(res)
        rel = F.relu(squeezed)
        tout = self.bno(rel)
        return tout.squeeze(-1)

When I run this code everything works fine, except when I print the confusion matrix every quadrant has a 0, and when I try to print learn.summary() I get the following error:

~/anaconda3/lib/python3.6/site-packages/fastai/callbacks/hooks.py in <listcomp>(.0)
    149         with hook_params(flatten_model(m))as hook_p:
    150             x = m.eval()(*x) if is_listy(x) else m.eval()(x)
--> 151             output_size = [((o.stored.shape[1:]) if o.stored is not None else None) for o in hook_o]
    152             params = [(o.stored if o.stored is not None else (None,None)) for o in hook_p]
    153     params, trainables = map(list,zip(*params))

AttributeError: 'list' object has no attribute 'shape'

I’m running as follows:

learn = Learner(data, Model_2(),  loss_func=nn.CrossEntropyLoss(), metrics=error_rate)
learn.fit_one_cycle(10, 3e-3)

I am running with the same data for the first model and the second model.

Any advice would be greatly appreciated! I hope this question clear. If not please let me know and I will change it accordingly.

1 Like
2

I have an update on this question. I’ve been following along with part 2 of the course and I tried my model in a custom training loop similar to that in lesson 2. When I run in the custom training loop it seems to work fine, the loss, error and accuracy are all changing. When I run the same model with fastai Learner none of the scores change and the other problems described above are still present. Is it possible that there is a bug in the Learner class? Or am I defining something incorrectly?

This is my training loop:

lr = 3e-3
n = len(data.train_ds)

model_2 = Model_2()

# same optimizer from part 2 lesson 2
opt = Optimizer(model_2.parameters(), lr=lr)

for epoch in range(20):
    model_2.train()
    for (xb0, xb1), yb in data.train_dl:
        pred = model_2(xb0, xb1)
        loss = loss_func(pred, yb)
        loss.backward()
        opt.step()
        opt.zero_grad()

    model_2.eval()
    with torch.no_grad():
        tot_loss,tot_acc,tot_err = 0.,0.,0.
        for (xb0, xb1), yb in data.valid_dl: 
            pred = model_2(xb0, xb1)
            tot_loss += loss_func(pred, yb)
            tot_acc  += accuracy (pred,yb)
            tot_err  += error_rate(pred,yb) 
        num_val = len(data.valid_dl)
        print(epoch, tot_loss/num_val, tot_acc/num_val, tot_err/num_val)

Note that I rewrote my model without the batch normalization to make it a bit simpler so it is now defined as follows:

out = 1
layers = 3

class Model_2(nn.Module):
    def __init__(self):
        super().__init__()
        self.l_in = nn.Linear(nv, nh)
        self.l_hidden = nn.GRU(nh, nh, layers, batch_first=True)
        self.l_out_1 = nn.Linear(nh, out)       

    def forward(self, x0, x):
        t = self.l_in(x.unsqueeze(2))
        res, hi = self.l_hidden(t, torch.zeros(layers, t.shape[0], nh))
        squeezed = self.l_out_1(res)
        rel = F.relu(squeezed)
        return rel.squeeze(-1)

I

3

I’ve been working on rnn for tabulat time series, too! Great job!
I’m confused a little bit here, where does x0 go in your forward pass?

1 Like
4

Hello! The x0 was a bit of a hack because my data is being passed in as a tuple and I didn’t need the first term. I actually ended up figuring this out yesterday. I needed to return only rel[:,-1,:] since for classification I am only interested in the last term of the sequence.

I am predicting reasonably well now, however, ‘learn.summary()’ still produces an error.

5

Any update to that summary problem? I’m facing the same.