TimeSeries

fastai fastai dev
101

You might check out this video tutorial on how to create an arrf file. You can ignore the second part of the tutorial about WEKA:

Creating ARFF Files for Weka

102

Does anyone has experience with seq2seq models for timeseries forecasting?
Have been trying some basic models like the one from Rachel’s exposed here with good success, but would love to hear on tricks/ideas about encoder/decoder architectures for this type of tasks. I am using:

  • GRU encode/decoder
  • Teacher forcing for training (with a nice Callback)
  • Basic Attention mechanics only on the decoder.

Here it is the seq2seq model from Rachel’s class modified for multivariate timeseries forecasting.

  • input: Multivariate
  • output: Uni (can easily be modified for multivariate output).
class Seq2SeqRNN(nn.Module):
    def __init__(self, enc_inp_sz, dec_inp_sz, nh, out_sl, nl=2, bidir=False, debug=False):
        super().__init__()
        self.nl,self.nh,self.out_sl = nl,nh,out_sl
        self.debug = debug
        self.enc_drop = nn.Dropout(0.15)
        self.gru_enc = nn.GRU(enc_inp_sz, nh, num_layers=nl, dropout=0.25, batch_first=True, bidirectional=bidir)
        self.out_enc = nn.Linear(2*nh if bidir else nh, nh, bias=False)
        self.gru_dec = nn.GRU(dec_inp_sz, nh, num_layers=nl, dropout=0.1, batch_first=True)
        self.out_drop = nn.Dropout(0.35)
        self.out = nn.Linear(nh, dec_inp_sz)
        self.pr_force = 0.
        self.bidir = bidir

    def encoder(self, bs, inp):
        h = self.initHidden(bs)
        enc_out, hid = self.gru_enc(inp, 2*h)
        if self.bidir:
            pre_hid = hid.view(2, self.nl, bs, self.nh).permute(1,2,0,3).contiguous()
            hid = pre_hid.view(self.nl, bs, 2*self.nh)
        hid = self.out_enc(hid)
        return enc_out, hid
    
    def decoder(self, dec_inp, h):
        outp, h = self.gru_dec(dec_inp, h)
        outp = self.out(self.out_drop(outp[:,0]))
        return h, outp
        
    def forward(self, inp, targ=None):
        bs, sl, _ = inp.shape
        _, h = self.encoder(bs, inp)
        if self.debug:     print(f'h:    {h.shape}')
        dec_inp = targ[:, [0], :] if targ.shape[1]>1 else targ
        if self.debug:     print(f'dec_inp:{dec_inp.shape}')
        res = [dec_inp]
        for i in range(self.out_sl-1):
            h, outp = self.decoder(dec_inp, h)
            if self.debug: print(f'outp: {outp.shape}')
            
            #teacher forcing
            if (targ is not None) and (random.random()<self.pr_force):
                dec_inp = targ[:,[i],:]
            else: 
                dec_inp = outp.unsqueeze(1)
            res.append(outp.unsqueeze(1))
        return torch.cat(res, dim=1)
    
    def initHidden(self, bs): return one_param(self).new_zeros((2 if self.bidir else 1)*self.nl, bs, self.nh)

and the corresponding TeacherForcing callback for fastai2:

class TeacherForcing(Callback):
    def __init__(self, end_epoch):
        self.end_epoch = end_epoch
    def begin_batch(self):
        self.learn.xb = self.learn.xb + self.learn.yb
    def begin_epoch(self):
        self.learn.model.pr = 1 - self.learn.epoch/self.end_epoch
    def begin_validate(self):
        "force forecasting"
        self.learn.model.pr = 0

you have to feed data as (x,y) with x is the input timeseries to instant T and y is the target timeseries from T to T+k, where k is the out_sl param on the model.
The Callback will add the y value to the xb param to be able to do teacher forcing.

1 Like
103

I just saw this in Arxiv, for folks working in Time Series Regression:
UCR Time Series Regression Archive

4 Likes
104

Nice, we have a benchmark to beat!

1 Like
105

Great finding Victor. Thanks for sharing!

106

Here’s a new paper from the team of Hassan Ismail Fawaz (author of InceptionTime), showing the application of Neural Architecture Search (NAS) for Time Series Classification:

6 Likes
107

Thank you so much for your help! Our team solved the problem by the following codes:

fo = open(“drive/My Drive/file.arff”, “a+”)
fo.write(arff_head_1)
fo.write(arff_head_data)
fo.write(arff_head_2)
fo.close()
data_content_eg = “‘0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0\n1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8,1.9,2.0’,3\n” # just an example
fo = open(“drive/My Drive/file.arff”, “a+”)
index = 0
for file_i in file_list:
data = pd.read_csv(path + file_i)
# Select Features
data = data[[‘Time’,‘X1’,‘X2’,‘X3’,‘outcome’]]
data[‘outcome’] = data[‘outcome’].astype(np.int64)
data_content = “’”
for ch_i in range(channel_dim):
data_ch = data.iloc[:, (ch_i+1)].fillna(0)
data_value = data_ch.values.reshape(1, data_ch.shape[0])
data_fill = np.zeros((1, (len_max-data_ch.shape[0])))
data_value = np.hstack((data_value, data_fill))
for data_i in range(len(data_value[0])):
if(data_i == (len(data_value[0])-1)):
data_content = data_content + str(data_value[0][data_i])
else:
data_content = data_content + str(data_value[0][data_i]) + “,”
if(ch_i == (channel_dim-1)):
data_content = data_content
else:
data_content = data_content + “\n”
fo.write(data_content)
fo.write("’," + str(patient_id[index]) + “\n”)
index = index + 1
fo.close()

We can use it to generate a file with multiple channels. Thank you!

108

Thank you so much!

1 Like
109

dear @tcapelle ,

I’m actually trying to tackle a power consumption forecasting using the seq2seq models. The modifications of Text forecasting Rachel’s model you provided seems very promising, but since I have no experience in forecasting tasks, can you provide the full script showcasing other functionalities in the code such as training, and maybe the extension you used for loading the data for such a problem…
Thank you in advance…

110
  • To ingest the data: I use a windowed dataset approach, reading a dataframe and creating x,y examples by moving a window over it.
  • To train: I use on_cycle, with a Learner with the TeacherForcing callback I posted here.

My processing classes will not work for you, they are very specific for my problem and type of data, you can easily build your own or reuse some of the code that is available around here.
Good luck!

111

New benchmark paper for multivariate time series classification:

It seems to me that authors are missing some algorithms like InceptionTime or ROCKET though.

3 Likes
112

I asked in another channel, but want to hear the opinion from you guys as well. These days transformer has been influential for NLP tasks and sequential data. Do you guys have any thought on its application for time-series applications, i.e. classification and forecast? Thank you.

113

There is an implementation of the transformer model applied to time series, I haven’t tried it though.

Best!

1 Like
114

There is also the Temporal Fusion Transformers
for Interpretable Multi-horizon Time Series Forecasting that appear to give excelent results.
There is a pytorch implementation over here. Let me know if you try them.

1 Like
116

Thanks all for all these info and resources.

I am wondering if any of the very good time series module described above can handle mixed sequence data? The same way fastai v2 Tabular module can handle continuous and categorical (using an extra embedding layer), I wondering if a sequence where some features are continuous and some others are categorical can be digested by any of the available fastai v2 extension packages for time series?

Thanks very much for sharing all this. It is very useful for new joiners.

1 Like
117

Does anyone know if there is a maintained timeseries package for fastai now that fastai2 has been made generally available as fastai version 2.0 ? There were a number of great packages that came out of the earlier timeseries study group, but none seem to have recent commits and all appear to have dependency on the fastai2 development release. My particular interest is in timeseries classification.

1 Like
118

Hey, I haven’t had time to update the repo to V2.
I recently did, check if everything is working and let me know please.
https://github.com/tcapelle/timeseries_fastai (last update Sept 18

3 Likes
119

Don’t think so, but would not be to hard to put some embedding layers, I may try that soon.

1 Like
120

Sounds good. Let me know if I can help and I will keep an eye on your repo.

2 Likes
121

Hi @oguiza! Will you update your repo https://github.com/timeseriesAI/timeseriesAI to V2 as well? I am trying to implement this paper on Shapelet discovery with fastai and your library is perfect to experiment with time series data. It would be good to commit a Shapelet discovery notebook to your repo as well!