Time series/ sequential data study group

Hi Jerron,
you can try something like this to create the 3d array:

samples=np.ones((10000,30))
step=50
arr=[]
for i in range(0,samples.shape[0],step):
__arr.append(np.transpose(samples[i:i+step-1]))
arr=np.array(arr)
arr.shape
(200, 30, 50)

Thanks for your reply.
So We do need to split the original big chunk into disjoint smaller pieces to feed timeseriesAI? Will overlapped pieces also work and provide more samples to train or will it do harm?
I saw you use step=50. Is there particular reason or it is just some random number handy?

I think it depends on the type of data, usually the samples have to be independent.

I used 50 because it divides 10,000 samples well

1 Like

Hi @jerron,
You are raising some interesting questions about data preparation, but
Iā€™d say itā€™s impossible to answer them a priori. It really depends on your use case.
When using a sliding window approach to split your single time series and create samples (chunks), there are 2 main hyperparameters: window length (how much history you need) and stride (how much is the window moved forward to generate each sample).

Here are a couple things I think would be important to consider when selecting those hyperparameters. Letā€™s say you have a single TS of length 180k steps gathered during 1h (50 steps per second).

  • How often will you need to predict once the model is trained? Will you need a new pred every second, ever 5 seconds? This will be helpful to determine your stride. If you need a pred every second you may select 50 as stride. If every 5 seconds 250. This may be determined by how often the predicted value may change (like class in a classification task).

  • How much history do you need to make a good prediction? This will depend on your use case, but window length may be larger than your stride to allow the net to learn certain patterns. This will require some testing. It may be useful to test quickly with a small sample size.

Those 2 hyperparameters will determine whether there is overlap or not. Itā€™s common to have overlap. In our previous use case, we may need a new pred every second, but may need 10 seconds of historical data to generate good predictions.

2 Likes

Hi @marteen,

This is an approach Iā€™ve seen used quite often when you have very long time series. But as you say, with this approach you may be missing relevant information.
What some people do is to calculate several rolling features to generate smaller multivariate time series. (This might be done in addition to the subsample method you are applying).
For example, you could calculate for each of the 300 samples rolling features every 2000 time steps (like mean, std, skewness, kurtosis, max, min, number of crosses above mean, ā€¦. - there are hundreds of features you could use). This would give 300 samples of multivariate time series of length 200 with each feature coming from a subsegment of length 2000.
There are packages like tsfresh or cesium that may be useful for this purpose.

3 Likes

Hi All,
Iā€™m interested to explore DL Temporal CNN model for forecasting time series dataset. Pls share some example coding using rstudio environment.

New here, following the course, but have a time series related question.

I have a multivariate time series problem, that Iā€™d like to use a CNN for. Iā€™ve done some experimentation, but the experimentation does not jive with what Iā€™ve seen being done/talked about, so I thought Iā€™d post and ask for some help or insight.

  1. Method by which we convert time series into images. Now, Iā€™ve seen and somewhat understand what Gramian Angular Fields (summation and difference) do. However, in terms of a multivariate time series (15 vars), does this still apply? If the answer is yes, is it only the limitation of resnet34/50 alone that it requires 3 channel images? Would I have to modify the input architecture of resnet to achieve analysis on a 15 channel image? EDIT. Saw the part about recurrence plots, and joint recurrence plots for multivariate time series. I donā€™t understand what it tries to accomplish, so Iā€™ll do some reading, but perhaps that might be a good thing to try.

  2. Because of my above question, I instead created this:
    a01p1s01
    This is an image that is 15x125x1. 15 variables, with there values normalized into grayscale (1 channel), over the 125 samples that occurred. It is rectangular. Itā€™s small. Itā€™s grayscale. It doesnā€™t do any of the complicated rendering a GASF, GADF, a MTF does, which I guess all try to respect time dependence. It easily captures more variables, and I think has no information loss (this I donā€™t know how to quantify, only intuition, aside from floating point to integer conversion for pixels). And yet, when I plug this into resnet34, I can easily achieve 98.9% accuracy. Thatā€™s better than the paper I got the data from (mind you, the paper was just comparing different machine learning models in two open source pieces of software, and top was with an ANN or SVM, didnā€™t try CNN). So, my question is, was this a proper way to transform a multivariate time series into an image?

  3. Is a grayscale, rectangular image a proper input to resnet34/50 ? I think I heard Jeremy say rectangles do well too (ā€œjust try itā€, to quote him). But then I also recall that resnet34 was trained on colored images, so colored image inputs are better.

Dear Ignacio and TSC community,
I just finished reading most of the 647 comments of the thread :slight_smile:
Iā€™m really impressed by the work you have doneā€¦
I started to work on time series classification more than 1 year ago where almost no available code was existing and had to stop for a while.
I just recover this subject a couple of weeks ago and thatā€™s amazing to see how fast you progressed.
Iā€™ve tested on Google colab the 2 main algorithms that you implemented on 2 different datasets for binary classification GOOD / BAD (one dataset is easy and another one more difficult):
. Inceptiontime give good results after 5 minuts of training as expected: I had already classiffied this dataset considering image as input with a standard Resnet of Conv2D with also good result but much slower traing time (~ 1 hour)
. Rocket algorithm give even better results specially on the most difficult dataset after less than 1 minut of training. I couldā€™t believe it !!!
Just one point on the second part of the Rocket notebbok: to allow GPU (thus fast speed) training on google colab, I had to change the configuration of XGBoost: the ā€œtree_methodā€ parameter has been changed from ā€œautoā€ to ā€œgpu_histā€.
I donā€™t know if it can be related to a local configuration of my laptop ?

I read the Rocket paper (thank you Angus). The idea of the feature extraction is really innovative: instead of dealing with sequentiel layers and trained weight to extract features, I wouldnā€™t have bet that taking a big amount of untrained random Convolutions could give such results.

Once again thanks a lot, it saved me a lot of timeā€¦ and it works !!!

I still have to test Rocket with a the multivariate dataset :slight_smile:

2 Likes

Thanks a lot @laurent! I agree a lot of participants have done a great job!

Yes, itā€™s pretty amazing what @angusde and his team built. I couldnā€™t believe it either :grinning:

I think itā€™s not an issue with you laptop. You are probably right that it may be better to change that hyperparameter to speed up XGBoost.

Great! Please, let us know how it works!

PS. Are you using timeseriesAI (based on fastai v2)? Iā€™ve already added a few tutorial nbs and have plans to add a few more in the next few weeks.

1 Like

yes, Iā€™m using timeseriesAI based on fastaiV2.
I used on google colab the notebook ā€œ02_ROCKET_a_new_SOTA_classifier.ipynbā€ and ā€œ01_Intro_to_Time_Series_Classification.ipynbā€ with my own dataset.

Just one question: I didnā€™t see the notebook related to data augmentation for time series on the ā€œnbsā€ folder. Has it been removed ? it is something that I would have liked to test with InceptionTime algorithmā€¦

Thanks again to the whole community :slight_smile:

1 Like

While not time-series exactly, it is tabular specific. For those interested Iā€™m trying to get fastai2 to support more frameworks such as Numpy, cuPY, Dask, etc. For those wanting to help, I have made a fastai2_tabular_hybrid repository with various todo lists :slight_smile: Any help from folks familiar with the libraries is absolutely valued :slight_smile:

4 Likes

I found an interesting 2020 BCI competition led by IEEE in EEG classification containing 5 different datasets including: motor learning tasks, microsleep detection, speech classification, limb movements, signal distortion problem. This seems to be a set of very relevant time series medical classification problems that you might find interesting.

Iā€™m personally a newb in deep learning, but Iā€™m sharing this for people who want to learn more, compete, or collaborate.

http://brain.korea.ac.kr/bci2021/competition.php

2 Likes

Hi all,

Probably such question has been asked before at this forum, but I couldnā€™t find the answer straight away. Therefore I thought to just give it a try, especially since my thesis deadline is approaching while not being satisfied with my results yet.

Iā€™m working on a time-series forecasting problem, in which we are considering climatological data. We have a large amount of time series (~100k). The data has been cleaned (no nans, same amount of observations per year, etc) and some features have been added. Iā€™ve performed some benchmarks using linear regression & ARIMA.

Currently my time-series data is formatted as one time-step per row, while each column is a different location. The other features are still in different table with all the metadata.

Now I hope to take it to the next step by implementing a hybrid forecasting model, which would not make a local (like LR and ARIMA), but global model. First step would (I guess) be to start using a LSTM. However, Iā€™m struggling to find the right design principles, as most of the examples I can find are performing forecasts on one time series (whereas I have many).

Do you know if there is a FastAI example which is working for many time series? If so, in which lecture can I find that one?

How would the input data of my model be? Especially considering the fact that I would like to add other features.

2 Likes

There are not many implementations of forecasting time series in fastai afaik. You can get some ideas from @takotabā€™s fastseq for univariate time series through the N-BEATS architecture. You can also take ideas from the dataloader there, adapt it for multiple time series and fit an LSTM to it.

3 Likes

@sirolf, to complement @vrodriguezf answer, you might also check out Amazon Labsā€™ time series forecasting repo called GluonTS. They have many models that cover multivariate time series forecasting. You might find some implementations that you might adapt to your use-case.

GluonTS uses Amazon MXNet (instead of Pytorch or TensorFlow). They implemented many state-of-the-art architectures ( DeepFactor, DeepAR, DeepState, GP Forecaster, GP Var, LST Net, N-BEATS, NPTS, Prophet, R Forecast, seq2seq, Simple FeedForward, Transformer, Trivial, and WaveNet). Many of them (DeepFactor, DeepAR, DeepState) also use categorical data (covariate variables) and use probabilistic forecasting

4 Likes

Dear all,

I am interested in investigating the use of autoencoders with timeseries data. So I was wondering if something similar to the work done in here (article) could be possible to achieve (as in set both the input and the output data of the databunch to be timeseries of similar or even different lengthā€¦)

thank you!

Hi all,

Iā€™ve been reading a lot about ROCKET here. Iā€™ve tried running it with the pytorch implementation of @oguiza (link below), but itā€™s extremely slowā€¦running a sample sized [60,600,300] takes nearly 30sec (GPU running on googles colab), yet when I test the same sample with InceptionTime it takes less than 1sec.

I thought maybe the authors usage of ā€˜torch.catā€™ within the for-loop is blocking the cuda stream, but I couldnā€™t get around that, on the contrary, adding buffers for intermediate values just slowed it down. Any idea what might be wrong? Maybe the large ā€œwidthā€ of the net (10000 concurrent cuda kernels on the same data) makes it an unsuited for GPU?

@sirolf

gluon-ts allows you to run multiple data against multiple state of the art time series algorithms.
Some of the algoā€™s deliver staggering results due to a hybrid ARIMA / RNN approach pioneered and implemented only over the past three years or so. Benchmarking is absurdly trivial once your datasets are correctly described.

https://gluon-ts.mxnet.io/examples/extended_forecasting_tutorial/extended_tutorial.html

The initial effort is worth doing it because Gluon ships the most advanced and most effective time-series forcasting models. I once had a pretty ugly TS problem I could not solve with any other lib, but that model from MIT implemented in Gluon got me some 3% MEPA, which was unheard of before.

1 Like

Iā€™ve tested N-BEATS extensively. I canā€™t get it to outperform a simple ResNet-style architecture on any of the time-series datasets Iā€™ve so far tried it on. Has anyone else had this problem?

By the way, the authors of the N-BEATS paper have finally released their code which you can see here: https://github.com/ElementAI/N-BEATS

Can someone rewrite this with fastai: https://github.com/smallGum/MLCNN-Multivariate-Time-Series