Time series/ sequential data study group

Part 1 (2019)
836

thanks :slight_smile: I really like tsai!

I donā€™t know if I shared this before. a while ago I did a small PR for fastaiā€™s xresnets to support Conv1d and Conv3d - so you can easily use xresnets with tsai :). I did some testing with different datasets and xresnets actually worked pretty well on the standard time series datasets (on par with inception time - sometimes even better).

model = xresnet34(ndim=1, c_in=dls.vars, n_out=dls.c, ks=3)

ndim: dimensions of the input (1=Conv1d, 2=Conv2d, 3=Conv3d)
ks: kernelsize - default 3, depending on the dataset I used kernel sizes up to 31

drop in replacement for InceptionTime:

dsid = 'NATOPS' 
X, y, splits = get_UCR_data(dsid, return_split=False)
tfms  = [None, [Categorize()]]
dsets = TSDatasets(X, y, tfms=tfms, splits=splits, inplace=True)
dls   = TSDataLoaders.from_dsets(dsets.train, dsets.valid, bs=[64, 128], batch_tfms=[TSStandardize()], num_workers=0)
model = xresnet34(ndim=1, c_in=dls.vars, n_out=dls.c)
learn = Learner(dls, model, metrics=accuracy)
learn.fit_one_cycle(25, lr_max=1e-2)
learn.plot_metrics()

For the NATOPS dataset the best result I got was 96% with xresnet34, ks=3 and lr=1e-2 (but thereā€™s quite a bit of variance). Maybe thatā€™s interesting for some of you.

2 Likes
837

In time series, we typically have longer sequences than in NLP.
Key, query and value computations can be expensive for long sequences.
This paper presents an interesting way to efficiently compute them.

Anyone had a look yet?
I would like to try this with TSAI. What will be a good way to do so?

Informer: Beyond Efficient Transformer for Long Sequence
Time-Series Forecasting

Pytorch Implementations

4 Likes
838

Hi @dnth,

Sorry for the late reply.

If you want to add a model to tsai youā€™ll need a few things:

  • Have access to a Pytorch version with an appropriate license
  • All Ā“tsaiĀ“ models have the same API:
    • Input is a 3d numpy array that is transformed into a 3d tensor of shape [batch_size x vars x time steps]. All models have the following arguments: c_in (#vars) and c_out (number of output nodes in the last linear layer). Some of them also have seq_len (for # time steps when required) and d (for shape of the output when itā€™s 1d or 2d).
    • Output may be 0-d, 1-d or 2-d per sample.
      Itā€™d be good if you can create an example where a time series with format [bs x vars x time steps] is passed to the model and an output is produced.
  • When this is available, Iā€™d need to review the other args and kwargs to ensure they are aligned to tsai naming used in other models (for maximum consistency).
  • Some additional work is required to make sure convenience functions like build_ts_model work (but thatā€™s the easy part).

Iā€™m very supportive of adding new, state-of-the-art time series models to tsai. I agree that models like Informer or Temporal Fusion Transformers would be great additions to the library.

Please, let me know if you decide to move forward with Informer and have any questions.

2 Likes
839

FYI, weā€™ve just added a new tutorial notebook to tsai. It has been jointly developed by Malcolm McLean (@Pomo) and myself. It demonstrates how to use MiniRocket in tsai.

MiniRocket is a new method developed by Angus Dempster et al (@angusde - paper) that has demonstrated state-of-the-art performance in Time Series Classification and Time Series Regression while being significantly faster than other methods.

The original MiniRocket code (cpu implementation in Python with numba acceleration) was released by the authors a few months ago. What Malcolm and I have done is to create a Pytorch implementation with gpu support that is even faster (3-25 x faster depending on your gpu). It also allows more flexibility as youā€™ll be able to use some common DL techniques (different preprocessing, data augmentation, etc).

Itā€™s a great addition to our toolbox. I really encourage you to take a look at the notebook and start using it.

Iā€™d really like to thank Malcolm for all the work heā€™s done to make this possible, as well as Angus for creating Rocket and MiniRocket, and for reviewing our implementation.

Note: The Rockets (Rocket, MiniRocket and MultiRocket) are such great models that Rocket has been added to HIVE-COTE and have further advanced the state-of-the-art in Time Series Classification. A new paper describing an updated HIVE-COTE v2.0 was released yesterday. However, HIVE-COTE is very slow and not scalable, while the new MiniRocket Pytorch implementation is very fast and scalable.

9 Likes
840

Great work :clap: :clap:
Cheers mrfabulous1 :smiley: :smiley:

841

My sincere thanks to Malcolm/@Pomo and Ignacio/@oguiza for their interest, and their brilliant work making the method more usable and accessible (and faster). As Ignacio mentioned, this also opens up the method to be used with some of the many rich features in the tsai library (eg, data augmentation).

While I did ā€˜review [the] implementationā€™, in reality this is massively overstating my contribution as there was really nothing to review, the work is excellent and stands completely on its own, and both Ignacio and Malcolm are far more capable than I am in the first place.

3 Likes
842

Maybe Iā€™m just too much of a beginner, but I am having way too much trouble getting started with tsai. I have so far found it easier to learn Python from scratch, build a json to image converter and run my time series as an image classification problem in fastai, than it is to run one line of tsai code.

For example:
In [5]:

dataset id

dsid = ā€˜NATOPSā€™
X, y, splits = get_UCR_data(dsid, return_split=False)
X.shape, y.shape, splits

What is X and y? This isnā€™t explained anywhere. Iā€™ve looked at the UCR data sets and none of that makes any set how the data is mapped to x and y, whatever they are.

Whereas in fastai, they explain it so that anyone can walk in off the street and start using deep learning, I find my only recourse with tsai is asking for an interpretation on this forum!

Again, Iā€™m a beginner, but Iā€™m having such a hard time transition from fastai to tsai that itā€™s preventing me from using your library.

843

X and y are numpy arrays. The dimensions of X are (#samples, #vars, #timesteps). This tutorial notebook can help you out!

1 Like
844

Unfortunately, this isnā€™t a great explanation and to be honest, I have used this library to get good results, but I would be able to do more if I understood how to take a data set and change it to a feature set that UCR is formatted.

Itā€™s really mind-numbing and is a real reason I stay away from time series because I just havenā€™t found a great resource to take for example the M5 Walmart data set and run it through this project would be awesome as itā€™s an interesting regression problem.

I am thankful all the work this community does, but I figured I would defend that statement as itā€™s something I am trying to figure out myself. I am working on a wait time multivariate lift line time series and getting good results, but probably out of dumb luck than anything else. I am also only getting one of the 12 lifts predicted at a time.

Again thank you for all the work, but I think as a newbie to time series that notebook is really confusing and hard to follow

845

I also struggled with this initially. Perhaps @Jdemlow @duality you find these discussions here: https://github.com/timeseriesAI/tsai/issues/25 and https://github.com/timeseriesAI/tsai/issues/85 useful for using TSAI with custom datasets/tasks when getting started.

846

I will take a look at these thanks a lot and I will edit this to say if it was super helpful :slight_smile:

I think the discussions will be something that I will use to try and learn more from. It is interesting that the library is focused on modeling only and the pre-processing isnā€™t ignored, but has a realization that there are so many different types of preprocessing pieces itā€™s to many to actually have tutorials on.

Maybe a good solution is that we all post examples on this thread and we help everyone else that comes later on see how that works so that we have more people using this magical library.

If I can solve my problem I will post the solution here reenacted of course

My problem right now is

Many different venues with many different lifts have varying wait times.

Our POC example we have used VAR and a deep LSTM model getting around 1.30 mae and the first stab with TSAI was 1.15-2.00 mae depending on the lift.

What ideally I would love to do is something that https://github.com/timeseriesAI/tsai/issues/85 talked about stacking the data sets per lifts as ā€œsession/windowsā€ and having the model take these into account. This problem is interesting because each lift has a correlation to the other.

I will flesh out the objective once I get a better example going.

847

Iā€™m an open source software maintainer myself, so I very much know the struggles of both creating complex projects and dealing with users. So above all, the first thing that has to be said is: itā€™s impossible to please everybody all the time. There are always things that can be improved.

Iā€™ve found the tsai project to be both useful and inspiring. It brings together an unfathomable amount of cutting edge technology, wisdom and experience. The fact that it is free to the world is extraordinary.

Iā€™m not so much of a newbie yet I still very much sympathise with the comments here about the projectā€™s inaccessibility. As always, there are some clear areas for improvement, for example documentation formatting, global imports, tests and types. Attending to those would certainly make things easier for newcomers (even experienced devs become ā€œnewbiesā€ when returning after an absence), but even then itā€™s not going to overcome the inherent difficulties in grappling with the fundamental complexities of this field.

Whatā€™s abundantly clear, is that the spirit of this project is generous and welcoming, which more than makes up for the strains of climbing the learning curve. So whilst it would certainly be nice to see some ā€œUXā€ improvements, Iā€™d never want that to overshadow the gratitude that all this wonderful work is made so freely available :star2:

3 Likes
848

Couldnā€™t agree more with the sentiment this is amazing and has huge unlock potential and is on the bleeding edge of everything and I have a love for this community and there is nothing but gratitude and I have been following this forum just to read it because it is so cool.

Fastai has always been about getting people in the door and going wild with helping and adding or coming up with ideas. Jeremy talks about how huge advancement in DL doesnā€™t always come from PHDs or software developers. Time series barrier for myself and I am sure others is getting a data set to the np.arrays in this library in a way that makes sense. The structure of data and the sequence ordering is so important, but to be make strides getting that understanding is crucial i think maybe I am wrong there. I have always found once I get the underlying structure of the data the rest falls into place.

to be clear this wasnā€™t a complaint this was a sorrow that I wish I could use the library and UCR data sets are extremely odd that takes nothing away from that fantastic work

849

Thank you for the replies everyone. I get a sense of the high levels of support offered here. I wish I could share the sentiment about the library but I canā€™t get started so I have nothing to share, though Iā€™m sure I would be grateful if I did!

So now Iā€™m tossing up whether to persist with image classification, drag myself over the coals and learn tsai, or learn the more time-tested and thoroughly documented methods like random forests.

Which one would be the quickest to learn from here? As stated above, I am fluent (not expert) in python, and have custom datasets that can be rebuilt as necessary for any implementation.

850

@duality if you are starting out I would take a look at the Rossmann sales competition, Jeromy has talked about it in several courses. He uses the entity embeddings method which is very well documented in lots of blog posts.

rossman_data_clean.ipynb
and
lesson6-rossmann.ipynb

@jdemlow
If you want to look at an example of using the M5 competition SidNg is a Fasiai student that did very well on the M5 competition using the entity embedding method.
His fastai post
His blog post
His kaggle kernel

If I ever have time I think it would be a lot of fun to create a study group or videos content that help people learn about Time series forecasting using Deep learning Methods.

4 Likes
851

Thanks for your feedback @duality. Iā€™m sorry to hear this. It wasnā€™t my intent to make it so difficult.

I understand how frustrating it is to try to learn something and find that for whatever reason you canā€™t make substantial progress. Iā€™ve been in that situation (and continue to be) quite often.

Iā€™ll give you a bit of background on tsai. Iā€™ve been working with time series for a few years now. When I started, I tried to apply DL to time series, but couldnā€™t find any library that met my needs. So I decided to create my own library. I did it while I took the fastai course. I always applied everything I learned to time series. Initially, I built it in a private repo. But later, I decided to make it public and share it with the fastai community, just in case anybody else might find it useful. I did this because Iā€™m very grateful for everything Iā€™ve learned here, and wanted to give something back. This is why itā€™s open-source. I share everything I build for my own use.

Unfortunately, I have my family and my own work, and I can only work on tsai in my spare time (which tends to be very limited :frowning: ). Iā€™d love to be able to dedicate more time to tsai because I have lots of ideas Iā€™d like to implement. But quite frankly I canā€™t.

It never intended to build a structured course (like fastai) that anybody with a little background on time series could follow. I have created a few tutorials though to try and demonstrate how the library can be used. But they are very time-consuming.

Data preparation is a particularly complex area. There are many types of data sources, formats, labels, etc. Thatā€™s why I have built some functionality to try to show how the input to tsai needs to look like. But apart from that, data preparation is out of the scope of this library.

Having said this, Iā€™d be more than happy to have contributors to tsai. Actually, the last 2 tutorial nbs have been prepared in conjunction with other forum participants (@williamsdoug and @Pomo).

I know the library can be improved in many different ways. And I appreciate any feedback I get. I always try to improve.

7 Likes
852

Hi @duality. Building on the comments by @vrodriguezf and @oguiza, timeseries analysis can very challenging since it is in a less of a mature state than other machine learning problems such as image, language and tabular data. While fastai offers both high-level (turn-key) and intermediate-level APIs, youā€™ll likely need to use the more complex intermediate-level APIs for timeseries analysis such as the APIs provided by tsai.

To the extent that you are able to transform a timeseries problem into an image problem (e.g. : plot or spectrogram image) and the results are good enough, then that is a fine approach. I believe Jeremey Howard discussed this in one of his lectures and Iā€™ve seen this approach used in some of the Physionet Challenge competitions.

To the extent that image conversion proves insufficient, then youā€™re likely need to deal with more complex intermediate level APIs. tsai includes a very powerful set of models and data transforms, but at the cost of a steeper learning curve. As with many packages still in the early development stage, documentation is sometimes limited. The tutorials can help, particularly if you add your own code to print out sample data (or types) when the documentation is insufficient.

In terms or your specific question What is X and y? X and y are a fairly standard convention used in various machine learning packages such as scikit-learn where:

  • X is the training data
  • y is the label data

Index ordering can vary by package, but is usually obvious if you print or plot the first couple rows or columns.

2 Likes
853

@Daniel.R.Armstrong @oguiza @williamsdoug

Thank you for the replies everyone and I must apologise if I can across as critical of the work you are doing here. I guess I was venting some frustration. I totally understand the amount of effort it takes to put forward something like this for free for the community. Then to have someone come along and complain must seem harsh.

I must admit that I overcame some of my fear by dissecting the UCR examples detail by detail. I found a dataset that is similar to mine and modelled it. I am still stuck, although now i have manually created my X, y and split variables. I just realised that the shape of the X data is back-to-front from what I expected, so will have to rework the whole thing, hopefully itā€™s a simple fix.

I am hoping it is simple because I donā€™t know what Iā€™m doing and if I get an error itā€™s unlikely Iā€™ll be able to fix it easily. But so far everything else seems to be loading ok. If I get a valid prediction tomorrow I will be ridiculously happy.

854

It didnā€™t work. My accuracy never gets any better than 50/50 for a classification task.

How do I troubleshoot this? Where should I go, what should I read?

Thanks

855

Can you share a gist showing what you are doing? Otherwise, itā€™s difficult to help with the information you have provided so far.