TimeSeries

fastai fastai dev
81

Thank you for the answer. What is not said is that I want to decouple encoder from decoder because input and output have different length (not a problem per se) and I have some prior info about the future that I want to append to the output of the encoder. Otherwise, yes, lstm_dec(lstm_enc(input)) would do the trick just fine. Also, encoder will be a CNN or other fancy architectures.

82

take a look at this repo it uses a CNN to encode the input, and an lstm layer to find the temporal relation.
My problem posted above about the sun forecasting does exactly this, concatenating some prior info about the sun to the lstm layer, after the encoding
I you need more info, or want to work together, I am very much interested on this type of encoder/stm/decoder problem right now.
I have a basic LSTM wrapper to take care of the hidden state and dropout:

class LSTM(Module):
    def __init__(self, input_dim, n_hidden, n_layers, bidirectional=False, p=0.2):
        self.rnn = WeightDropout(nn.LSTM(input_dim, n_hidden, n_layers, batch_first=True, bidirectional=bidirectional), p)
        self.h = None

    def reset(self):
        self.h = None

    def forward(self, x):
        raw, h = self.rnn(x, self.h)
        self.h = [h_.detach() for h_ in h]
        return raw, h

then use a learner with the Reset Callback. The main model is like this:

class BasicModel(Module):
  def __init__(self, cnn_encoder, n_features, n_hidden, n_lstm_layers):
    self.encoder = cnn_encoder
    self.n_hidden = n_hidden
    self.lstm = LSTM(512, n_hidden, n_lstm_layers, batch_first=True)
    self.head = nn.Linear(n_hidden+n_features, 1)

  def forward(self, x, features):
    "x are the images, features are the timeseries"
    x = torch.stack(x, dim=1)  #stack images together, to form a sequence o images (bs, seq_len, 3, h, w)
    x = self.encoder(x)  #rencode images with a resnet, cut after pool. returns a (bs, seq_len, 512) tensor
    x, _ = self.lstm(torch.cat(x, features.permute(0,2,1), dim=-1))
    return self.head(x)

  def reset(self): self.lstm.reset()

there are some reshapes missing, but you get the idea.

2 Likes
83

There is a lot of data available:

  • The Swimcat datasets containt various types of sky images, for classification and segmentation task
  • The WSISEG dataset (we have been using this one) with whole sky segmentation masks.
  • NREL has a lot of public data from yhear 2010 accompanied with weather stations mesures.
84

Thank you! It is a really interesting approach. There are some things that are not familiar to me yet (I’m still a noob :)), specially the cnn_encoder stuff.

I had a look at you action recognition repo. One question: How do you pass from a problem with one single output (the class of the action) to a problem with a forecast of 10-20 future points in a time series?

85

We can make a call with Ignacio to discuss the details, it is very straightforward.

  • The ccn_encoder is just a resnet created with create_model
  • The time series output is free from the lstm. It will output the same number of outputs as inputs.
86

Hi everyone!

I have been wondering if there is a way to use transfer learning for time series data. Similar to pre-training a language model and fine-tuning a text classifier, couldn’t we use iterative time series forecasting as a self-supervised pre-training method before fine-tuning e.g. a classifier on a fixed window length? This could be useful if large amounts of unlabelled time series data are available, but labelled examples are scarce.

Does anyone have thoughts on this or has maybe even tried it out?

1 Like
87

Hi @stefan-ai,

I think this is definetely a great idea, and worth exploring :grinning:.
I’d be very interested to learn if you make any progress in this area.

Personally I haven’t used any self-supervised time series implementation yet, but I think the approach should work.

A new paper published this month addresses this topic:
Jawed, S., Grabocka, J., & Schmidt-Thieme, L. (2020, May). Self-supervised Learning for Semi-supervised Time Series Classification. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 499-511). Springer, Cham.
It’s freely available online.
It’d be great to see a fastai implementation of this approach!

PS: @Epoching wrote an excellent blog post on how to apply this type training to images using fastai.
Another great blog post that describes SSL for images using fastai has been written by @JoshVarty.
Both be useful if you or someone else pursue this SSL approach.

3 Likes
88

Thanks a lot @oguiza for pointing me to these resources. If I get anywhere I will share my results here for sure.

I have been mostly thinking about an RNN based approach similar to ULMFit. But if I remember correctly, I read in one of your earlier posts that turning time series into images and using CNNs has worked better for you, right?

1 Like
89

I’m definitely interested in how this turns out! Keep us updated :slight_smile:.

90

@hfawaz has also an excellent paper about transfer learning for time series classification. If I remember correctly, the conclusions highlighted the importance of the similarity between datasets when transferring knowledge from one to another.

I wonder what would be the intuition of a model trained with massive amounts of time series data in a self-supervised fashion. In the same way that the first layers of a pre-trained model for images recognize basic components of an image, such as corners or gradients, would there be an equivalent to think of for time series? Unlike images and text, two time series from different domains do not necessarily need to share anything apart from having a time axis.

4 Likes
91

Thanks for the info! That is what I did, +/- some variants. However, I moved a bit away from LSTM for now. A bunch of CNNs (timewise) + some linear layers on top, work better. I get the feeling that there is more to LSTM so later I will try to do some attention like mechanism where to inject the prior future. Also, there is Oguiza’s ResNet and Inception adaptations to time series. Hmm, lots of directions and so little time . . .

92

In relation to the talk about transfer learning and time series, I just saw this paper accepted on KDD 2020: Multi-Source Deep Domain Adaptation with Weak Supervision for Time-Series Sensor Data

Paper: https://arxiv.org/abs/2005.10996
Code: https://github.com/floft/codats

4 Likes
93

Yeah, that’s really an intriguing idea. It’d be great to have model trained across multiple TS datasets, that can be then fine-tuned!

1 Like
94

Thanks for your reply @vrodriguezf. I will check out that paper as well.

I mostly had the situation in mind where you have a lot of unlabelled time series data and some labelled examples from the same data source. I believe this could already be very helpful for many practical projects. The question is just how much unlabelled data you need to make this work and if the features learned from forecasting are useful for downstream classification tasks.

When it comes to a general time series model trained on large amounts of data from different domains, it’s also not clear to me what would be the underlying learned features and if they are useful for transferring to other domains. But for sure it would be very powerful if it works.

95

I have a question, how are you storing timeseries metadata on your libraries? more precisely I am interested on storing a TimeIndex to be shown with the timeseries on the show_batch and show_results methods, instead of plain integers.

96

That’s a good question!
I don’t use metadata in my timeseries, so have never had this issue. Anyways, here’s my 2 cents:

  • I know this is not ideal, but if you want your metadata to be sliced with your data, could you store your metadata as another feature/channel? I don’t know if they could be stored with the same type as your ts data. You would need to add a value per sequence step though, and then apply a mask to select the channels you need.
  • Alternatively, you could store them in a pandas dataframeusing your time index as index, and each feature/ channel as a pd.Series in a column (like they do in sktime).
1 Like
97

Well, just to be clear, I’ve achieved very good results creating images from univariate time series. However, the process is pretty slow, and I would not recommend it for large datasets, or time series with many features/ sequence steps.
In general, I tend to use 1d CNNs architectures based on InceptionTime or ResNet (you can find both in timeseriesAI if you are interested) with raw data as input.
CNNs reached SOTA level on some datasets and, in my experience, work much better than RNNs for both univariate and multivariate time series classification.

3 Likes
98

I’ve come across Vaex, and thought it might be something you can use if you decide to go with a pandas like solution (which quite frankly may be the only way to store multiple dtypes together):

Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. It calculates statistics such as mean, sum, count, standard deviation etc, on an N-dimensional grid for more than a billion ( 10^9 ) samples/rows per second . Visualization is done using histograms , density plots and 3d volume rendering , allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).

It reminds me of np.memmap, but with pd dataframes.

1 Like
99

Thank you for all the detailed introduction! However, I am not sure how to generate a file like “NATOPS_TRAIN.arff”. I know how to convert my data (in csv) to the file like “NATOPSDimension1_Train.arff”, which contains only one dimension. How can I combine different dimensions together? Thank you!

100

Hi @DogarXu,

Why would you generate an arff file? I’ve never generated one myself. This is a file format the UCR time series classification archive happens to use, but there’s no need to use it.
You can just create X and y as numpy arrays. You can see an example of this here.
Please, let me know if you have any further issues/ questions.