Time series/ sequential data study group

oguiza · November 28, 2019, 10:04am

I’ll try to answer your questions as best I can.

I’ve looked for centering to have some more context, but haven’t found it. In any case I think Angus has provided more info on the normalization that needs to be done.

You are right. I just used the show_batch method to visualize the rocket features, but they don’t have any temporal meaning.

No, it’s not my invention . I just picked it from the code the authors have released.

I’m not sure what you mean by this, but if there’s any way to improve the Pytroch implementation please, I’ll be more than happy to do it.

angusde · November 28, 2019, 10:03pm

On the ‘centering’ point: yes, here, this means subtracting the mean of the kernel weights, thus making the mean of the kernel weights zero. In short, it seems to improve accuracy, so that’s what we do. For multivariate, there is more than one way to approach this…

Pomo · November 30, 2019, 2:20am

Thanks for the answers.

The improvement would be to avoid creating a bunch of nn.conv1d’s and their initialized parameters that get thrown away. But the gain would likely be negligible.

Willem · December 4, 2019, 8:49am

Love your work here guys. I’m new to fast.ai with a background in old school econometrics. This topic piqued my interest the most. Hopefully I’ll keep motivated during the course and can start helping out soon.

oguiza · December 4, 2019, 10:45am

It’s great to have you here Willem!
Look forward to reading your posts.

farid · December 11, 2019, 8:15pm

For those interested in financial time series forecasting, here below is a very recent review (Nov. 29th 2019)

Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019

adrian · December 11, 2019, 10:58pm

Thanks @angusde for the link to to variable length time series paper. I have a related problem below-but difference to your groups paper is that I need to predict class as well as location of class within a larger time series.

Does anyone know of datasets for and or papers discussing time series classification with constant length series but variable length labels. Eg daily stock prices at minute time step for 100 stocks with contiguous time sections of the data labelled in say 5 categories where categories may be 5min-1hr in length.

As @hfawaz and co discuss in the ‘Deep Learning for Time Series Classification: a review’ paper it may be possible to create labelled training data in ucr format with labels for constant length series by eg fitting a spline through the data and subsampling to pad each labelled section to equal length but cant do that for the test data.

hfawaz · December 12, 2019, 8:45am

Hi @adrian,

Indeed a linear interpolation is one solution but you risk changing the too much the shape of the time series.
In your specific case, you could divide each series into subseries of 5-min length (equal to the minimum labeled length) ?

sknepal · December 13, 2019, 7:50am

Sorry this maybe a little bit off-topic. But, can anybody recommend the current SOTA approaches or methods that have worked for them for time series imputation? I have multivariate time series streams coming from different wearable sensors that needs imputation. Sort of doing a literature review to see what’s the best approach right now.

Thank you.

oguiza · December 13, 2019, 11:20am

Hi @sknepal,
You may want to take a look at the Multivariate Time Series Imputation section in paperswithcode (link).
There are a bunch of interesting papers (some of the with code).

vrodriguezf · December 13, 2019, 11:58am

You can also take a look at the R package imputeTS, which gathers a lot of popular imputation methods for time series. I am not sure if it has multivariate support.

adrian · December 13, 2019, 2:37pm

Good idea, will benchmark against transformation of the series to images+segmentation.

Will_Riker · December 15, 2019, 7:39pm

Me too. I come from a economics/ econometrics background and the application of ML to economic time series data is super interesting!

Also (mind you I’m relatively new to coding), I have created a library the interfaces with the TD Ameritrade API if anyone is interested. It allows you to to pull real time stock/index data, historical data on a variety of time frames, as well as pull account info and place trades. I predominately wanted it for current and historical index data for ML models, and it seems to do well at that.

sknepal · December 23, 2019, 2:15am

Thank you. I’m looking into it.

Currently working on ways to do time series based analysis (incl. but not limited to classification) based on change models and others.

sachinruk · December 24, 2019, 11:46am

Hope this isn’t off topic but I created profetorch, which is the concepts of fb-prophet applied in pytorch + fastai. https://sachinruk.github.io/ProFeTorch/index.html. Would appreciate any feedback.

novarac23 · December 29, 2019, 5:15pm

Hi friends!

I’ve built a little GRU model using PyTorch that is trying to predict a price of shampoo sales from this very tiny and old shampoo dataset, here’s the link to it: https://raw.githubusercontent.com/jbrownlee/Datasets/master/shampoo.csv

Here’s the model:

class GRU(nn.Module):
    def __init__(self, input_size=1, hidden_size=100, num_of_recurrent_layers=2, output_size=1):
        super().__init__()
        self.gru = nn.GRU(input_size, hidden_size, num_of_recurrent_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
        self.memory_cell = torch.zeros(2, 2, hidden_size)
        
    def forward(self, x):
        res, memory_cell = self.gru(x.view(len(x) ,1, -1), self.memory_cell)
        self.memory_cell = memory_cell.detach()
        result = self.fc(res.view(len(x), -1))
        return result[-1]

Model was inspired by amazing people on the interwebz and I can hardly take all of the credit here

I’ve been playing around with shapes and sizes of inputs and outputs for the last few days and I finally got the model to a place where it’s performing really well.

Now the main thing I don’t understand is that in all of the papers/posts I read that use pytorch to solve univariate problem almost all of them, at the end of the forward method take the last item of the predicted array as a prediction and I can’t figure out why.

I’m doing the same thing with my model and it works great but I don’t understand why. Here’s the rundown of shapes and sizes in forward:

res -> 2x1x100
result -> 2x1

Input to a linear layer is converting res into 2x100 which makes sense but the part that does not make sense is that there are two things returned from gru and that the last one is the one I need. I was expecting 1x100

any ideas?

Any feedback/response much appreciated,

Nik

novarac23 · December 30, 2019, 12:26am

Ok so after a few espressos I think I figured it out. Here’s the pic that helped out a lot:

Basically what’s going on is that res (or output in the pic on the right) in my implementation of a GRU is all of the states from the last layer “depth” wise. memory_cell is states of last layer “time” wise.

So res has a shape of 2x1x100 because I have two inputs and each of them had a state (aka activation) which I then use to put through a linear layer to get an actual prediction. Reason why I need -1 is because I really only care about last item in the output sequence since I’m predicting shampoo sales for next month.

Does this sound right?

Here’s the stack overflow post that helped: https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm

Also here’s the video that cleared up a lot of the confusion https://www.youtube.com/watch?v=Bl6WVj6wQaE (~11:30 is when Rachel starts talking about output vs hidden state)

Pomo · December 31, 2019, 6:00am

Hello Oguiza,

I downloaded your repository timeseriesAI-master a month ago, and installed numba. At that time, “05_ROCKET_a_new_SOTA_classifier” ran perfectly without error. But I just now went back to the same notebook and find an error. As far as I know, I made no changes to the conda environment, but who knows? Can you help?

pytorch: 1.2.0
fastai : 1.0.59
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0]

What I have done:

Downloaded timeseriesAI-master again, with no changes.
Removed numba and reinstalled numba 0.46.0.
Restarted Jupyter.

The error occurs with
kernels = generate_kernels(seq_len, 10000)

The error is quite verbose, but here goes…

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/targets/base.py in get_constant_generic(self, builder, ty, val)
    498         try:
--> 499             impl = self._get_constants.find((ty,))
    500             return impl(self, builder, ty, val)

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/targets/base.py in find(self, sig)
     49         if out is None:
---> 50             out = self._find(sig)
     51             self._cache[sig] = out

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/targets/base.py in _find(self, sig)
     58         else:
---> 59             raise NotImplementedError(self, sig)
     60 

NotImplementedError: (<numba.targets.base.OverloadSelector object at 0x7fd647b34c90>, (reflected list(int64),))

During handling of the above exception, another exception occurred:

NotImplementedError                       Traceback (most recent call last)
~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/errors.py in new_error_context(fmt_, *args, **kwargs)
    716     try:
--> 717         yield
    718     except NumbaError as e:

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/lowering.py in lower_block(self, block)
    259                                    loc=self.loc, errcls_=defaulterrcls):
--> 260                 self.lower_inst(inst)
    261 

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/lowering.py in lower_inst(self, inst)
    302             ty = self.typeof(inst.target.name)
--> 303             val = self.lower_assign(ty, inst)
    304             self.storevar(val, inst.target.name)

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/lowering.py in lower_assign(self, ty, inst)
    482                 const = self.context.get_constant_generic(self.builder, valty,
--> 483                                                           pyval)
    484                 # cast it to the variable type

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/targets/base.py in get_constant_generic(self, builder, ty, val)
    501         except NotImplementedError:
--> 502             raise NotImplementedError("Cannot lower constant of type '%s'" % (ty,))
    503 

NotImplementedError: Cannot lower constant of type 'reflected list(int64)'

During handling of the above exception, another exception occurred:

LoweringError                             Traceback (most recent call last)
<ipython-input-8-4e463a985c18> in <module>
----> 1 kernels = generate_kernels(seq_len, 10000)

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    418                     e.patch_message('\n'.join((str(e).rstrip(), help_msg)))
    419             # ignore the FULL_TRACEBACKS config, this needs reporting!
--> 420             raise e
    421 
    422     def inspect_llvm(self, signature=None):

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    351                 argtypes.append(self.typeof_pyval(a))
    352         try:
--> 353             return self.compile(tuple(argtypes))
    354         except errors.ForceLiteralArg as e:
    355             # Received request for compiler re-entry with the list of arguments

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/dispatcher.py in compile(self, sig)
    766             self._cache_misses[sig] += 1
    767             try:
--> 768                 cres = self._compiler.compile(args, return_type)
    769             except errors.ForceLiteralArg as e:
    770                 def folded(args, kws):

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/dispatcher.py in compile(self, args, return_type)
     75 
     76     def compile(self, args, return_type):
---> 77         status, retval = self._compile_cached(args, return_type)
     78         if status:
     79             return retval

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/dispatcher.py in _compile_cached(self, args, return_type)
     89 
     90         try:
---> 91             retval = self._compile_core(args, return_type)
     92         except errors.TypingError as e:
     93             self._failed_cache[key] = e

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/dispatcher.py in _compile_core(self, args, return_type)
    107                                       args=args, return_type=return_type,
    108                                       flags=flags, locals=self.locals,
--> 109                                       pipeline_class=self.pipeline_class)
    110         # Check typing error if object mode is used
    111         if cres.typing_error is not None and not flags.enable_pyobject:

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler.py in compile_extra(typingctx, targetctx, func, args, return_type, flags, locals, library, pipeline_class)
    526     pipeline = pipeline_class(typingctx, targetctx, library,
    527                               args, return_type, flags, locals)
--> 528     return pipeline.compile_extra(func)
    529 
    530 

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler.py in compile_extra(self, func)
    324         self.state.lifted = ()
    325         self.state.lifted_from = None
--> 326         return self._compile_bytecode()
    327 
    328     def compile_ir(self, func_ir, lifted=(), lifted_from=None):

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler.py in _compile_bytecode(self)
    383         """
    384         assert self.state.func_ir is None
--> 385         return self._compile_core()
    386 
    387     def _compile_ir(self):

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler.py in _compile_core(self)
    363                 self.state.status.fail_reason = e
    364                 if is_final_pipeline:
--> 365                     raise e
    366         else:
    367             raise CompilerError("All available pipelines exhausted")

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler.py in _compile_core(self)
    354             res = None
    355             try:
--> 356                 pm.run(self.state)
    357                 if self.state.cr is not None:
    358                     break

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler_machinery.py in run(self, state)
    326                     (self.pipeline_name, pass_desc)
    327                 patched_exception = self._patch_error(msg, e)
--> 328                 raise patched_exception
    329 
    330     def dependency_analysis(self):

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler_machinery.py in run(self, state)
    317                 pass_inst = _pass_registry.get(pss).pass_inst
    318                 if isinstance(pass_inst, CompilerPass):
--> 319                     self._runPass(idx, pass_inst, state)
    320                 else:
    321                     raise BaseException("Legacy pass in use")

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler_machinery.py in _runPass(self, index, pss, internal_state)
    279             mutated |= check(pss.run_initialization, internal_state)
    280         with SimpleTimer() as pass_time:
--> 281             mutated |= check(pss.run_pass, internal_state)
    282         with SimpleTimer() as finalize_time:
    283             mutated |= check(pss.run_finalizer, internal_state)

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/compiler_machinery.py in check(func, compiler_state)
    266 
    267         def check(func, compiler_state):
--> 268             mangled = func(compiler_state)
    269             if mangled not in (True, False):
    270                 msg = ("CompilerPass implementations should return True/False. "

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/typed_passes.py in run_pass(self, state)
    378             state.library.enable_object_caching()
    379 
--> 380         NativeLowering().run_pass(state) # TODO: Pull this out into the pipeline
    381         lowered = state['cr']
    382         signature = typing.signature(state.return_type, *state.args)

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/typed_passes.py in run_pass(self, state)
    323                 lower = lowering.Lower(targetctx, library, fndesc, interp,
    324                                        metadata=metadata)
--> 325                 lower.lower()
    326                 if not flags.no_cpython_wrapper:
    327                     lower.create_cpython_wrapper(flags.release_gil)

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/lowering.py in lower(self)
    177         if self.generator_info is None:
    178             self.genlower = None
--> 179             self.lower_normal_function(self.fndesc)
    180         else:
    181             self.genlower = self.GeneratorLower(self)

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/lowering.py in lower_normal_function(self, fndesc)
    218         # Init argument values
    219         self.extract_function_arguments()
--> 220         entry_block_tail = self.lower_function_body()
    221 
    222         # Close tail of entry block

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/lowering.py in lower_function_body(self)
    243             bb = self.blkmap[offset]
    244             self.builder.position_at_end(bb)
--> 245             self.lower_block(block)
    246 
    247         self.post_lower()

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/lowering.py in lower_block(self, block)
    258             with new_error_context('lowering "{inst}" at {loc}', inst=inst,
    259                                    loc=self.loc, errcls_=defaulterrcls):
--> 260                 self.lower_inst(inst)
    261 
    262     def create_cpython_wrapper(self, release_gil=False):

~/anaconda3/envs/fastai3/lib/python3.7/contextlib.py in __exit__(self, type, value, traceback)
    128                 value = type()
    129             try:
--> 130                 self.gen.throw(type, value, traceback)
    131             except StopIteration as exc:
    132                 # Suppress StopIteration *unless* it's the same exception that

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/errors.py in new_error_context(fmt_, *args, **kwargs)
    723         from numba import config
    724         tb = sys.exc_info()[2] if config.FULL_TRACEBACKS else None
--> 725         six.reraise(type(newerr), newerr, tb)
    726 
    727 

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/numba/six.py in reraise(tp, value, tb)
    667         if value.__traceback__ is not tb:
    668             raise value.with_traceback(tb)
--> 669         raise value
    670 
    671 else:

LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
Cannot lower constant of type 'reflected list(int64)'

File "fastai_timeseries/exp/rocket_functions.py", line 16:
def generate_kernels(input_length, num_kernels, kss=[7, 9, 11], pad=True, dilate=True):
    candidate_lengths = np.array((kss))
    ^

[1] During: lowering "kss = arg(2, name=kss)" at /home/malcolm/fastaiActive/ROCKET/timeseriesAI-master20191230/fastai_timeseries/exp/rocket_functions.py (16)

-------------------------------------------------------------------------------
This should not have happened, a problem has occurred in Numba's internals.
You are currently using Numba version 0.46.0.

Please report the error message and traceback, along with a minimal reproducer
at: https://github.com/numba/numba/issues/new

If more help is needed please feel free to speak to the Numba core developers
directly at: https://gitter.im/numba/numba

Thanks in advance for your help in improving Numba!
--------------------------------------

Do you have any clue about what is going on here? This numba error might as well be ancient Sumerian for all I can understand of it.

Thanks for any help.

scottcha · January 7, 2020, 5:35pm

@Pomo I’m also seeing the issue in the rocket nb. I’ve added a github issue to the repro:

s.s.o · January 8, 2020, 3:46pm

some new papers might be interesting

Temporal Tensor Transformation Network for Multivariate Time Series Prediction
https://arxiv.org/abs/2001.01051

Root Cause Detection Among Anomalous Time Series Using Temporal State Alignment
https://arxiv.org/abs/2001.01056