Temporal Convolution Networks in FastAI


(Theodoros Galanos) #1

Hello everyone,

I just wanted to drop an idea here for a network architecture that could be included in FastAI. I hope this isn’t redundant (i.e. it has already implemented).

Recently, there have been some implementations of CNNs that are (at least from the published research) seemingly outperforming RNN models in different tasks. These so-called Temporal Convolutional Networks has various advantages over RNNs (faster, longer memory, training, etc.). The most famous perhaps architecture of this is WaveNet (https://arxiv.org/pdf/1609.03499.pdf).

I would say that, at least for someone at my level, it is not the easiest architecture to decipher and certainly not the easiest to write in code which I feel makes it a good candidate, apart from the perfomance, for FastAI.

There are a few implementations out there, the most popular pyTorch seems to be this one: https://github.com/locuslab/TCN/. There is also a higher level API for Keras, which I think is quite amazing. It offers simple calls that generate different types of TCNs with various input parameters for the user to use (https://github.com/philipperemy/keras-tcn). It might be a good example to base a FastAI implementation on.

Hope this is interesting to some of you!

Kind regards,
Theodore.


(Philippe) #2

Hey Theodore,

I am the author of https://github.com/philipperemy/keras-tcn .

I think it’s a good idea to have Temporal Convolutions Networks in FastAI.

I’ll be ready to provide some help for the implementation.

Best regards,
Philippe


(Theodoros Galanos) #3

Philippe I wonder if you have time or plan to work in trying to do this for fastai v1.

Would be awesome I think for us to play with this but of course I am not so sure how easy it will be to do so.

Kind regards,

Theodore.


(Philippe) #4

I’m motivating my buddies to help implement it for FastAI v1.

I’m sure it will be a good add to the framework. Not sure yet how difficult would it be. Let’s see that :slight_smile:

Best regards,
Philippe


(Kachi O.) #5

I started working on TCN with fastai (v0.7) a few months back (github repo). The goal was to train a TCN language model and finetune it for text classification, similar to Jeremy’s ULMFIT approach which used LSTMs.

However, I couldn’t get the perplexity low enough. I would like to revisit this project with fastai v1.