I just wanted to drop an idea here for a network architecture that could be included in FastAI. I hope this isn’t redundant (i.e. it has already implemented).
Recently, there have been some implementations of CNNs that are (at least from the published research) seemingly outperforming RNN models in different tasks. These so-called Temporal Convolutional Networks has various advantages over RNNs (faster, longer memory, training, etc.). The most famous perhaps architecture of this is WaveNet (https://arxiv.org/pdf/1609.03499.pdf).
I would say that, at least for someone at my level, it is not the easiest architecture to decipher and certainly not the easiest to write in code which I feel makes it a good candidate, apart from the perfomance, for FastAI.
There are a few implementations out there, the most popular pyTorch seems to be this one: https://github.com/locuslab/TCN/. There is also a higher level API for Keras, which I think is quite amazing. It offers simple calls that generate different types of TCNs with various input parameters for the user to use (https://github.com/philipperemy/keras-tcn). It might be a good example to base a FastAI implementation on.
Hope this is interesting to some of you!
I am the author of https://github.com/philipperemy/keras-tcn .
I think it’s a good idea to have Temporal Convolutions Networks in FastAI.
I’ll be ready to provide some help for the implementation.
Philippe I wonder if you have time or plan to work in trying to do this for fastai v1.
Would be awesome I think for us to play with this but of course I am not so sure how easy it will be to do so.
I’m motivating my buddies to help implement it for FastAI v1.
I’m sure it will be a good add to the framework. Not sure yet how difficult would it be. Let’s see that
I started working on TCN with fastai (v0.7) a few months back (github repo). The goal was to train a TCN language model and finetune it for text classification, similar to Jeremy’s ULMFIT approach which used LSTMs.
However, I couldn’t get the perplexity low enough. I would like to revisit this project with fastai v1.
@Gabriel_Syme, @philipperemy, @kachio:
Did you make any progress on this project? I played with TCN last year for stock prediction (without much success). Now would like to experiment with audio classification using fastai2 DataLoaders and Learner.
The first step would be to implement pmnist_test in a demo notebook, using fastai2.
Comments, ideas, directions?