Has anyone used Sliced RNN that came out this July? 136X faster than standard RNN seems to be a very promising addition to QRNN and ULMFiT, and there is potential to increase the speed even more. The author has a Github repo and even though the code was in Python 2.7 and TF+Keras, the unique part seems to be to slice the sequences into smaller chunks.
If nobody has experimented with it I’ll try to give it a shot and see how much faster it gets!