An interesting technique for data augmentation specific to RNNs from “Data Noising as Smoothing in Neural Network Language Models” by Xie et al (ICLR 2017) (arXiv):
In this work, we consider noising primitives as a form of data augmentation
for recurrent neural network-based language models.