Just as transfer learning has been well-established in vision and now is taking off in NLP, will a similar phenomenon occur with data augmention and NLP? In the last lesson Jeremy mentioned there’s big potential for devolpement of NLP augmentation techniques.
I’ve been thinking about it and experimenting for a little while now, and I’m looking to learn as much as I can about current practices and suggestions on what might be the most fruitful things to try. I know there are challenges and it can be more task specific than vision, but obviously there’s an opportunity.
EventualIy I think it would be great if fastai becomes the first library to support built in text transformations. If there are augmentation techniques which work well with many common NLP tasks, then perhaps they could be implemented as transformations for use with the text learner.
I would like to ask the community if you have used data augmentation in your NLP tasks to please share your experience.
Recently, I’ve been trying to use ‘backtranslation’ to create paraphrased augmented training instances. This is using machine translation to translate from the source to a ‘pivot’ language and then back again. I had a lot of AWS credits which were expiring, so as an experiment I used amazon translate api to translate all the IMDb reviews from English to German and then back to English. An example:
Ghost of Dragstrip Hollow is a typical 1950’s teens in turmoil movie. It is not a horror or science fiction movie. Plot concerns a group of teens who are about to get kicked out of their “hot rod” club because they cannot meet the rent. Once kicked out, they decide to try an old Haunted House. The only saving grace for the film is that the “ghost” (Paul Blaisdell in the She Creature suit) turns out to be an out of work movie monster played by Blaisdell.
Ghost of Dragstrip Hollow is a typical 50s teenager in turbulence film. It’s not a horror or science fiction movie. Action concerns a group of teenagers who are about to throw out of their “Hot Rod” club because they cannot fulfill the rent. Once they’re kicked out, they decide to try an old Haunted House. The only saving grace for the film is that the “ghost” (Paul Blaisdell in the She Creature suit) turns out to be a movie monster played by Blaisdell.
The meaning is preserved and there is some paraphrasing, although it is very close to the original. I read in a comment that somebody felt that modern machine translation is too good now for paraphrasing and they recommend using an older pre-DL MOSES model to generate instances further from the originals.
I have not tested the augmented IMDb set to see if it offers an improvement yet, but I will update when I do. If anybody else is interested in trying I have put up the full augmented archive here:
Beyond paraphrasing, I’ve seen other methods motioned:
- Replacing words with synonyms
- Perturbations (letter, word, or sentence level). See https://github.com/noisemix/noisemix
- Generative(GAN/auto encoders)
Would love to learn of anybody’s experience, ideas or intuitions regarding text data augmentation!