I can also confirm that for classification tasks, a small portion of wikipedia was good enough (22m, sampled from full). I am looking forward to see the great work of Kasper (and your lm) integrated into fastai. Still getting OOM for larger wiki texts (GPU Optimizations Central).
One quick question @piotr.czapla : Am I correct that current fastai (v 1.0.39) does not read bilstm models?