250M of pure language text sounds like a reasonable starting point.
I make the Language Models for both Hindi and Indonesian using the code that I shared above.
The wikitext-103 is an English only pretrained model. That cannot be used for any other language.