Training language model to predict characters

[kennysong]

For future readers, I had the same question and answered it in this thread:

Character-level language model

(For fastai 1.0.60.) Quite a late answer, but I also needed to build a character-level LM, and I didn’t find any good code snippets. Here’s what I came up with after digging around in fastai/text/transform.py. This is my custom CharTokenizer class. It’s super simple, mostly copied from the SpacyTokenizer class. The character-level tokenization logic is in the tokenizer() method, which you can customize. class CharTokenizer(BaseTokenizer): def __init__(self, lang:str='no_lang'): '…