Training language model to predict characters

[BMarcin (Marcin Borzymowski)]

Hi! I would like to use Language Models for predicting characters . My input data are words (one word on each line in txt). I’ve decided to use TextList for storing values (see code below). data_lm = (TextList.from_df(df, cols ="text") .split_by_rand_pct(0.1, seed=101) .label_for_lm() .databunch(bs=10, num_workers=0)). I know that I need my custom preprocessing (also with tokenizer) and I don’t know how to implement one. Is there anyone who tried something like this?

My main idea is to generate words with distribution learned from training words.

[kennysong]

For future readers, I had the same question and answered it in this thread:

Character-level language model

(For fastai 1.0.60.) Quite a late answer, but I also needed to build a character-level LM, and I didn’t find any good code snippets. Here’s what I came up with after digging around in fastai/text/transform.py. This is my custom CharTokenizer class. It’s super simple, mostly copied from the SpacyTokenizer class. The character-level tokenization logic is in the tokenizer() method, which you can customize. class CharTokenizer(BaseTokenizer): def __init__(self, lang:str='no_lang'): '…