How to load data for Masked Language Modeling

[Richard-Wang]

Hi, I am trying to pretrain a model from scratch with masked language model (MLM) objective as bert, and I am thinking about loading the data.

• Is there an off-the-shelf dataloader or snippet I can use?

• If there’s no such thing, which level ( dataset , datablock , …) should I start ? How would you do it ?

Thanks

[morgan (Morgan McGuire)]

Hey @Richard-Wang, did you manage to find an answer to this? Looking to share a language model tutorial notebook too

[Richard-Wang]

Hi @morgan !

I am sorry for this too late reply, I just had my whole heart set on organizing all things together.

So the result is this post (and the following releases)

Pretrain MLM and fintune on GLUE with fastai - 1 - Masked laguage model callback and Electra callback

I created callbacks for masked language modeling and replaced token detection (Electra). Please help me test / check / improve the code, and also I am seeking a chance to PR to fastai2. richardyy1188/ Pretrain-MLM-and-finetune-on-GLUE-with-fastai/MLM_pretraing_with_fastai.ipynb Take a peek at it: class MaskedLanguageModel(Callback): @delegates(mask_tokens) def __init__(self, mask_tok_id, special_tok_ids, vocab_size, **kwargs): self.mask_tokens = partial(mask_tokens, …

It will be a series from pretraing mlm model to multi-task finetune on GLUE. You can follow my Twitter for updates of this series.

I will also be grateful if you can try the code on your gpus and with a larger corpus to see if it truely can reach the accuracy reported (which it should be) . And also help me debug the fp16 issue mentioned in the post.