How can I use a pretrained model for a language model learner?

mlnoob · July 16, 2019, 6:47am

In the past, you were able to do something like

learn = language_model_learner(data_lm, pretrained_model=URLs.WT103, drop_mult=0.3)

however, the API has changed and I can’t find in the docs on how to load a pretrained model. The closest I got was

learn = language_model_learner(data_lm, AWD_LSTM, pretrained_fnames=URLs.WT103, pretrained=True, drop_mult=.3)

which then looks for a models/t.pkl file which I don’t have.

magiclantern · July 22, 2019, 7:46pm

I’ve found the fastai docs website to be very helpful in answering questions like this - I’ve had similar and still am learning more about the language model learner. Take a look at https://docs.fast.ai/text.learner.html#language_model_learner for high level details on how the language_model_learner works.

The function definition says that pretrained defaults to True and pretrained_fnames defaults to None. When looking at the source code for the language_model_learner available here:

github.com

fastai/fastai/blob/master/fastai/text/learner.py#L201


    for k in config.keys():
        if k.endswith('_p'): config[k] *= drop_mult
    tie_weights,output_p,out_bias = map(config.pop, ['tie_weights', 'output_p', 'out_bias'])
    init = config.pop('init') if 'init' in config else None
    encoder = arch(vocab_sz, **config)
    enc = encoder.encoder if tie_weights else None
    decoder = LinearDecoder(vocab_sz, config[meta['hid_name']], output_p, tie_encoder=enc, bias=out_bias)
    model = SequentialRNN(encoder, decoder)
    return model if init is None else model.apply(init)


def language_model_learner(data:DataBunch, arch, config:dict=None, drop_mult:float=1., pretrained:bool=True,
                           pretrained_fnames:OptStrTuple=None, **learn_kwargs) -> 'LanguageLearner':
    "Create a `Learner` with a language model from `data` and `arch`."
    model = get_language_model(arch, len(data.vocab.itos), config=config, drop_mult=drop_mult)
    meta = _model_meta[arch]
    learn = LanguageLearner(data, model, split_func=meta['split_lm'], **learn_kwargs)
    url = 'url_bwd' if data.backwards else 'url'
    if pretrained or pretrained_fnames:
        if pretrained_fnames is not None:
            fnames = [learn.path/learn.model_dir/f'{fn}.{ext}' for fn,ext in zip(pretrained_fnames, ['pth', 'pkl'])]
        else:

It shows that if you don’t pass in a pretrained model, one is loaded by default (if you are using the AWD_LSTM or Transformer architectures.)