ULMFiT for other languages

I am currently trying to train a classifier based on Norwegian text. My first step is to fine-tune a pre-trained Norwegian language model (https://github.com/mollerhoj/Scandinavian-ULMFiT). I have imported the two files:

  1. norwegian_enc.pth
  2. norwegian_itos.pkl

My training data was made by:
data_lm = (TextList.from_df(df, “”, cols=‘sentence’)
.random_split_by_pct(0.2)
.label_for_lm()
.databunch(bs=bs))

where df simply looks like:
sentence
I den norske delegasjonen er blant andre klima…
Generalsekretær i sjakkforbundet, Geir Nesheim…

KeyError: ‘0.encoder.weight’

When trying to load the pre-trained resources, I get an error

learn = language_model_learner(data_lm, AWD_LSTM, pretrained=False, drop_mult=0.7, pretrained_fnames=[“norwegian_enc”, “norwegian_itos”])

/usr/local/lib/python3.6/dist-packages/fastai/text/learner.py in language_model_learner(data, arch, config, drop_mult, pretrained, pretrained_fnames, **learn_kwargs)
216 model_path = untar_data(meta[url] , data=False)
217 fnames = [list(model_path.glob(f’*.{ext}’))[0] for ext in [‘pth’, ‘pkl’]]
–> 218 learn.load_pretrained(*fnames)
219 learn.freeze()
220 return learn

/usr/local/lib/python3.6/dist-packages/fastai/text/learner.py in load_pretrained(self, wgts_fname, itos_fname, strict)
77 wgts = torch.load(wgts_fname, map_location=lambda storage, loc: storage)
78 if ‘model’ in wgts: wgts = wgts[‘model’]
—> 79 wgts = convert_weights(wgts, old_stoi, self.data.train_ds.vocab.itos)
80 self.model.load_state_dict(wgts, strict=strict)
81

/usr/local/lib/python3.6/dist-packages/fastai/text/learner.py in convert_weights(wgts, stoi_wgts, itos_new)
28 def convert_weights(wgts:Weights, stoi_wgts:Dict[str,int], itos_new:Collection[str]) -> Weights:
29 “Convert the model wgts to go with a new vocabulary.”
—> 30 dec_bias, enc_wgts = wgts.get(‘1.decoder.bias’, None), wgts[‘0.encoder.weight’]
31 wgts_m = enc_wgts.mean(0)
32 if dec_bias is not None: bias_m = dec_bias.mean(0)

KeyError: ‘0.encoder.weight’

Does anyone know what the reason for this could be?

1 Like

Could be because fastai made some breaking changes. Check this post Language_model_learner not working as before?

If that does not work, try to load it this way:

import pickle
itos = pickle.load( open( “res/norwegian_itos.pkl”, “rb” ) )
vocab = Vocab(itos)

data_lm = TextLMDataBunch.from_df(path=“res/model/ulmfit”, train_df=df_train, valid_df=df_valid, vocab=vocab)
learn = language_model_learner(data_lm, AWD_LSTM, pretrained=False, drop_mult=0.5)
learn.load_encoder(‘norwegian_enc’)