I am currently trying to train a classifier based on Norwegian text. My first step is to fine-tune a pre-trained Norwegian language model (https://github.com/mollerhoj/Scandinavian-ULMFiT). I have imported the two files:
- norwegian_enc.pth
- norwegian_itos.pkl
My training data was made by:
data_lm = (TextList.from_df(df, “”, cols=‘sentence’)
.random_split_by_pct(0.2)
.label_for_lm()
.databunch(bs=bs))
where df simply looks like:
sentence
I den norske delegasjonen er blant andre klima…
Generalsekretær i sjakkforbundet, Geir Nesheim…
…
KeyError: ‘0.encoder.weight’
When trying to load the pre-trained resources, I get an error
learn = language_model_learner(data_lm, AWD_LSTM, pretrained=False, drop_mult=0.7, pretrained_fnames=[“norwegian_enc”, “norwegian_itos”])
/usr/local/lib/python3.6/dist-packages/fastai/text/learner.py in language_model_learner(data, arch, config, drop_mult, pretrained, pretrained_fnames, **learn_kwargs)
216 model_path = untar_data(meta[url] , data=False)
217 fnames = [list(model_path.glob(f’*.{ext}’))[0] for ext in [‘pth’, ‘pkl’]]
–> 218 learn.load_pretrained(*fnames)
219 learn.freeze()
220 return learn
/usr/local/lib/python3.6/dist-packages/fastai/text/learner.py in load_pretrained(self, wgts_fname, itos_fname, strict)
77 wgts = torch.load(wgts_fname, map_location=lambda storage, loc: storage)
78 if ‘model’ in wgts: wgts = wgts[‘model’]
—> 79 wgts = convert_weights(wgts, old_stoi, self.data.train_ds.vocab.itos)
80 self.model.load_state_dict(wgts, strict=strict)
81
/usr/local/lib/python3.6/dist-packages/fastai/text/learner.py in convert_weights(wgts, stoi_wgts, itos_new)
28 def convert_weights(wgts:Weights, stoi_wgts:Dict[str,int], itos_new:Collection[str]) -> Weights:
29 “Convert the model wgts to go with a new vocabulary.”
—> 30 dec_bias, enc_wgts = wgts.get(‘1.decoder.bias’, None), wgts[‘0.encoder.weight’]
31 wgts_m = enc_wgts.mean(0)
32 if dec_bias is not None: bias_m = dec_bias.mean(0)
KeyError: ‘0.encoder.weight’
Does anyone know what the reason for this could be?