Loading pretrained models in fastaiV1 (Ulmfit)

(Vishnu Kumar Kailash Kumar) #21

@angelinayy, Nope but the function to convert from .h5 to .pth worked well for me.

(Beatrice Paige) #22

this worked for some of the senior batches as well

(Angelina Yang) #23

hi, thank you! the convert function worked for me too. did you run something like below to import the .pth and the itos103.pkl files?

I still have a keyerror: KeyError : ‘1.decoder.bias’

Did you have something like this? thanks!

learn = language_model_learner(data_lm, pretrained_fnames=[pre_LM_path, pre_itos2_path], drop_mult=0.3)



I have the same problem as @angelinayy. In my case:
First i train on my own datasets LM:

data = TextLMDataBunch.from_csv(dir_name, file_name, text_cols=0)
learn = language_model_learner(data)
learn.fit(epochs=10, lr=0.01)

Second i want use this model as pretrained so:

new_data = TextLMDataBunch.from_csv(dir_name, file_name_2, text_cols=0)
learn = language_model_learner(data, pretrained_fnames=[‘path/to/lm_encoder’, ‘path/to/itos’])

and i have the same error:

File “…/fastai/text/learner.py”, line 20, in convert_weights
dec_bias, enc_wgts = wgts[‘1.decoder.bias’], wgts[‘0.encoder.weight’]
KeyError: ‘1.decoder.bias’

Any suggestions or solutions?


(Angelina Yang) #25

hello, I think this may work. I still have errors but not this one. we can connect on this, but you can try download these 2 files in the below link. thank you!


(ali baltschun) #26

did you find the solution @kazutzu ??
i have a same problem


Yes, i use to save model this:

torch.save(learn.model.state_dict(), 'path/to/lm_encoder.pth')

and then i load the weights in this way:

learn = language_model_learner(data)
learn.load_pretrained(wgts_fname='path/to/lm_encoder.pth'), itos_fname='path/to/itos.pkl')

in this way torch.save() save all layers weigth . In original save_encoder() is this:

torch.save(self.model[0].state_dict(), 'path')

(ali baltschun) #28

@kazutzu that code using v.07 or v.1 ?
and how to load the .h5 before you export to .pth ?
thanks for your time

(basque) #29

same issue with KeyError: '1.decoder.bias'. any solution?