What is itos in Language Models

#1

Hello,
Noob question here. What is itos in fastai language models and how can I generate it from language model the model that I trained

0 Likes

(urmas pitsi) #2

Check this:

https://docs.fast.ai/text.transform.html

1 Like

#3

Thank you

0 Likes

(dinesh) #4

Hii I didn’t get the answer. Can you share exact code code to save itos of a language model?

I ran the following code:
pickle.dump(data.vocab.itos, open(PATH, ‘wb’))

Then I loaded in this way,
learn = language_model_learner(data_lm, AWD_LSTM , pretrained_fnames = (‘model_path.pth’,‘itos_path’), pretrained= False, drop_mult=0.3)

I got the following error:

TypeError Traceback (most recent call last)
in
1 # Define language model learner
----> 2 learn = language_model_learner(data_lm, AWD_LSTM , pretrained_fnames = (‘fine_tuned_LM_Cap_EA_AD_OI.pth’,‘itos’), pretrained= False, drop_mult=0.3)

/home/AIX_Common/files/opt/anaconda2/envs/ai-gpu/lib/python3.6/site-packages/fastai/text/learner.py in language_model_learner(data, arch, config, drop_mult, pretrained, pretrained_fnames, **learn_kwargs)
215 if pretrained_fnames is not None:
216 fnames = [learn.path/learn.model_dir/f’{fn}.{ext}’ for fn,ext in zip(pretrained_fnames, [‘pth’, ‘pkl’])]
–> 217 learn.load_pretrained(*fnames)
218 learn.freeze()
219 return learn

/home/AIX_Common/files/opt/anaconda2/envs/ai-gpu/lib/python3.6/site-packages/fastai/text/learner.py in load_pretrained(self, wgts_fname, itos_fname, strict)
73 “Load a pretrained model and adapts it to the data vocabulary.”
74 old_itos = pickle.load(open(itos_fname, ‘rb’))
—> 75 old_stoi = {v:k for k,v in enumerate(old_itos)}
76 wgts = torch.load(wgts_fname, map_location=lambda storage, loc: storage)
77 if ‘model’ in wgts: wgts = wgts[‘model’]

TypeError: ‘int’ object is not iterable

Please let me know where I went wrong…

0 Likes