Language Model Zoo 🦍


(Daniel Cañueto) #247

Great work! I’ve tried to reproduce the notebook but I haven’t found how to get the backbone model used. How could I get this model? Thanks so much!


(Francisco Ingham) #248

Hey Daniel, I could share the weights with you so you can load them. I’ll try to upload them to Github tomorrow, if not possible I’ll upload them to GDrive.

Thanks for the useful comments in Github as well.


(Daniel Cañueto) #249

Great! Thanks a lot.


#250

Hi, thanks for sharing your model. The official fastai examples use .h5 format to save their models while yours uses .pt

When I try to torch.load() your language model, I get an AttributeError:

AttributeError: Can't get attribute '_rebuild_tensor_v2' on <module 'torch._utils' from '.../torch/_utils.py'>

This probably results from a version mismatch. Any ideas on how to overcome this (without having to retrain)? I’d really like to test out your pre-trained model.


(Amitabha) #251

i am also devote myself to the model’s apply to chinese


(Amitabha) #252

i am intrested in you try in chinese


(Amitabha) #253

i am also the problem this make me confused


(Amitabha) #254

you are lucky i am only 8G


(Amitabha) #255

so what your suggest for 8G RAM


(Thomas) #256

Last I looked, it was the .h5 file name but not necessarily the the .h5 format - at least grepping for h5py, it only shows up in the inceptionv4.py.
This error happens when you use an old version of PyTorch. I usually use some form of master, but 0.4.1 should do. If you don’t want to change your main PyTorch version (you should, though!), you can load them with that and then save the model parameters with numpy or even h5py.

Best regards

Thomas


(Amitabha) #257

when i rum pre_lm.py the errors are fllowing:
dir_path data/en_data; cuda_id 0; cl 12; bs 64; backwards False; lr 0.001; sampled True; pretrain_id
Traceback (most recent call last):
File “pretrain_lm.py”, line 53, in
if name == ‘main’: fire.Fire(train_lm)
File “/home/yhl/anaconda3/envs/fastai/lib/python3.6/site-packages/fire/core.py”, line 127, in Fire
component_trace = _Fire(component, args, context, name)
File “/home/yhl/anaconda3/envs/fastai/lib/python3.6/site-packages/fire/core.py”, line 366, in _Fire
component, remaining_args)
File “/home/yhl/anaconda3/envs/fastai/lib/python3.6/site-packages/fire/core.py”, line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File “pretrain_lm.py”, line 42, in train_lm
learner,crit = get_learner(drops, 15000, sampled, md, em_sz, nh, nl, opt_fn, tprs)
File “/home/yhl/fastai/courses/dl2/imdb_scripts/sampled_sm.py”, line 85, in get_learner
m = to_gpu(get_language_model(md.n_tok, em_sz, nhid, nl, md.pad_idx, decode_train=False, dropouts=drops))
File “/home/yhl/fastai/courses/dl2/imdb_scripts/sampled_sm.py”, line 46, in get_language_model
rnn_enc = RNN_Encoder(n_tok, em_sz, n_hid=nhid, n_layers=nlayers, pad_token=pad_token,dropouti=dropouts[0], wdrop=dropouts[2], dropoute=dropouts[3], dropouth=dropouts[4])
TypeError: init() got an unexpected keyword argument ‘n_hid’


(Amitabha) #258

when i run the pretrain_lm.py

Traceback (most recent call last):
File “pretrain_lm.py”, line 53, in
if name == ‘main’: fire.Fire(train_lm)
File “/home/yhl/anaconda3/envs/fastai/lib/python3.6/site-packages/fire/core.py”, line 127, in Fire
component_trace = _Fire(component, args, context, name)
File “/home/yhl/anaconda3/envs/fastai/lib/python3.6/site-packages/fire/core.py”, line 366, in _Fire
component, remaining_args)
File “/home/yhl/anaconda3/envs/fastai/lib/python3.6/site-packages/fire/core.py”, line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File “pretrain_lm.py”, line 42, in train_lm
learner,crit = get_learner(drops, 15000, sampled, md, em_sz, nh, nl, opt_fn, tprs)
File “/home/yhl/fastai/courses/dl2/imdb_scripts/sampled_sm.py”, line 85, in get_learner
m = to_gpu(get_language_model(md.n_tok, em_sz, nhid, nl, md.pad_idx, decode_train=False, dropouts=drops))
File “/home/yhl/fastai/courses/dl2/imdb_scripts/sampled_sm.py”, line 46, in get_language_model
rnn_enc = RNN_Encoder(n_tok, em_sz, n_hid=nhid, n_layers=nlayers, pad_token=pad_token,dropouti=dropouts[0], wdrop=dropouts[2], dropoute=dropouts[3], dropouth=dropouts[4])
TypeError: init() got an unexpected keyword argument ‘n_hid’

can you help me i am looking forward to you reply


#259

You don’t seem to have the latest version of the library (the variable nhid was renamed n_hid in a PR a few months ago). Just do a git pull in the fastai folder.


(Amitabha) #260

Thank you for your prompt reply.It make me benfit a lot.
Thank you very much.
Wishes you to be happy daily, has a good mood!


(Francisco Ingham) #261

I suggest doing the same in smaller chunks. I would increasingly double the number of chunks till you no longer get a memory error.


(Amitabha) #262

Thank you for your kind suggest.
I found the question is result form the “np.save()”
So i split the tran.csv solved the question.
Thank you very much.
Wishes you to be happy daily, has a good mood!


(Monique Monteiro) #263

Hi Fabio,

What perplexity did you get? I’ve also implemented a Portuguese language with ULMFit.

Regarding your question about predicting the next word, I’m using sampling according to the softmax predicted distribution. At least, the texts get a little more “random”…

Cheers,
Monique