Hi,
I am just trying to reproduce the code found in the doc (here) where we want to fine-tune a pretrained language model.
I first created a language model with:
train_lm = (TextList.from_df(df=df[:4*split], cols=['sentences']).random_split_by_pct() .label_for_lm() .databunch())
which seems to work well sinceshow_batch. gives me:
| idx | text |
|---|---|
| 0 | xxbos xxmaj we have been to this place many times , and always have great food , wine , and service . xxbos xxmaj my friend xxunk her chicken and xxunk xxunk . xxbos i 'm xxunk up for my next visit . xxbos xxmaj xxunk fresh so brought hot hot hot to your table . xxbos xxmaj the place 's decor and hidden xxunk made for a good xxunk |
| 1 | xxunk … its quite xxunk - xxunk , and xxunk your xxunk to make this place look xxunk and modern . xxbos xxmaj price no more than a xxmaj xxunk xxunk but way better . xxbos i love it . xxbos a great place to xxunk up for some food and drinks … xxbos xxmaj the bagel was huge . xxbos xxmaj we ordered some beef and xxunk soup dishes |
And then I defined a language model learner:
learn = language_model_learner(data_lm, pretrained_model=URLs.WT103_1)
learn.fit_one_cycle(2, 1e-2)
learn.save('mini_train_lm')
learn.save_encoder('mini_train_encoder')
However, I got the following error:
RuntimeError Traceback (most recent call last)
<ipython-input-617-28be5a339a08> in <module>
----> 1 learn = language_model_learner(train_lm, pretrained_model=URLs.WT103_1)
2 learn.fit_one_cycle(2, 1e-2)
3 learn.save('mini_train_lm')
4 learn.save_encoder('mini_train_encoder')
~\Anaconda3\envs\tensorflow\lib\site-packages\fastai\text\learner.py in language_model_learner(data, bptt, emb_sz, nh, nl, pad_token, drop_mult, tie_weights, bias, qrnn, pretrained_model, pretrained_fnames, **kwargs)
136 fnames = [list(model_path.glob(f'*.{ext}'))[0] for ext in ['pth', 'pkl']]
137 learn.load_pretrained(*fnames)
--> 138 learn.freeze()
139 if pretrained_fnames is not None:
140 fnames = [learn.path/learn.model_dir/f'{fn}.{ext}' for fn,ext in zip(pretrained_fnames, ['pth', 'pkl'])]
~\Anaconda3\envs\tensorflow\lib\site-packages\fastai\basic_train.py in freeze(self)
181 "Freeze up to last layer."
182 assert(len(self.layer_groups)>1)
--> 183 self.freeze_to(-1)
184
185 def unfreeze(self):
~\Anaconda3\envs\tensorflow\lib\site-packages\fastai\basic_train.py in freeze_to(self, n)
175 for g in self.layer_groups[:n]:
176 for l in g:
--> 177 if not self.train_bn or not isinstance(l, bn_types): requires_grad(l, False)
178 for g in self.layer_groups[n:]: requires_grad(g, True)
179
~\Anaconda3\envs\tensorflow\lib\site-packages\fastai\torch_core.py in requires_grad(m, b)
102 if not ps: return None
103 if b is None: return ps[0].requires_grad
--> 104 for p in ps: p.requires_grad=b
105
106 def trainable_params(m:nn.Module)->ParamList:
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
Any idea where it could come from?
Thanks,
Nicolas