I have looked through the forum and haven’t been any to find any topics on ULMFIT for regression for fast at v2. I have a fairly simple problem of trying to predict a score between 1-6 (Float) given a couple 100 words of text. I thought I would just be able to use the normal ULMFIT code and have a TextBlock and RegressionBlock and then change the loss function to MSE or RMSE. However, I have been running into errors with this. Does any one have any suggestions?
Hey,
It works great pretty much exactly as you stated, and can be done for 1 or more label columns 
Here’s a code sample:
dls_clas = TextDataLoaders.from_df(df=df, valid_pct=0.1, seed=42, text_col='text', text_vocab=vocab, bs=bs, y_names=labels, y_block=RegressionBlock(len(labels)), label_col=labels) learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5, n_out=len(labels), y_range=(1, 6), metrics=[R2Score()], cbs=cbs, ) if default_device().type == 'cuda': learn = learn.to_fp16() learn = learn.load_encoder('finetuned_lm') lr = 3e-2 n_epochs = 5 learn.fit_one_cycle(n_epochs, lr) learn.unfreeze() learn.fit_one_cycle(n_epochs*2, slice(lr / 10 / (2.6 ** 4), lr / 10))
Let us know if you are running into any problems, preferably together with your code and stack trace!
1 Like
Thank you for your quick response. I have tried a similar approach but have running into a “RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!”
I am trying to figure out how to add my code like that, sorry I am new to this forum
dls_clas = DataBlock(
blocks=(TextBlock.from_df('MS', vocab=dls.vocab),RegressionBlock),
get_x = ColReader('text'), get_y=ColReader('Target'),
splitter=RandomSplitter(.2)).dataloaders(data, bs=128, seq_len=72)
learn = text_classifier_learner(dls_clas, AWD_LSTM, loss_func=rmse,drop_mult=0.5,y_range=(1,6),
metrics=rmse).to_fp16()
if default_device().type == 'cuda':
learn = learn.to_fp16()
learn = learn.load_encoder('finetuned')
learn.fit_one_cycle(1, 2e-2)Hey,
That error means that you’re mixing up CPU and GPU - can you check whether you have a GPU and whether pytorch recognizes it? You can also manually move the model and the data to the correct device if needed and then you won’t get that error anymore.
Yes, I am using Paperspace with a GPU and am able to the run the code from the class successfully. It is using a Quadro m4000 and says that Pytorch is using the GPU (when I do
torch.cuda.is_available())
Hey,
The best resource for NLP with fastai is the documentation - especially this text tutorial. You should upgrade your fastai installation to the latest pip install -U fastai fastcore, follow the same procedure as in the tutorial, and plug in something like the code chunk I gave you above when you perform the final text regression step. You can do all this in a Colab notebook first if you would like to make sure it’s working without having to pay any money.
If it doesn’t work, please paste your full code and any errors you get so we can help you debug it
Thank you, I have tried following this guide and have received the same error that I mentioned above. It appears like the error is coming because of when I do loss_func = rmse. But when I change it to MSE, I receive a different error.
dls_lm = DataBlock(blocks=(TextBlock.from_df('Mission Statement',is_lm=True)),
get_x=ColReader('text'), splitter=RandomSplitter(.2)).dataloaders(df,bs=128, seq_len=50)
learn = language_model_learner( dls_lm, AWD_LSTM, drop_mult=0.3, metrics=[accuracy, Perplexity()]).to_fp16()
learn.fit_one_cycle(1, 2e-2)
learn.save('1epoch')
learn = learn.load('1epoch')
learn.unfreeze()
learn.fit_one_cycle(10, 2e-3)
learn.save_encoder('finetuned'
dls_clas = DataBlock(
blocks=(TextBlock.from_df('Mission Statement', vocab=dls_lm.vocab),RegressionBlock),
get_x = ColReader('text'), get_y=ColReader('xOTIOverall'),
splitter=RandomSplitter(.2)
).dataloaders(df, bs=128, seq_len=72)
learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5, loss_func=rmse, metrics=[R2Score(),rmse])
if default_device().type == 'cuda':
learn = learn.to_fp16()
learn = learn.load_encoder('finetuned')
learn.fit_one_cycle(1, 2e-2)
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-94-07c3fbbfd15e> in <module>
----> 1 learn.fit_one_cycle(1, 2e-2)
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)
110 scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
111 'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}
--> 112 self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
113
114 # Cell
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
203 self.opt.set_hypers(lr=self.lr if lr is None else lr)
204 self.n_epoch = n_epoch
--> 205 self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
206
207 def _end_cleanup(self): self.dl,self.xb,self.yb,self.pred,self.loss = None,(None,),(None,),None,None
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
152
153 def _with_events(self, f, event_type, ex, final=noop):
--> 154 try: self(f'before_{event_type}') ;f()
155 except ex: self(f'after_cancel_{event_type}')
156 finally: self(f'after_{event_type}') ;final()
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_fit(self)
194 for epoch in range(self.n_epoch):
195 self.epoch=epoch
--> 196 self._with_events(self._do_epoch, 'epoch', CancelEpochException)
197
198 def fit(self, n_epoch, lr=None, wd=None, cbs=None, reset_opt=False):
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
152
153 def _with_events(self, f, event_type, ex, final=noop):
--> 154 try: self(f'before_{event_type}') ;f()
155 except ex: self(f'after_cancel_{event_type}')
156 finally: self(f'after_{event_type}') ;final()
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_epoch(self)
188
189 def _do_epoch(self):
--> 190 self._do_epoch_train()
191 self._do_epoch_validate()
192
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_epoch_train(self)
180 def _do_epoch_train(self):
181 self.dl = self.dls.train
--> 182 self._with_events(self.all_batches, 'train', CancelTrainException)
183
184 def _do_epoch_validate(self, ds_idx=1, dl=None):
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
152
153 def _with_events(self, f, event_type, ex, final=noop):
--> 154 try: self(f'before_{event_type}') ;f()
155 except ex: self(f'after_cancel_{event_type}')
156 finally: self(f'after_{event_type}') ;final()
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in all_batches(self)
158 def all_batches(self):
159 self.n_iter = len(self.dl)
--> 160 for o in enumerate(self.dl): self.one_batch(*o)
161
162 def _do_one_batch(self):
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in one_batch(self, i, b)
176 self.iter = i
177 self._split(b)
--> 178 self._with_events(self._do_one_batch, 'batch', CancelBatchException)
179
180 def _do_epoch_train(self):
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
152
153 def _with_events(self, f, event_type, ex, final=noop):
--> 154 try: self(f'before_{event_type}') ;f()
155 except ex: self(f'after_cancel_{event_type}')
156 finally: self(f'after_{event_type}') ;final()
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_one_batch(self)
164 self('after_pred')
165 if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
--> 166 self('after_loss')
167 if not self.training or not len(self.yb): return
168 self('before_backward')
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in __call__(self, event_name)
130 def ordered_cbs(self, event): return [cb for cb in sort_by_run(self.cbs) if hasattr(cb, event)]
131
--> 132 def __call__(self, event_name): L(event_name).map(self._call_one)
133
134 def _call_one(self, event_name):
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastcore/foundation.py in map(self, f, gen, *args, **kwargs)
218 def range(cls, a, b=None, step=None): return cls(range_of(a, b=b, step=step))
219
--> 220 def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
221 def argwhere(self, f, negate=False, **kwargs): return self._new(argwhere(self, f, negate, **kwargs))
222 def filter(self, f=noop, negate=False, gen=False, **kwargs):
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastcore/basics.py in map_ex(iterable, f, gen, *args, **kwargs)
603 res = map(g, iterable)
604 if gen: return res
--> 605 return list(res)
606
607 # Cell
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastcore/basics.py in __call__(self, *args, **kwargs)
593 if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
594 fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 595 return self.func(*fargs, **kwargs)
596
597 # Cell
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _call_one(self, event_name)
134 def _call_one(self, event_name):
135 assert hasattr(event, event_name), event_name
--> 136 [cb(event_name) for cb in sort_by_run(self.cbs)]
137
138 def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in <listcomp>(.0)
134 def _call_one(self, event_name):
135 assert hasattr(event, event_name), event_name
--> 136 [cb(event_name) for cb in sort_by_run(self.cbs)]
137
138 def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/callback/core.py in __call__(self, event_name)
42 (self.run_valid and not getattr(self, 'training', False)))
43 res = None
---> 44 if self.run and _run: res = getattr(self, event_name, noop)()
45 if event_name=='after_fit': self.run=True #Reset self.run to True at each end of fit
46 return res
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/callback/rnn.py in after_loss(self)
31 def after_loss(self):
32 if not self.training: return
---> 33 if self.alpha != 0.: self.learn.loss += self.alpha * self.out.float().pow(2).mean()
34 if self.beta != 0.:
35 h = self.raw_out
/opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/torch_core.py in __torch_function__(self, func, types, args, kwargs)
315
316 def __torch_function__(self, func, types, args=(), kwargs=None):
--> 317 with torch._C.DisableTorchFunction(): ret = _convert(func(*args, **(kwargs or {})), self.__class__)
318 if isinstance(ret, TensorBase): ret.set_meta(self, as_copy=True)
319 return ret
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!Hey,
It seems we are not getting anywhere. Like I said before, you can move your data and model between devices as needed. If you would like for me to be able to help you debug an error like that, then you need to provide fully reproducible code, such as a Colab notebook (which includes access to your dataset or a similar one). Chances are something is wrong with your environment / data / code, and it’s not possible to help you unless you provide that.
Sorry I meant that I did switch environments multiple times, from Paperspace to Google Colab and reran the code from scratch following the guides and still received the errors. Looking into it in more detail, it looks like the problem comes from setting the loss_func to RMSE. When I wrote my own RMSE error function, I no longer receive that error.