Is the language model broken in version 1.0.38?

Part 1 (2019)
1

In the official notebook and when I first ran it, the prediction for sentence i liked this movie because was:

i liked this movie because it was clearly a movie . xxmaj so i gave it a 2 out of 10 . xxmaj so , just say something . xxbos xxmaj this is a really stunning picture , light years off of the late xxmaj
i liked this movie because it would be a good one for those who like deep psychological and drama and you can go see this movie if you like a little slow motion and some magic should n't be there . i would give it

And now on 1.0.38 when I run the same notebook I get following output:

i liked this movie because gloating ligabue starlet stinko ruzowitzky dives due thatcherite cable jafri battlespace delinda fetisov sind elena loggerheads arguing gta growers kidnappings bank horler pinscreen ritson walter substantiate missiles jann handgun readers desecrate thingies bewilderingly habibi mounting provincial horaire spazz rigs retake
i liked this movie because ffolkes loftier cleverly ascots sesshomaru boppers munkar dishwater elephantine reportedly keach slipping poet pringle translators valedictorian alway interpersonal screwdriver howver bunuels cowan breckinridge predominantly deviated gall vasili jouvet spoon vine yearling festers honoured exploit boutonnat hazlehurst astoundingly pixelization hardgore learns

Same problem is also reported by @tank13 here.

2

Indeed it’s broken. This came from the fact the softmax was taken on the wrong dimension with our latest changes, I’ve fixed it in master (this commit).

3 Likes
3

Thank You Sylvain. When can we expect 1.0.39 to be out?

4

I’d say soon-ish, will discuss with Jeremy.

5

Now :slight_smile:

2 Likes
6

I am calling the new FBeta metric as follows:

learn = text_classifier_learner(data_clas, drop_mult=0.5)
learn.load_encoder('fine_tuned_enc')
f1_label = FBeta()
learn.metrics=[accuracy, f1_label]
learn.freeze()

Getting below error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-38-31b16a16d38f> in <module>
----> 1 learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/train.py in fit_one_cycle(learn, cyc_len, max_lr, moms, div_factor, pct_start, wd, callbacks, **kwargs)
     20     callbacks.append(OneCycleScheduler(learn, max_lr, moms=moms, div_factor=div_factor,
     21                                         pct_start=pct_start, **kwargs))
---> 22     learn.fit(cyc_len, max_lr, wd=wd, callbacks=callbacks)
     23 
     24 def lr_find(learn:Learner, start_lr:Floats=1e-7, end_lr:Floats=10, num_it:int=100, stop_div:bool=True, **kwargs:Any):

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/basic_train.py in fit(self, epochs, lr, wd, callbacks)
    170         callbacks = [cb(self) for cb in self.callback_fns] + listify(callbacks)
    171         fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics,
--> 172             callbacks=self.callbacks+callbacks)
    173 
    174     def create_opt(self, lr:Floats, wd:Floats=0.)->None:

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     92     except Exception as e:
     93         exception = e
---> 94         raise e
     95     finally: cb_handler.on_train_end(exception)
     96 

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     87             if not data.empty_val:
     88                 val_loss = validate(model, data.valid_dl, loss_func=loss_func,
---> 89                                        cb_handler=cb_handler, pbar=pbar)
     90             else: val_loss=None
     91             if cb_handler.on_epoch_end(val_loss): break

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/basic_train.py in validate(model, dl, loss_func, cb_handler, pbar, average, n_batch)
     52             if not is_listy(yb): yb = [yb]
     53             nums.append(yb[0].shape[0])
---> 54             if cb_handler and cb_handler.on_batch_end(val_losses[-1]): break
     55             if n_batch and (len(nums)>=n_batch): break
     56         nums = np.array(nums, dtype=np.float32)

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/callback.py in on_batch_end(self, loss)
    237         "Handle end of processing one batch with `loss`."
    238         self.state_dict['last_loss'] = loss
--> 239         stop = np.any(self('batch_end', not self.state_dict['train']))
    240         if self.state_dict['train']:
    241             self.state_dict['iteration'] += 1

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/callback.py in __call__(self, cb_name, call_mets, **kwargs)
    185     def __call__(self, cb_name, call_mets=True, **kwargs)->None:
    186         "Call through to all of the `CallbakHandler` functions."
--> 187         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
    188         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    189 

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/callback.py in <listcomp>(.0)
    185     def __call__(self, cb_name, call_mets=True, **kwargs)->None:
    186         "Call through to all of the `CallbakHandler` functions."
--> 187         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
    188         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    189 

~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/metrics.py in on_batch_end(self, last_output, last_target, **kwargs)
    116     def on_batch_end(self, last_output:Tensor, last_target:Tensor, **kwargs):
    117         preds = last_output.argmax(-1).view(-1)
--> 118         cm = ((preds==self.x[:, None]) & (last_target==self.x[:, None, None])).sum(dim=2, dtype=torch.float32)
    119         self.cm += cm
    120 

RuntimeError: Expected object of backend CUDA but got backend CPU for argument #2 'other'

Is the FBeta usage correct?

7

Hey @nikhil_no_1. Keep in mind that the default for beta=2 and for average="binary" in this implementation. You are most likely not interested in the binary case so you should rather pick something like this:

f1 = FBeta(beta=1, average="macro")

Btw there was an issue regarding the computation of some metrics when utilizing a gpu. I did the testing on a cpu exclusively so I totally missed that.
Plus you had to specify the number of classes in n_classes but this necessity should be removed and the issue should be fixed now.
Not sure if the updated version is already out.