Language Model Learner with Mid-level API

fastai
1

Hi all. I’m trying to adapt the code in Fastbook 11_midlevel_data to customize the language model learner, but I think I’m not understanding how all the pieces go together. The reason for this is I would like to eventually add in a custom Transform to standardize certain language in my dataset, hopefully increasing the signal around certain tokens by decreasing the analogous vocabulary to be learned.

The code I have thus far to check the pipeline is below. It works through dls_lm.show_batch() but breaks when I try to run the Learner.

files = get_text_files(path, folders = folders)
cut = int(len(files)*0.8)
splits = [list(range(cut)), list(range(cut,len(files)))]


tfms = [Tokenizer.from_folder(path), Numericalize]
dsets = Datasets(files, [tfms, tfms], splits=splits)
dls_lm = LMDataLoader(dsets, shuffle=True, bs=32)
dls_lm.show_batch(max_n=2)

learn = language_model_learner(
    dls_lm, AWD_LSTM, drop_mult=0.3, 
    metrics=[accuracy, Perplexity()]).to_fp16()
learn.fit_one_cycle(1, 2e-2)
2

Can you share the entire stacktrace of the error? Also, maybe you’ve seen it, but the wikitext tutorial also gives a nice breakdown of the mid level api

3

Here’s the stacktrace. Thank you for the wiki example, I’m reviewing now.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-1399d04bd20f> in <module>
     12     dls_lm, AWD_LSTM, drop_mult=0.3,
     13     metrics=[accuracy, Perplexity()]).to_fp16()
---> 14 learn.fit_one_cycle(1, 2e-2)

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/logargs.py in _f(*args, **kwargs)
     54         init_args.update(log)
     55         setattr(inst, 'init_args', init_args)
---> 56         return inst if to_return else f(*args, **kwargs)
     57     return _f

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)
    111     scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
    112               'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}
--> 113     self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
    114 
    115 # Cell

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/logargs.py in _f(*args, **kwargs)
     54         init_args.update(log)
     55         setattr(inst, 'init_args', init_args)
---> 56         return inst if to_return else f(*args, **kwargs)
     57     return _f

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    205             self.opt.set_hypers(lr=self.lr if lr is None else lr)
    206             self.n_epoch = n_epoch
--> 207             self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
    208 
    209     def _end_cleanup(self): self.dl,self.xb,self.yb,self.pred,self.loss = None,(None,),(None,),None,None

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    153 
    154     def _with_events(self, f, event_type, ex, final=noop):
--> 155         try:       self(f'before_{event_type}')       ;f()
    156         except ex: self(f'after_cancel_{event_type}')
    157         finally:   self(f'after_{event_type}')        ;final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in __call__(self, event_name)
    131     def ordered_cbs(self, event): return [cb for cb in sort_by_run(self.cbs) if hasattr(cb, event)]
    132 
--> 133     def __call__(self, event_name): L(event_name).map(self._call_one)
    134 
    135     def _call_one(self, event_name):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/foundation.py in map(self, f, gen, *args, **kwargs)
    224     def range(cls, a, b=None, step=None): return cls(range_of(a, b=b, step=step))
    225 
--> 226     def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
    227     def argwhere(self, f, negate=False, **kwargs): return self._new(argwhere(self, f, negate, **kwargs))
    228     def filter(self, f=noop, negate=False, gen=False, **kwargs):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/basics.py in map_ex(iterable, f, gen, *args, **kwargs)
    541     res = map(g, iterable)
    542     if gen: return res
--> 543     return list(res)
    544 
    545 # Cell

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/basics.py in __call__(self, *args, **kwargs)
    531             if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
    532         fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 533         return self.fn(*fargs, **kwargs)
    534 
    535 # Cell

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _call_one(self, event_name)
    135     def _call_one(self, event_name):
    136         assert hasattr(event, event_name), event_name
--> 137         [cb(event_name) for cb in sort_by_run(self.cbs)]
    138 
    139     def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in <listcomp>(.0)
    135     def _call_one(self, event_name):
    136         assert hasattr(event, event_name), event_name
--> 137         [cb(event_name) for cb in sort_by_run(self.cbs)]
    138 
    139     def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/core.py in __call__(self, event_name)
     42                (self.run_valid and not getattr(self, 'training', False)))
     43         res = None
---> 44         if self.run and _run: res = getattr(self, event_name, noop)()
     45         if event_name=='after_fit': self.run=True #Reset self.run to True at each end of fit
     46         return res

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/fp16.py in before_fit(self)
     84 
     85     def before_fit(self):
---> 86         assert self.dls.device.type == 'cuda', "Mixed-precision training requires a GPU, remove the call `to_fp16`"
     87         if self.learn.opt is None: self.learn.create_opt()
     88         self.model_pgs,self.master_pgs = get_master(self.opt, self.flat_master)

AttributeError: 'NoneType' object has no attribute 'type'
4

Ah I got it! The answer was in the Wikitext example. I think duplicating the tfms was the problem, instead of just creating a single-item list. This code works:

files = get_text_files(path, folders = folders)
cut = int(len(files)*0.8)
splits = [list(range(cut)), list(range(cut,len(files)))]


tfms = [Tokenizer.from_folder(path), Numericalize]
dsets = Datasets(files, [tfms], splits=splits, dl_type=LMDataLoader)
dls_lm = dsets.dataloaders(bs=32, seq_len=80)

dls_lm.show_batch(max_n=2)

learn = language_model_learner(
    dls_lm, AWD_LSTM, drop_mult=0.3, 
    metrics=[accuracy, Perplexity()]).to_fp16()
learn.fit_one_cycle(1, 2e-2)

I had also started to munge something together using the DataBlocks API, but was struggling to figure out how to change the output_dir so I can save various tokenizer iterations. I think it’s not passing through right. But in any case I think the above code is cleaner than what I was working on:

items = partial(get_text_files, folders=folders)

dls_lm = DataBlock(
    blocks=TextBlock.from_folder(path, is_lm=True, 
                                 rules=rules, 
                                 skip_if_exists=False),
    get_items=items, splitter=RandomSplitter(0.1),
).dataloaders(path, path=path.parent, bs=128, seq_len=80, shuffled=True)
2 Likes
5

cool, re output_dir, did you pass it to TextBlock.from_folder()? At a glance it looks like it should pass through from there to tokenize_folder where it is needed…

6

I haven’t tried the tokenization mods yet, but I believe I sorted out that I could pass it directly to Tokenizer if I keep things somewhat separated going into the Dataloaders.