Hi All,

Apologies for the delay, I’ve got too many side projects going on, and training the language model took me about 30 mins / epoch, which I will have to investigate later. Here’s the notes from lecture 10. Looks like the readling list is growing with some good paper suggestions!

  • Tim

Lesson 10

NLP Classification and Translation

We are going to look at NLP this week

Review of Object Detection:

  • See Lesson 8 / Lesson 9



Recall in image classification, with conv_learner it has a standard way to stick a layer ontop (any custom head) and have it do anything we like. Now we have flexibility to understand rotations, or solve other interesting problems. Lets take this concept and apply this to text.

NLP + Computer vision techniques Roadmap:

  • Learn to find word structures from images aka, captioning
  • Learn to find images to word structures
  • Learn to find images from images
  • Image enhancement - upscaling low photos into high res
  • Segmenting a picture into objects

torchtext to fastai.text

torchtext is great, but its slow, it doesn’t run in parallel, and it doesn’t remember. fastai.text is a combination of torchtext and fastai.nlp. Consider fastai.nlp deprecated (outdated) at this point.

Import libraries

import sys
from fastai.text import *
import html

IMDB - Internet Movie Database

weblink to python notebook

Get the dataset

!tar -xf aclImdb.tgz

Check the file directories:

!ls ~/data/aclImdb/
    ->imdbEr.txt  imdb.vocab	README	test  train

Check the file counts

!ls /home/paperspace/data/aclImdb/train/all | wc -l
!ls /home/paperspace/data/aclImdb/test/all | wc -l

Look at directories

!ls /home/paperspace/data/aclImdb/
imdbEr.txt  imdb.vocab	models	README	test  tmp  train
!ls /home/paperspace/data/aclImdb/train
all		 neg  unsup	     urls_neg.txt  urls_unsup.txt
labeledBow.feat  pos  unsupBow.feat  urls_pos.txt

Create a CLAS - classification and LM - language model

# create paths to save future features

BOS = 'xbos'  # beginning-of-sentence tag
FLD = 'xfld'  # data field tag
CLASSES = ['neg', 'pos', 'unsup']


Turning Text into Numbers

  • How do we turn sentences into numbers?

First lets prepare the imdb dataset, by creating a panda dataframe from the collection of .txt files

def get_texts(path):
    This function will go through the aclImdb folder
    and create the necessary datasets
    # initializes the text and labels collections
    texts,labels = [],[]

    # for each sentiment
    for idx,label in enumerate(CLASSES):
        # will go through the 
        for fname in (path/label).glob('*.*'):
            # open the file and append the filetext
            # open 
    return np.array(texts),np.array(labels)

trn_texts,trn_labels = get_texts(PATH/'train')
val_texts,val_labels = get_texts(PATH/'test')

Look at a sample of the text

 "Basically, Cruel Intentions 2 is Cruel Intentions 1, again, only poorly done. The story is exactly the same as the first one (even some of the lines), with only a few exceptions. The cast is more unknown, and definitely less talented. Instead of being seductive and drawing me into watching it, I ended up feeling dirty because it compares to watching a soft-core porn. I'm not sure whether to blame some of the idiotic lines on the actors or the writers...and I always feel bad saying that, because I know how hard it is to do both...but it was basically a two-hour waste of my life. It literally amazes me that some movies get made, and this is no exception...I can't believe they'd make a third one.")

Prepare dataframe

col_names = ['labels','text']

# shuffle the indexes in place
trn_idx = np.random.permutation(len(trn_texts))
val_idx = np.random.permutation(len(val_texts))

#shuffle texts
trn_texts = trn_texts[trn_idx]
val_texts = val_texts[val_idx]

#shuffle the labels 
trn_labels = trn_labels[trn_idx]
val_labels = val_labels[val_idx]

#create dataframe
df_trn = pd.DataFrame({'text':trn_texts, 'labels':trn_labels}, columns=col_names)
df_val = pd.DataFrame({'text':val_texts, 'labels':val_labels}, columns=col_names)

# saving training and validation dataset
df_trn[df_trn['labels']!=2].to_csv(CLAS_PATH/'train.csv', header=False, index=False)
df_val.to_csv(CLAS_PATH/'test.csv', header=False, index=False)

#write the classes
(CLAS_PATH/'classes.txt').open('w').writelines(f'{o}\n' for o in CLASSES)

Create trn and validation texts with different test size

from sklearn.model_selection import train_test_split

trn_texts,val_texts = train_test_split(np.concatenate([trn_texts,val_texts]), test_size=0.1)
len(trn_texts), len(val_texts)
(90000, 10000)

Create Language model LM dataframes and save

df_trn = pd.DataFrame({'text':trn_texts, 'labels':[0]*len(trn_texts)}, columns=col_names)
df_val = pd.DataFrame({'text':val_texts, 'labels':[0]*len(val_texts)}, columns=col_names)

df_trn.to_csv(LM_PATH/'train.csv', header=False, index=False)
df_val.to_csv(LM_PATH/'test.csv', header=False, index=False)

Language Model Tokens

  • tokenization - turn a sentence into words with some specific rules, correcting for puncutation.
  • fixup() - there’s always strange encodings in text, here are a couple that will replace some outlier characters
  • get_texts() - will iterate through all files and collect the in-file text
  • get_all() - will call get_texts() repeatedly for each row of the dataframe, pulling text from the source text files and returning tokens and labels


  • Tokenizer().proc_all_mp(partition_by_cores(texts))

proc_all_mp function

def proc_all_mp(ss, lang='en'):
        ncpus = num_cpus()//2
        with ProcessPoolExecutor(ncpus) as e:
            return sum(, ss, [lang]*len(ss)), [])

partition_by_cores function

def partition_by_cores(a):
    return partition(a, len(a)//num_cpus() + 1)

ProcessPoolExecutor is from a Python 3 standard library:
More on multiprocessing

# generalized use of ProcessPoolExecutor
import concurrent.futures
def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES,, PRIMES)):
            print('%d is prime: %s' % (number, prime))
when given to pandas, it won't return a full dataframe, 
but it will return an iterator. It will return sub-sized chunks
over and over again
re1 = re.compile(r'  +')

def fixup(x):
    """ Cleans up erroroneus characters"""
    x = x.replace('#39;', "'").replace('amp;', '&').replace('#146;', "'").replace(
        'nbsp;', ' ').replace('#36;', '$').replace('\\n', "\n").replace('quot;', "'").replace(
        '<br />', "\n").replace('\\"', '"').replace('<unk>','u_n').replace(' @.@ ','.').replace(
        ' @-@ ','-').replace('\\', ' \\ ')
    return re1.sub(' ', html.unescape(x))

def get_texts(df, n_lbls=1):
    # pull the labels out from the dataframe
    labels = df.iloc[:,range(n_lbls)].values.astype(np.int64)
    # pull the full FILEPATH for each text
    # BOS is a flag to indicate when a new text is starting
    texts = f'\n{BOS} {FLD} 1 ' + df[n_lbls].astype(str)
    # Sometimes, text has title, or other sub-sections. We will record all of these
    for i in range(n_lbls+1, len(df.columns)): texts += f' {FLD} {i-n_lbls} ' + df[i].astype(str)
    texts = texts.apply(fixup).values.astype(str)

    # Tokenize the data
    tok = Tokenizer().proc_all_mp(partition_by_cores(texts))
    return tok, list(labels)

def get_all(df, n_lbls):
    tok, labels = [], []
    for i, r in enumerate(df):
        tok_, labels_ = get_texts(r, n_lbls)
        tok += tok_;
        labels += labels_
    return tok, labels

Install Spacy if you don’t have it

!pip install spacy
Requirement already satisfied: spacy in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages
Requirement already satisfied: numpy>=1.7 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: murmurhash<0.29,>=0.28 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: cymem<1.32,>=1.30 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: thinc<6.11.0,>=6.10.1 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: six in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: html5lib==1.0b8 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: pathlib in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: ujson>=1.35 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: dill<0.3,>=0.2 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: regex==2017.4.5 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: ftfy<5.0.0,>=4.4.2 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: msgpack-python==0.5.4 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: msgpack-numpy==0.4.1 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from spacy)
Requirement already satisfied: wrapt in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from thinc<6.11.0,>=6.10.1->spacy)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from thinc<6.11.0,>=6.10.1->spacy)
Requirement already satisfied: cytoolz<0.9,>=0.8 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from thinc<6.11.0,>=6.10.1->spacy)
Requirement already satisfied: termcolor in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from thinc<6.11.0,>=6.10.1->spacy)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: idna<2.7,>=2.5 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: certifi>=2017.4.17 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: wcwidth in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy)
Requirement already satisfied: toolz>=0.8.0 in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages (from cytoolz<0.9,>=0.8->thinc<6.11.0,>=6.10.1->spacy)

Download English Model

!python -m spacy download en
  Downloading (37.4MB)
e[K    100% |████████████████████████████████| 37.4MB 6.7MB/s ta 0:00:0111
e[?25h  Requirement already satisfied (use --upgrade to upgrade): en-core-web-sm==2.0.0 from in /home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages

e[93m    Linking successfule[0m

    You can now load the model via spacy.load('en')
import spacy
nlp = spacy.load('en')

Read in our Language model data, extract texts

 __spec__ = "ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>)"
df_trn = pd.read_csv(LM_PATH/'train.csv', header=None, chunksize=chunksize)
df_val = pd.read_csv(LM_PATH/'test.csv', header=None, chunksize=chunksize)

tok_trn, trn_labels = get_all(df_trn, 1)
tok_val, val_labels = get_all(df_val, 1)

Make working tmp directory & save tokens

(LM_PATH/'tmp').mkdir(exist_ok=True)'tmp'/'tok_trn.npy', tok_trn)'tmp'/'tok_val.npy', tok_val)

Load tokenization & check top tokens

tok_trn = np.load(LM_PATH/'tmp'/'tok_trn.npy')
tok_val = np.load(LM_PATH/'tmp'/'tok_val.npy')

freq = Counter(p for o in tok_trn for p in o)
[('the', 1208449),
 ('.', 992545),
 (',', 986614),
 ('and', 587567),
 ('a', 583520),
 ('of', 525412),
 ('to', 484871),
 ('is', 393923),
 ('it', 341485),
 ('in', 337351),
 ('i', 307751),
 ('this', 270410),
 ('that', 261107),
 ('"', 237920),
 ("'s", 222037),
 ('-', 188209),
 ('was', 180235),
 ('\n\n', 179009),
 ('as', 166145),
 ('with', 159253),
 ('for', 158601),
 ('movie', 157735),
 ('but', 150659),
 ('film', 144618),
 ('you', 123979)]

recreate a single entry

  • xbos - start stream
  • xfld - start field
  • word on caps - how do get the semantic impact of CAPS vs. normal version?
  • t_up - add a tag in front to indicate the next word is uppercase
' '.join(tok_trn[0])
'\n xbos xfld 1 i saw this movie at the dragon*con 2006 independent film festival . it was awarded 2 awards at that festival and rightfully so . this is probably the best short horror film i \'ve ever seen . the simplicity of camera usage really works.the main character is brilliant . his acting is quite good and is believable . the 3 cameras in the room with tim russel make his insanity that much more believable . i love it . i have talked with mike and he says that they are in the process of making a feature film compassing the first three chapters together . i ca n\'t wait . i will be first in line for that film . the effects of the " mirror " creatures are used so well . you do n\'t see them for very long so it scares the pants off of you when you do . i recommend this film to anyone who wants to watch a good horror movie for once . best 32 minutes of spine tingling horror i \'ve ever seen . thanks mike .'

Setup the Vocabulary and add a term for unknown and for padding

max_vocab = 60000
min_freq = 2

# index to word
itos = [o for o,c in freq.most_common(max_vocab) if c>min_freq]
itos.insert(0, '_pad_')
itos.insert(0, '_unk_')

# word to index
stoi = collections.defaultdict(lambda:0, {v:k for k,v in enumerate(itos)})

# create a array of token_indices 
trn_lm = np.array([[stoi[o] for o in p] for p in tok_trn])
val_lm = np.array([[stoi[o] for o in p] for p in tok_val])

# save the i'tmp'/'trn_ids.npy', trn_lm)'tmp'/'val_ids.npy', val_lm)
pickle.dump(itos, open(LM_PATH/'tmp'/'itos.pkl', 'wb'))

Example of a numerical representation

' '.join([str(val) for val in trn_lm[0]])
'40 41 42 39 12 235 13 23 44 2 0 3368 1662 25 1331 3 10 18 8833 261 2474 44 14 1331 5 9383 51 3 13 9 263 2 138 364 200 25 12 159 143 129 3 2 5416 7 371 9080 83 58696 305 122 9 556 3 35 136 9 204 66 5 9 842 3 2 379 3992 11 2 655 21 1895 13095 113 35 4853 14 93 67 842 3 12 133 10 3 12 36 3515 21 1555 5 34 566 14 45 33 11 2 1648 7 251 6 820 25 0 2 105 299 8000 312 3 12 196 29 881 3 12 104 37 105 11 367 22 14 25 3 2 306 7 2 15 3006 15 2204 33 345 51 88 3 26 57 29 82 111 22 69 216 51 10 2702 2 3719 141 7 26 68 26 57 3 12 404 13 25 8 273 48 505 8 126 6 66 200 23 22 301 3 138 13558 249 7 7191 21684 200 12 159 143 129 3 1179 1555 3'

Load from checkpoint

trn_lm = np.load(LM_PATH/'tmp'/'trn_ids.npy')
val_lm = np.load(LM_PATH/'tmp'/'val_ids.npy')
itos = pickle.load(open(LM_PATH/'tmp'/'itos.pkl', 'rb'))

(60002, 90000)

Instead of pretraining on imagenet, for NLP we will use a large subset of wikipedia

Previously in lesson 4, we trained a language model that was state of the art. Using a pre-selected text articles from wikipedia, a language model was trained, and the weights were saved.

wikitext103 model - weights

#! wget -nH -r -np -P {PATH}

How to use a pretrained model - must have same network sizes

  • em_sz - embedding sizes for vectors (400)

  • nh - number of hidden (# of activations)

  • nl - number of layers (hidden)

  • model type: AWD LSTM link

Some work will have to be done to map the pre-trained vocabulary to the current vocab that we are working with. Any words not found in pre-trained vocab, we will use global mean values.

em_sz,nh,nl = 400,1150,3

# set filepaths
PRE_PATH = PATH/'models'/'wt103'
PRE_LM_PATH = PRE_PATH/'fwd_wt103.h5'

# load weights (returns a dictionary)
wgts = torch.load(PRE_LM_PATH, map_location=lambda storage, loc: storage)

# pull out embedding weights
# sized vocab x em_sz 
enc_wgts = to_np(wgts['0.encoder.weight'])
row_m = enc_wgts.mean(0)

# load pre trained vocab to index mappings
itos2 = pickle.load((PRE_PATH/'itos_wt103.pkl').open('rb'))
stoi2 = collections.defaultdict(lambda:-1, {v:k for k,v in enumerate(itos2)})

# create a pre-trained -> current corpus vocab to vocab mapping
# initialize an empty matrix
new_w = np.zeros((vs, em_sz), dtype=np.float32)

# loop through by row index and insert the correct embedding
for i,w in enumerate(itos):
    r = stoi2[w]
    new_w[i] = enc_wgts[r] if r>=0 else row_m

# create our torch `state` that we will load later
wgts['0.encoder.weight'] = T(new_w)
wgts['0.encoder_with_dropout.embed.weight'] = T(np.copy(new_w))
wgts['1.decoder.weight'] = T(np.copy(new_w))

Language Model Creation

  • wd - weight decay
  • bptt - back prop through time
  • bs - batchsize

We will be performing a continuous process of

    given words --> predict next word

Lesson 4 - note the best loss

[ 0.      4.3926  4.2917]                                       
[ 1.       4.37693  4.28255]                                  
[ 2.       4.37998  4.27243]                                  
[ 3.       4.34284  4.24789]                                  
[ 4.      4.3287  4.2317]                                     
[ 5.       4.28881  4.20722]                                  
[ 6.       4.24637  4.18926]                                  
[ 7.       4.23797  4.17644]  

Pretrained Model - we already start with a better score

epoch      trn_loss   val_loss   accuracy                     
    0      4.332359   4.120674   0.289563  
    1      4.247177   4.067932   0.294281 

Comparison to Word2Vec

Word2Vec - single embedding matrix. Each word has a matrix and thats it. It’s a single layer (input) from a pretrained model. It’s from a linear model on a co-occurance matrix.

opt_fn = partial(optim.Adam, betas=(0.8, 0.99))

trn_dl = LanguageModelLoader(np.concatenate(trn_lm), bs, bptt)
val_dl = LanguageModelLoader(np.concatenate(val_lm), bs, bptt)
md = LanguageModelData(PATH, 1, vs, trn_dl, val_dl, bs=bs, bptt=bptt)

drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*0.7

learner= md.get_model(opt_fn, em_sz, nh, nl, 
    dropouti=drops[0], dropout=drops[1], wdrop=drops[2], dropoute=drops[3], dropouth=drops[4])

learner.metrics = [accuracy]

Language Model Components:

1. DataLoader

2. ModelData

3. Model

1. Language Model Loader

Key Notes:

  • Batch size is not the count of things in a batch. In this context, batch_size is more batch_ct
  • Randomness-> size instead of order - if we grab 70 at a time, and do a new epoch, the data will be exactly the same, and all the batches will be identical. In images we would shuffle, but that doesn’t work in the language model, because it is trying to learn the sentence. So if order can’t change, let’s randomly change the sequence length
class LanguageModelLoader():
    """Returns tuples of mini-batches."""
    def __init__(self, nums, bs, bptt, backwards=False):
        # assign values,self.bptt,self.backwards = bs,bptt,backwards
        # batchify the numbers. Based on the batchsize
        # subdivide the data. Note: batchsize 64
        # 640,000 would be broken into 64 x 10,000 = self.batchify(nums)
        # initialize other values
        self.i,self.iter = 0,0
        self.n = len(

    def __iter__(self):
        """ Iterator implementation"""
        # start from zero
        self.i,self.iter = 0,0
        # will continually pull data out
        while self.i < self.n-1 and self.iter<len(self):
            if self.i == 0:
                seq_len = self.bptt + 5 * 5
                bptt = self.bptt if np.random.random() < 0.95 else self.bptt / 2.
                seq_len = max(5, int(np.random.normal(bptt, 5)))
            res = self.get_batch(self.i, seq_len)
            self.i += seq_len
            self.iter += 1
            # yields the value
            yield res

    def __len__(self): return self.n // self.bptt - 1

    def batchify(self, data):
        """splits the data into batch_size counts of sets"""
        nb = data.shape[0] //
        data = np.array(data[:nb*])
        data = data.reshape(, -1).T
        if self.backwards: data=data[::-1]
        # returns the transpose
        # have batch_size number of columns 
        return T(data)

    def get_batch(self, i, seq_len):
        source =
        seq_len = min(seq_len, len(source) - 1 - i)
        return source[i:i+seq_len], source[i+1:i+1+seq_len].view(-1)


class LanguageModelData():
    - a training data loader
    - a validation data loader
    - a test loader
    - a saving path
    - model parameteres
    def __init__(self, path, pad_idx, nt, trn_dl, val_dl, test_dl=None, bptt=70, backwards=False, **kwargs):
        self.path,self.pad_idx,self.nt = path,pad_idx,nt
        self.trn_dl,self.val_dl,self.test_dl = trn_dl,val_dl,test_dl

    def get_model(self, opt_fn, emb_sz, n_hid, n_layers, **kwargs):
        m = get_language_model(self.nt, emb_sz, n_hid, n_layers, self.pad_idx, **kwargs)
        model = LanguageModel(to_gpu(m))
        return RNN_Learner(self, model, opt_fn=opt_fn)

Our Language Model extends the basic model

We are overriding one method that returns a list of all your layer groups.

class LanguageModel(BasicModel):
    def get_layer_groups(self):
        m = self.model[0]
        return [*zip(m.rnns, m.dropouths), (self.model[1], m.dropouti)]

Extend the learner class, set default to cross entropy

class RNN_Learner(Learner):
    def __init__(self, data, models, **kwargs):
        super().__init__(data, models, **kwargs)
        self.crit = F.cross_entropy

    def save_encoder(self, name): save_model(self.model[0], self.get_model_path(name))
    def load_encoder(self, name): load_model(self.model[0], self.get_model_path(name))

Review of the RNN Encoder

AWD Paper

Link to the github

  • Embedding input layer
  • 1 x LSTM layer per layer asked for nl
  • The rest are places to put dropout


  • Call the Embedding Layer
  • Add some drop out
  • Call RNN layer
  • Append to outputs
  • drop out
import torch.nn as nn

class RNN_Encoder(nn.Module):

    """A custom RNN encoder network that uses
        - an embedding matrix to encode input,
        - a stack of LSTM layers to drive the network, and
        - variational dropouts in the embedding and LSTM layers
        The architecture for this network was inspired by the work done in
        "Regularizing and Optimizing LSTM Language Models".


    def __init__(self, ntoken, emb_sz, nhid, nlayers, pad_token, bidir=False,
                 dropouth=0.3, dropouti=0.65, dropoute=0.1, wdrop=0.5):
        """ Default constructor for the RNN_Encoder class
                bs (int): batch size of input data
                ntoken (int): number of vocabulary (or tokens) in the source dataset
                emb_sz (int): the embedding size to use to encode each token
                nhid (int): number of hidden activation per LSTM layer
                nlayers (int): number of LSTM layers to use in the architecture
                pad_token (int): the int value used for padding text.
                dropouth (float): dropout to apply to the activations going from one LSTM layer to another
                dropouti (float): dropout to apply to the input layer.
                dropoute (float): dropout to apply to the embedding layer.
                wdrop (float): dropout used for a LSTM's internal (or hidden) recurrent weights.

        self.ndir = 2 if bidir else 1 = 1
        self.encoder = nn.Embedding(ntoken, emb_sz, padding_idx=pad_token)
        self.encoder_with_dropout = EmbeddingDropout(self.encoder)
        self.rnns = [nn.LSTM(emb_sz if l == 0 else nhid, (nhid if l != nlayers - 1 else emb_sz)//self.ndir,
             1, bidirectional=bidir, dropout=dropouth) for l in range(nlayers)]
        if wdrop: self.rnns = [WeightDrop(rnn, wdrop) for rnn in self.rnns]
        self.rnns = torch.nn.ModuleList(self.rnns), self.initrange)

        self.emb_sz,self.nhid,self.nlayers,self.dropoute = emb_sz,nhid,nlayers,dropoute
        self.dropouti = LockedDropout(dropouti)
        self.dropouths = nn.ModuleList([LockedDropout(dropouth) for l in range(nlayers)])

    def forward(self, input):
        """ Invoked during the forward propagation of the RNN_Encoder module.
            input (Tensor): input of shape (sentence length x batch_size)
            raw_outputs (tuple(list (Tensor), list(Tensor)): list of tensors evaluated from each RNN layer without using
            dropouth, list of tensors evaluated from each RNN layer using dropouth,
        sl,bs = input.size()
        if bs!

        emb = self.encoder_with_dropout(input, dropout=self.dropoute if else 0)
        emb = self.dropouti(emb)

        raw_output = emb
        new_hidden,raw_outputs,outputs = [],[],[]
        for l, (rnn,drop) in enumerate(zip(self.rnns, self.dropouths)):
            current_input = raw_output
            with warnings.catch_warnings():
                raw_output, new_h = rnn(raw_output, self.hidden[l])
            if l != self.nlayers - 1: raw_output = drop(raw_output)

        self.hidden = repackage_var(new_hidden)
        return raw_outputs, outputs

    def one_hidden(self, l):
        nh = (self.nhid if l != self.nlayers - 1 else self.emb_sz)//self.ndir
        return Variable(,, nh).zero_(), volatile=not

    def reset(self):
        self.weights = next(self.parameters()).data
        self.hidden = [(self.one_hidden(l), self.one_hidden(l)) for l in range(self.nlayers)]

Choosing Dropout

If you have less data for your language model, you will need more dropout. If you have more data, you will need less dropout. Otherwise the following dropout numbers were selected by experimentation:

drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*0.7

Note on : 0.7 if you are overfitting, increase the number, if you are underfitting, decrease this number

Normally we look at cross-entropy loss. But comparing CE Loss, if you are right, you should be very confident. Accuracy only cares if the answer was right or wrong, often times is more stable to track.

Back to the Language Model

opt_fn = partial(optim.Adam, betas=(0.8, 0.99))

trn_dl = LanguageModelLoader(np.concatenate(trn_lm), bs, bptt)
val_dl = LanguageModelLoader(np.concatenate(val_lm), bs, bptt)
md = LanguageModelData(PATH, 1, vs, trn_dl, val_dl, bs=bs, bptt=bptt)

drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*0.7

learner= md.get_model(opt_fn, em_sz, nh, nl, 
    dropouti=drops[0], dropout=drops[1], wdrop=drops[2], dropoute=drops[3], dropouth=drops[4])

learner.metrics = [accuracy]

note to reader - this block takes a long time to run (single epoch) - 30mins/epoch on 8 core 32 GB

import time

# fit a single cycle
lrs = lr
start = time.time(), 1, wds=wd, use_clr=(32,2), cycle_len=1)
print("time to train 1 epoch,", time.time()-start)'lm_last_ft')
HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                     
    0      4.354009   4.18011    0.285487  

time to train 1 epoch, 1856.5503034591675

this will take a while to run - 15 epochs! make sure you have the time / computing resources

# search for a learning rate, then run for 15 epoches
learner.lr_find(start_lr=lrs/10, end_lr=lrs*10, linear=True)

start = time.time(), 1, wds=wd, use_clr=(20,10), cycle_len=15)
print("Time to train,", time.time() - start)

After fitting the model we save it

We save the trained model weights and separately save the encoder part of the LM model as well. This will serve as our backbone in the classification task model.

# saves the model'lm1')

# saves just the RNN encoder (rnn_enc)

From FASTAI reference notebook - Results of the 15 epoches

epoch      trn_loss   val_loss   accuracy
    0      4.332359   4.120674   0.289563
    1      4.247177   4.067932   0.294281
    2      4.175848   4.027153   0.298062
    3      4.140306   4.001291   0.300798
    4      4.112395   3.98392    0.302663
    5      4.078948   3.971053   0.304059
    6      4.06956    3.958152   0.305356
    7      4.025542   3.951509   0.306309
    8      4.019778   3.94065    0.30756 
    9      4.027846   3.931385   0.308232
    10     3.98106    3.928427   0.309011
    11     3.97106    3.920667   0.30989 
    12     3.941096   3.917029   0.310515
    13     3.924818   3.91302    0.311015
    14     3.923296   3.908476   0.311586

Create the Classifier Tokens

The classifier model is basically a linear layer custom head on top of the LM backbone. Setting up the classifier data is similar to the LM data setup except that we cannot use the unsup movie reviews this time.

# read in the data again
df_trn = pd.read_csv(CLAS_PATH/'train.csv', header=None, chunksize=chunksize)
df_val = pd.read_csv(CLAS_PATH/'test.csv', header=None, chunksize=chunksize)

# get the tokens
tok_trn, trn_labels = get_all(df_trn, 1)
tok_val, val_labels = get_all(df_val, 1)
(CLAS_PATH/'tmp').mkdir(exist_ok=True)'tmp'/'tok_trn.npy', tok_trn)'tmp'/'tok_val.npy', tok_val)'tmp'/'trn_labels.npy', trn_labels)'tmp'/'val_labels.npy', val_labels)
tok_trn = np.load(CLAS_PATH/'tmp'/'tok_trn.npy')
tok_val = np.load(CLAS_PATH/'tmp'/'tok_val.npy')

# We load the integer to vocab that we saved before
itos = pickle.load((LM_PATH/'tmp'/'itos.pkl').open('rb'))
stoi = collections.defaultdict(lambda:0, {v:k for k,v in enumerate(itos)})

# create all matricies with indices
trn_clas = np.array([[stoi[o] for o in p] for p in tok_trn])
val_clas = np.array([[stoi[o] for o in p] for p in tok_val])

# then save the matricies'tmp'/'trn_ids.npy', trn_clas)'tmp'/'val_ids.npy', val_clas)


Now we can create our final model, a classifier which is really a custom linear head over our trained IMDB backbone. The steps to create the classifier model are similar to the ones for the LM.

# we load our numpy arrays with indexes (representing vocab)
trn_clas = np.load(CLAS_PATH/'tmp'/'trn_ids.npy')
val_clas = np.load(CLAS_PATH/'tmp'/'val_ids.npy')

# we load our labels
trn_labels = np.squeeze(np.load(CLAS_PATH/'tmp'/'trn_labels.npy'))
val_labels = np.squeeze(np.load(CLAS_PATH/'tmp'/'val_labels.npy'))

# set up our model parameters
bptt,em_sz,nh,nl = 70,400,1150,3
vs = len(itos)

# select our optimizer
# also pick a batch size as big as you can that doesn't run out of memory
opt_fn = partial(optim.Adam, betas=(0.8, 0.99))
bs = 48

min_lbl = trn_labels.min()
trn_labels -= min_lbl
val_labels -= min_lbl

In the classifier, unlike LM, we need to read a movie review at a time and learn to predict the it’s sentiment as pos/neg. We do not deal with equal bptt size batches, so we have to pad the sequences to the same length in each batch. To create batches of similar sized movie reviews, we use a sortish sampler method invented by @Smerity and @jekbradbury

The sortishSampler cuts down the overall number of padding tokens the classifier ends up seeing.

Note: If documents are different lengths, they should be padded to be the same size. Luckily fastai does this automatically

Process Optimizing Note: Put the short documents first (with some randomness)

# create basic text datasets
trn_ds = TextDataset(trn_clas, trn_labels)
val_ds = TextDataset(val_clas, val_labels)

# sort the docs based on size. 
# validation will be explicitly short -> long
# training, which sorts loosely
trn_samp = SortishSampler(trn_clas, key=lambda x: len(trn_clas[x]), bs=bs//2)
val_samp = SortSampler(val_clas, key=lambda x: len(val_clas[x]))

# then we create our dataloaders as before but with a [sampler] parameter
trn_dl = DataLoader(trn_ds, bs//2, transpose=True, num_workers=1, pad_idx=1, sampler=trn_samp)
val_dl = DataLoader(val_ds, bs, transpose=True, num_workers=1, pad_idx=1, sampler=val_samp)
md = ModelData(PATH, trn_dl, val_dl)

Load our Pretrained Model, and train the last Layer

We will pass hidden layer details

layers=[em_sz*3, 50, c]
  • em_sz*3 - inputsize
  • 50 - output of first layer
  • c - output of the 2nd layer

Why x3? - Concat pooling

We take the average pooling over the sequence, the max pooling, and the final pooling and concatenating them all together

pass in drop out details

drops=[dps[4], 0.1]

pass the AWD dropout parameters:

dropouti=dps[0], wdrop=dps[1], dropoute=dps[2], dropouth=dps[3])
# setup our dropout rates
dps = np.array([0.4,0.5,0.05,0.3,0.4])*0.5
m = get_rnn_classifer(bptt, 20*70, c, vs, emb_sz=em_sz, n_hid=nh, n_layers=nl, pad_token=1,
          layers=[em_sz*3, 50, c], drops=[dps[4], 0.1],
          dropouti=dps[0], wdrop=dps[1], dropoute=dps[2], dropouth=dps[3])

opt_fn = partial(optim.Adam, betas=(0.7, 0.99))

# define our RNN learner
learn = RNN_Learner(md, TextModel(to_gpu(m)), opt_fn=opt_fn)
learn.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learn.metrics = [accuracy]

# set our learning rate
# we will use discriminative learning rates for different layers
lrm = 2.6
lrs = np.array([lr/(lrm**4), lr/(lrm**3), lr/(lrm**2), lr/lrm, lr])

# Now we load our language model from before
# but freeze everything except the last layer
wd = 1e-7
wd = 0

# find the optimal learning rate

Train the last Layer

A Jupyter Widget
epoch      trn_loss   val_loss   accuracy                      
    0      0.365457   0.185553   0.928719, 1, wds=wd, cycle_len=1, use_clr=(8,3))'clas_0')

What if we freeze everything except the last two layers?

epoch      trn_loss   val_loss   accuracy                      
    0      0.340473   0.17319    0.933125 
learn.freeze_to(-2), 1, wds=wd, cycle_len=1, use_clr=(8,3))'clas_1')

Now lets try and train the whole model

Note that the state of the art is 0.941, which is beaten in 3-4 epoches

epoch      trn_loss   val_loss   accuracy                      
    0      0.337347   0.186812   0.930782  
    1      0.284065   0.318038   0.932062                      
    2      0.246721   0.156018   0.941747                      
    3      0.252745   0.157223   0.944106                      
    4      0.24023    0.159444   0.945393                      
    5      0.210046   0.202856   0.942858                      
    6      0.212139   0.149009   0.943746                      
    7      0.21163    0.186739   0.946553                      
    8      0.186233   0.1508     0.945218                      
    9      0.176225   0.150472   0.947985                      
    10     0.198024   0.146215   0.948345                      
    11     0.20324    0.189206   0.948145                      
    12     0.165159   0.151402   0.947745                      
    13     0.165997   0.146615   0.947905 
learn.unfreeze(), 1, wds=wd, cycle_len=14, use_clr=(32,10))

Fun thing to try: do the same thing, but reverse the document

When you combine the forward and backward models and average the results, we get a 95% accuracy!
The previous state of the art result was 94.1% accuracy (5.9% error). With bidir we get 95.4% accuracy (4.6% error).

Check out the paper


fastai.text performance on text sets

discriminative learning rates - renamed from differential learning rates

gradual unfreezing learning rates - unfreezing a layer at a time

Tweaked approach to cyclical learning rates CLR - based on idea from Leslie smith

Only do 1 cycle that goes up quickly and goes down slower afterwards. Currently implemented in fastai. First number is ratio of highest learning rate to the lowest ratio rate. 32. Second number is the ratio between teh first peak and the last peak. First epoch to be upward, and 9 down, = 10

link to the paper

BPTT - Normal RNN vs. MultiBatch RNN:

Key difference is that hte normal RNN encoder. We can BPTT chunk at a time, and predice the next word. But for the classifier, we need to do the entire doc, or the entire movie review. The entire review could be 2000 words, and we can’t fit all in memory.

Sends back only as many activations as we have decided to keep. If your max is 1000, but your doc is 2000. It will go through and only keep the most recent 1000 activations.

Results vs. customized algorithms

how do the different techniques affect the score?

  • What happens with different dataset sizes
  • What happens if different techniques are turned on and off?


That looks great! My only little issue is I don’t agree that the batch size isn’t really a batch size. I think it is - but there’s also a sequence length. I do agree it’s somehow a little different to how we have seen batches done in the past however…

@jeremy can you turn this post into a wiki so we can copy the markdown, please? :slight_smile:


1 Like

Shouldn’t ‘unk’ map to a different int than ‘pad’?

Nevermind. Its just inserting it to begining of list :slight_smile:

How can I use the fine-tuned language model to generate (sample) text? I could not find any appropriate methods for it. Can anyone guide me on it. (Note: I just started using fastai)