How to generate predictions from the new Language Model (using text.py)?


(Oren Dar) #1

Hey all!

In part 1, we could generate predictions from a LM (in my case, get it to turn prompts into full sentences) using something like

def test(s):
    m=learner.model
    #m[0].bs = 1
    t=TEXT.numericalize(s)
    res,*_ = m(t)
    i = 0
    prnt = ''
    while prnt != '.' and i < bptt:
        n=res[-1].topk(2)[1]
        n = n[1] if n.data[0]==0 else n[0]
        prnt = TEXT.vocab.itos[n.data[0]]
        print(prnt, end=' ')
        res,*_ = m(n[0].unsqueeze(0))
        i += 1

However, that was using torchtext - I looked around the source code of text.py and couldn’t figure out how to do this with the new class. I’m not interested in a classifier, just in the language model.

Thank you!


(Jeremy Howard) #2

It’s just a regular pytorch model so you can just pass it a pytorch variable and get back predictions. Have a try and let us know if you get stuck.


(Antti) #3

I am stuck here as well, and am probably doing something silly…

Trying to get predictions in the following way (looked into Lesson 6 for inspiration)

t contains encoding [5, 3, 14206, 1253, 52053, 50, 993, 3786]

but when I try
pred = m(V(T(np.array(t))))

I get this error message
~/dl/fastai/courses/dl2/fastai/lm_rnn.py in forward(self, input)
79 dropouth, list of tensors evaluated from each RNN layer using dropouth,
80 """
---> 81 sl,bs = input.size()
82 if bs!=self.bs:
83 self.bs=bs

ValueError: not enough values to unpack (expected 2, got 1)

Any thoughts would be appreciated :slight_smile:


(Jason McGhee) #4

I would take a look at the Test section of the translate notebook!


(Kevin Bird) #5

Try adding another bracket around the t. So m(V(T(np.array([t]))))


(Xu Fei) #6

Can’t believe this is the solution! It really took me a while and I should have searched earlier. Thanks Kevin!


#7

Hi all,

I’m trying to run the same code from Lesson 4 with the language model trained at the different word corpus and after running this piece:

for i in range(50):
    n=res[-1].topk(2)[1]
    n = n[1] if n.data[0]==0 else n[0]
    print(TEXT.vocab.itos[n.data[0]], end=' ')
    res,*_ = m(n[0].unsqueeze(0))
print('...')

I get the same error:

~/fastai/courses/dl1/fastai/lm_rnn.py in forward(self, input)
     91             dropouth, list of tensors evaluated from each RNN layer using dropouth,
     92         """
---> 93         sl,bs = input.size()
     94         if bs!=self.bs:
     95             self.bs=bs

ValueError: not enough values to unpack (expected 2, got 1)

It’s probably something very simple and I’m not getting it… Please help.


#8

Are you getting this error with the original Lesson 4 notebook or after introducing the modifications discussed above? Try to check out a fresh copy of the fastai github repo and also update your conda dependencies. With this PR merged this morning, predictions should work out of the box in that lesson again.


(Karl) #9

This is how I’m generating text with the Part 2 type language model. This function takes a raw text string, tokenizes it, numericalizes it and creates a variable from the numericalized text. I ran into the ValueError: not enough values to unpack (expected 2, got 1) issue and fixed it by adding a unit axis to the variable with V(np.array(s_nums))[None].

For predicting text I found that torch.topk resulted in the model repeating itself a lot. I get better looking predictions using torch.multinomial instead.

def sample_model(m, s, l=50):
    s_toks = Tokenizer().proc_text(s)
    s_nums = [stoi[i] for i in s_toks]
    s_var = V(np.array(s_nums))[None]

    m[0].bs=1
    m.eval()
    m.reset()

    res, *_ = m(s_var)
    print('...', end='')

    for i in range(l):
        r = torch.multinomial(res[-1].exp(), 2)
        #r = torch.topk(res[-1].exp(), 2)[1]
        if r.data[0] == 0:
            r = r[1]
        else:
            r = r[0]
        
        word = itos[to_np(r)[0]]
        res, *_ = m(r[0].unsqueeze(0))
        print(word, end=' ')
    m[0].bs=bs