How to generate predictions from the new Language Model (using text.py)?

Hey all!

In part 1, we could generate predictions from a LM (in my case, get it to turn prompts into full sentences) using something like

def test(s):
    m=learner.model
    #m[0].bs = 1
    t=TEXT.numericalize(s)
    res,*_ = m(t)
    i = 0
    prnt = ''
    while prnt != '.' and i < bptt:
        n=res[-1].topk(2)[1]
        n = n[1] if n.data[0]==0 else n[0]
        prnt = TEXT.vocab.itos[n.data[0]]
        print(prnt, end=' ')
        res,*_ = m(n[0].unsqueeze(0))
        i += 1

However, that was using torchtext - I looked around the source code of text.py and couldn’t figure out how to do this with the new class. I’m not interested in a classifier, just in the language model.

Thank you!

2 Likes

It’s just a regular pytorch model so you can just pass it a pytorch variable and get back predictions. Have a try and let us know if you get stuck.

1 Like

I am stuck here as well, and am probably doing something silly…

Trying to get predictions in the following way (looked into Lesson 6 for inspiration)

t contains encoding [5, 3, 14206, 1253, 52053, 50, 993, 3786]

but when I try
pred = m(V(T(np.array(t))))

I get this error message
~/dl/fastai/courses/dl2/fastai/lm_rnn.py in forward(self, input)
79 dropouth, list of tensors evaluated from each RNN layer using dropouth,
80 """
---> 81 sl,bs = input.size()
82 if bs!=self.bs:
83 self.bs=bs

ValueError: not enough values to unpack (expected 2, got 1)

Any thoughts would be appreciated :slight_smile:

1 Like

I would take a look at the Test section of the translate notebook!

Try adding another bracket around the t. So m(V(T(np.array([t]))))

2 Likes

Can’t believe this is the solution! It really took me a while and I should have searched earlier. Thanks Kevin!

Hi all,

I’m trying to run the same code from Lesson 4 with the language model trained at the different word corpus and after running this piece:

for i in range(50):
    n=res[-1].topk(2)[1]
    n = n[1] if n.data[0]==0 else n[0]
    print(TEXT.vocab.itos[n.data[0]], end=' ')
    res,*_ = m(n[0].unsqueeze(0))
print('...')

I get the same error:

~/fastai/courses/dl1/fastai/lm_rnn.py in forward(self, input)
     91             dropouth, list of tensors evaluated from each RNN layer using dropouth,
     92         """
---> 93         sl,bs = input.size()
     94         if bs!=self.bs:
     95             self.bs=bs

ValueError: not enough values to unpack (expected 2, got 1)

It’s probably something very simple and I’m not getting it… Please help.

Are you getting this error with the original Lesson 4 notebook or after introducing the modifications discussed above? Try to check out a fresh copy of the fastai github repo and also update your conda dependencies. With this PR merged this morning, predictions should work out of the box in that lesson again.

This is how I’m generating text with the Part 2 type language model. This function takes a raw text string, tokenizes it, numericalizes it and creates a variable from the numericalized text. I ran into the ValueError: not enough values to unpack (expected 2, got 1) issue and fixed it by adding a unit axis to the variable with V(np.array(s_nums))[None].

For predicting text I found that torch.topk resulted in the model repeating itself a lot. I get better looking predictions using torch.multinomial instead.

def sample_model(m, s, l=50):
    s_toks = Tokenizer().proc_text(s)
    s_nums = [stoi[i] for i in s_toks]
    s_var = V(np.array(s_nums))[None]

    m[0].bs=1
    m.eval()
    m.reset()

    res, *_ = m(s_var)
    print('...', end='')

    for i in range(l):
        r = torch.multinomial(res[-1].exp(), 2)
        #r = torch.topk(res[-1].exp(), 2)[1]
        if r.data[0] == 0:
            r = r[1]
        else:
            r = r[0]
        
        word = itos[to_np(r)[0]]
        res, *_ = m(r[0].unsqueeze(0))
        print(word, end=' ')
    m[0].bs=bs
2 Likes

Yes, I’m getting this error with original notebook code. The only part changed is the dataset–I use another set of text files. I’ve pulled the repo and noticed there were some changes made to the lesson 4 notebook but, unfortunately, the code is still throwing me that error. Anything else I could do?

Tried using this and not working:

NameError: name 'stoi' is not defined

I’m using the Lesson 4 code and looks like I need to load Lesson 10 dependencies. Do you think I should try doing that or it’s too much of a change in code between the lessons?

stoi is a dictionary mapping strings to numerical values. For Lesson 4 where you use torchtext I think you would use functions in TEXT.

def proc_str(s): return TEXT.preprocess(TEXT.tokenize(s))
def num_str(s): return TEXT.numericalize([proc_str(s)])

def sample_model(m, s, l=50):
    t = num_str(s)
    m[0].bs=1
    m.eval()
    m.reset()
    res,*_ = m(t)
    print('...', end='')

    for i in range(l):
        r = torch.multinomial(res[-1].exp(), 2)
        if r.data[0] == 0:
            r = r[1]
        else:
            r = r[0]
        word = TEXT.vocab.itos[to_np(r)[0]]
        res, *_ = m(r[0].unsqueeze(0))
        print(word, end=' ')
    m[0].bs=bs

Thanks, but now I get this error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-14-e936e3eee8eb> in <module>()
     23     m[0].bs=bs
     24 
---> 25 sample_model(m,s)

<ipython-input-14-e936e3eee8eb> in sample_model(m, s, l)
     18         else:
     19             r = r[0]
---> 20         word = TEXT.vocab.itos[to_np(r)[0]]
     21         res, *_ = m(r[0].unsqueeze(0))
     22         print(word, end=' ')

IndexError: too many indices for array

@KarlH Can you please take a look?

@shoof Do you mind posting the whole piece of code you used to make predictions please?

This is due to the PyTorch’s changes of data dimension. Change n[0].unsqueeze(0) to n[0].unsqueeze(0).unsqueeze(0) can solve the problem. In addition, since in this example tensor n only contains one element after n = n[1] if n.data[0]==0 else n[0], so you want to avoid using n[0]. So in summary this is what you need:
FROM

    print(TEXT.vocab.itos[n.data[0]], end=' ')
    res,*_ = m(n[0].unsqueeze(0))

TO

    print(TEXT.vocab.itos[n], end=' ')
    res,*_ = m(n.unsqueeze(0).unsqueeze(0))

Hi all,

I built a language model as follow:

data_lm = TextLMDataBunch.from_csv(PATH,valid='val')
learn = RNNLearner.language_model(data_lm, drop_mult=0.5,nh=500d)
learn.fit(4, 3e-3,wd=1e-6)
learn.save('/storage/model')

And after training:

state_dict = torch.load('/storage/model.pth', map_location=lambda storage, loc: storage)
learn.model.load_state_dict(state_dict)
torch.autograd.Variable(np.array(learn.data.train_ds.vocab.numericalize('my_word')).T)
m = learn.model
m[0].bs=1
m.eval()
m.reset()
#line with error
res,*_ = m(v)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-133-b784f1b0a350> in <module>
      3 m.eval()
      4 m.reset()
----> 5 res,*_ = m(v)

/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     90     def forward(self, input):
     91         for module in self._modules.values():
---> 92             input = module(input)
     93         return input
     94 

/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

/notebooks/fastai/text/models.py in forward(self, input)
    104 
    105     def forward(self, input:LongTensor) -> Tuple[Tensor,Tensor]:
--> 106         sl,bs = input.size()
    107         if bs!=self.bs:
    108             self.bs=bs

ValueError: not enough values to unpack (expected 2, got 1)

Any tip for fix?