How to get the language model predictions and probabilities

Part 2 & Alumni (2018)
1

After loading a pretrained language model or fine-tuning a language model on the target data, I don’t know how to get the language model predictions and associated probabilities on a validation set

LM_PATH=Path('data/imdb_lm/')
tok_trn = np.load(LM_PATH/'tmp'/'tok_trn.npy')
tok_val = np.load(LM_PATH/'tmp'/'tok_val.npy')
trn_lm = np.array([[stoi[o] for o in p] for p in tok_trn])
val_lm = np.array([[stoi[o] for o in p] for p in tok_val])
# Take just 10 examples for testing purposes
trn_lm = trn_lm[:5]
val_lm = val_lm[:5]

trn_dl = LanguageModelLoader(np.concatenate(trn_lm), bs, bptt)
val_dl = LanguageModelLoader(np.concatenate(val_lm), bs, bptt)
md = LanguageModelData(PATH, 1, vs, trn_dl, val_dl, bs=bs, bptt=bptt)

drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*0.7
learn = md.get_model(opt_fn, em_sz, nh, nl, 
    dropouti=drops[0], dropout=drops[1], wdrop=drops[2], dropoute=drops[3], dropouth=drops[4])

learn.metrics = [accuracy]
# Load the language model pre-trained  on WikiText103
learn.load('bwd_wt103.h5')

learn.data.test_dl = val_dl
preds = learn.predict(is_test=True)

Then, I don’t know how to show the language model predictions and probabilities for an example (a chunk of text, bptt) in the val_dl. Anyone knows how to get them? Thanks in advance.

1 Like
2

What was the output you got for running the above code? I am also trying to get the predictions but unable to do so ?

3

my input contains 2 batches (bs = 2)

for item in val_dl:
    print('#####')
    x, y = item
    print(x.shape)
    print(y.shape)

#####
torch.Size([95, 2])
torch.Size([190])
#####
torch.Size([66, 2])
torch.Size([132])

but I don’t know why preds has shape (322, 60000)

learn.data.test_dl = val_dl
preds = learn.predict(is_test=True)
print(preds.shape)

(322, 60000)

I know 60000 is the vocabulary size but I dont know why it’s 322, it’s close to 320 = 190 + 130 the sizes of y in the input batches. Anyone knows where does the number 322 come from?