Now , I want to get the probabilty of a word in a context. So it should me something like: learn_generate.get_probability(context = "The car is", word = "red)

You need to use predict that will return to you the mos likely word, its index and (most importantly) the tensor of probabilities. Then pick the index of the word “red” (correspondence is in wiki_data.vocab.stoi for strings to ids).

Forgot about that. You should copy and paste the code from Learner.predict then, to be able to get what you want.
That makes me think it’s not ideal we replace predict in this case, this should maybe have another name.

@sgugger Sorry but I am not sure what you meant - copy the source code?
this part?

def predict(self, text:str, n_words:int=1, no_unk:bool=True, temperature:float=1., min_p:float=None, sep:str=' ',
decoder=decode_spec_tokens):
"Return the `n_words` that come after `text`."
ds = self.data.single_dl.dataset
self.model.reset()
xb,yb = self.data.one_item(text)
new_idx = []
for _ in range(n_words): #progress_bar(range(n_words), leave=False):
res = self.pred_batch(batch=(xb,yb))[0][-1]
#if len(new_idx) == 0: self.model[0].select_hidden([0])
if no_unk: res[self.data.vocab.stoi[UNK]] = 0.
if min_p is not None:
if (res >= min_p).float().sum() == 0:
warn(f"There is no item with probability >= {min_p}, try a lower value.")
else: res[res < min_p] = 0.
if temperature != 1.: res.pow_(1 / temperature)
idx = torch.multinomial(res, 1).item()
new_idx.append(idx)
xb = xb.new_tensor([idx])[None]
return text + sep + sep.join(decoder(self.data.vocab.textify(new_idx, sep=None)))

(from here: https://github.com/fastai/fastai/blob/master/fastai/text/learner.py#L116)
?
How would that help?

@sgugger Oh sorry I didn’t notice it should be from basic learner
But I still get an error:

batch = wiki_toy_data.one_item(“red”)
res = learn_generate.pred_batch(batch=batch)
raw_pred,x = grab_idx(res,0,batch_first=True),batch[0]
norm = getattr(learn_generate.data,‘norm’,False)
if norm:
x = learn_generate.data.denorm(x)
if norm.keywords.get(‘do_y’,False): raw_pred = self.data.denorm(raw_pred)
ds = learn_generate.data.single_ds
pred = ds.y.analyze_pred(raw_pred)
x = ds.x.reconstruct(grab_idx(x, 0))
y = ds.y.reconstruct(pred, x) if has_arg(ds.y.reconstruct, ‘x’) else ds.y.reconstruct(pred)

And get:

TypeError Traceback (most recent call last)

<ipython-input-23-0587895e696a> in <module>() 8 pred = ds.y.analyze_pred(raw_pred) 9 x = ds.x.reconstruct(grab_idx(x, 0)) —> 10 y = ds.y.reconstruct(pred, x) if has_arg(ds.y.reconstruct, ‘x’) else ds.y.reconstruct(pred)

3 frames

/usr/local/lib/python3.6/dist-packages/fastai/text/transform.py in <listcomp>(.0) 132 def textify(self, nums:Collection[int], sep=’ ') -> List[str]: 133 “Convert a list of nums to their tokens.” --> 134 return sep.join([self.itos[i] for i in nums]) if sep is not None else [self.itos[i] for i in nums] 135 136 def getstate(self):

TypeError: only integer tensors of a single element can be converted to an index

@sgugger But the how I can get the probability of the word “red” in a specific context?
Meaning I want to get P("red"|“The car is”) , but pred[0][data.vocab.stoi["red"] will just return p("red")?

I think I solved it. I’m pasting my function below the predict (my reference):

def predict(self, text:str, n_words:int=1, no_unk:bool=True, temperature:float=1., min_p:float=None, sep:str=' ',
decoder=decode_spec_tokens):
"Return `text` and the `n_words` that come after"
self.model.reset()
xb,yb = self.data.one_item(text)
new_idx = []
for _ in range(n_words): #progress_bar(range(n_words), leave=False):
res = self.pred_batch(batch=(xb,yb))[0][-1]
#if len(new_idx) == 0: self.model[0].select_hidden([0])
if no_unk: res[self.data.vocab.stoi[UNK]] = 0.
if min_p is not None:
if (res >= min_p).float().sum() == 0:
warn(f"There is no item with probability >= {min_p}, try a lower value.")
else: res[res < min_p] = 0.
if temperature != 1.: res.pow_(1 / temperature)
idx = torch.multinomial(res, 1).item()
new_idx.append(idx)
xb = xb.new_tensor([idx])[None]
return text + sep + sep.join(decoder(self.data.vocab.textify(new_idx, sep=None)))
def get_prob_of_word_in_context(self, context: str, word: str):
self.model.reset()
xb,yb = self.data.one_item(context)
res = self.pred_batch(batch=(xb, yb))[0][-1]
normalized_scores = F.softmax(res)
index_of_word = self.data.vocab.stoi[word]
prob_of_word_given_context = normalized_scores[index_of_word]
return prob_of_word_given_context

first we reset the model, feed it with the context (like the predict part), and pred_batch the same way.
res is a vector in the dimension of the number of words (total vocab.stoi). Meaning it contains a distribution of score for each possible word. If we normalize it by softmax, we’ll get a probability distribution. Next, we search for the word we want, getting its index with stoi, and that’s how we get the probability for it.