Text Generation and Beam search


(Abu Fadl) #1

Are there examples (sample code) of text generation implemented in fastai based on ULMFiT (other than simply predict next tokens from language model learner)? I found a few references to beam search:
@bfarzin 's post (Developer chat) and Improving Text Generation and Share your work here ✅ but could not find code examples. Any pointers?


#2

beam_search will be in the next release of fastai (and already is present in master).


(Abu Fadl) #3

This is great news. How do I pass a non-English pretrained files (itos.pkl and lm.pth)? pretrained_fnames is no longer accepted. If AWD_LSTM is specified as arch with pretrained=True, URLs.WT103_1 gets downloaded. I tried copying my files to data/models/ but doesn’t seem to work.


#4

You’re right, that’s missing now, I forgot to put them back there. Will add that tonight or tomorrow.


(Abu Fadl) #5

Is it feasible to also filter out unknowns in beam search (similar to no_unk=True in learner.predict)?


#6

It’s not implemented yet, but a PR could add that.


(Abu Fadl) #7

Unfortunately, I am still on the code consumer side. I hope to cross the border someday. Right now, my contribution is to appreciate hard work by all those at https://github.com/fastai/fastai/graphs/contributors especially the top ones :slight_smile:


(Abdu Issa) #8

This is very cool stuff. I tried to implement beam search by loading a language_model_learner, training it, then typing this line of code:

data_lm = TextLMDataBunch.load(path, ‘data_lm’, bs=32)
learn = language_model_learner(data_lm, AWD_LSTM, pretrained=True, drop_mult=0.3)
learn.load(‘final_fit_render’)
learn.beam_search(‘hello’, 1)

But alas, i am getting this error:
RuntimeError: CUDA error: device-side assert triggered

learn.predict(‘hello’, 1) still works though.

Is there something different i should be doing?


(Abu Fadl) #9

Try changing the beam size (1000 by default) and try a few words instead of 1. I tried generating movie reviews (kernel: https://www.kaggle.com/abedkhooli/textgen-fastai143/) based on the IMDB dataset and got better results from learner prediction. Beam search may either require parameter tuning or has a cyclic issue in implementation (I am not a movie goer but results have so much repetition to suggest an obvious problem).


(Bobak Farzin) #10

one line change + param. I have a PR with this included:
if no_unk: out[:,self.data.vocab.stoi[UNK]] = -float('Inf')


(Abu Fadl) #11

That’s great, @bfarzin - I guess it’s not merged yet. Have you tested the quality of text generated with beam_search? I am still getting poor results on the IMDB reviews (even compared to learner.predict).
Here are two examples (100 words based on imdb train and test, using wt103_1, start with [‘when’, ‘i really’], temperature=0.75, top_k=10,beam_sz=100). Trying to get rid of the repetition but not there yet:

  1. “when xxbos when i went to see this movie , i thought it was going to be good . i was wrong . The acting was bad , the plot was bad , and the acting was terrible . The only thing that kept me watching was the fact that it was supposed to take place in outer space . In fact , i do n’t know if it was supposed to take place in outer space . In fact , i do n’t know if it was supposed to take place in outer space . In fact”
  2. "i really xxbos i really wanted to like this movie . However , i found it to be one of the worst movies i have ever seen in my entire life . " The English Patient " is one of the worst movies i have ever seen in my entire life . " The English Patient " is one of the worst movies i have ever seen .

SPOILER It happens , you get your first impression when we learn how this film will not go over as long in our life as i can
"


(Cahya) #12

Is there any chance with ULMFiT to get an impressive result on text generation like the one from openai (https://blog.openai.com/better-language-models/) recently? :slight_smile: