So I ran in to the same issue with the songs. I’ve seen this exact problem jeremy outlined a solution here Configuring stateful lstm cell in the the language model
This is a common problem with text generation networks which is compounded when songs have a lot of repeating chorus. One non machine learning way of solving this issue is picking the top 5 most probable word instead of the top one which keeps the song from repeating.
The other solution i’ve heard is using a pretrained embedding like https://github.com/facebookresearch/fastText