Hi all - for anyone who wants to use AITextGen, available here: GitHub - minimaxir/aitextgen: A robust Python tool for text-based AI training and generation using GPT-2.
Just wanted to give a couple of tips. I have learned the hard way that the Collab notebook (the “fine-tune” one) does not work out of the box. You’ll need to implement one of the solutions in this thread: ImportError: cannot import name '_TPU_AVAILABLE' from 'pytorch_lightning.utilities' · Issue #200 · minimaxir/aitextgen · GitHub
I tend to just add “pip install -q pytorch-lightning==1.7.7” to the pip install cell, and have not had any problems since. Of course, the path to victory may change in the future, as the product changes.
Also, I have found Jeremy’s graduated learning rate technique very helpful here. The predecessor (?) textgenrnn used to have a dropout rate option, but getting that in aitextgen requires a recompilation (big no-no for me…to this day I don’t think I have ever successfully recompiled anything). So reducing the learning rate a little at a time is the way to go. Happy text genning!