This thread Language Model Zoo 🦍 recommends running the following line if problems persist:
torch.cuda.empty_cache()