Fast.ai has a great interface for implementing adaptive switching of GAN model training. I have been experimenting with different thresholds for minimum loss before switching.
Some things I have noticed:
- If the GAN Discriminator ever gets too accurate, the Generator model will collapse to an overfitted state
- At higher losses, the Generator model improves poorly (based on my monkey-brain visual analysis)
I sometimes worry that visual analysis is not a good way to evaluate a model until the very end. I am curious if there is a healthy equilibrium for GAN losses, or if I need to ‘trust the process’ more when training and stop worry about intermediate states that look like state collapse.
I’d love to hear some experience from others playing around with GANs.