As you are trying to generate new images the approach you are trying isn’t quite right.
You may follow this nice introduction to generative models.
fastai supports training of basic GAN(Generative Adversarial Networks), but I haven’t tried it. Here is the documentation.