Diffusion models and fastai Dynamic U-Net

Hi everyone. I’ve been studying diffusion models for the past couple of months and using a U-Net for the diffusion/generation process within that framework seems to be the done thing in the literature. However, the U-Nets used in SOTA diffusion models are huge, and training them from scratch is incredibly costly. This got me thinking if there’s a smart, fastai-y way to train a diffusion model based on a U-Net on a budget, similar to how using a pre-trained backbone and “NoGAN” training was utilised to train GANs in the course. Would transfer learning for the U-Net backbone make sense in a diffusion framework?