Stable diffusion: resources and discussion

brismith · October 13, 2022, 12:58pm

I haven’t been able to get the notebooks working with the M1 - just getting noise (also issues with float64). Using cpu worked ok - but the speed dropped to about 2:40 per image. I switched over to my old box, a Linux machine with an 1080TI and the default dream.py images are produced in around 11 seconds - so I might see if I can get that working with the notebooks.

wyquek · October 13, 2022, 4:42pm

I could run all of the nb (except the last cell) with a linux 1080 8G and avoid a cuda error if I reduce the width by half

height = 512                    
width = 256

Didn’t get the same image as in the original nb, but pretty decent “half” images

brismith · October 13, 2022, 5:54pm

Thanks - and I saw the same memory issues. The deep dive notebook also runs well on the 1080TI with some kernel restarts ;).

jimmiemunyi · October 14, 2022, 5:12am

Sharing this video suggested by Tanishq on Twitter. Nice overview of SD although he does leave out some details, like adding noise on the encoded output by the VAE and not on the original image.
Overall nice video.

babayaga · October 14, 2022, 8:47am

wyquek · October 14, 2022, 1:43pm

it’s possible to run all cells at one go on Nvidia 1080 8G with some minor modifications to the nb, mostly consisting of splitting up some large cells into smaller ones

github.com

Qbiwan/diffusion-nbs/blob/master/Stable Diffusion Deep Dive_Nvidia_1080_8G.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5dcb2f23-7bcb-42d6-ae00-c336222dcd93",
   "metadata": {},
   "source": [
    "# Stable Diffusion Deep Dive\n",
    "\n",
    "Stable Diffusion is a powerful text-to-image model. There are various websites and tools to make using it as easy as possible. It is also [integrated into the Huggingface diffusers library](https://huggingface.co/blog/stable_diffusion) where generating images can be as simple as:\n",
    "```python\n",
    "from diffusers import StableDiffusionPipeline\n",
    "pipe = StableDiffusionPipeline.from_pretrained(\"CompVis/stable-diffusion-v1-4\", revision=\"fp16\", torch_dtype=torch.float16, use_auth_token=True).to(\"cuda\")\n",
    "image = pipe(\"An astronaught scuba diving\").images[0]\n",
    "\n",
    "```\n",
    "\n",
    "In this notebook we're going to dig into the code behind these easy-to-use interfaces, to see what is going on under the hood. We'll begin by re-creating the functionality above as a scary chunk of code, and then one by one we'll inspect the different components and figure out what they do. By the end of this notebook that same sampling loop should feel like something you can tweak and modify as you like. "
   ]
  },

This file has been truncated. show original

Also, the smaller images look better if you use landscape height=256 and width=512

amanmadaan · October 15, 2022, 6:05am

I was curious about training diffusion models for generating text, so have put together a minimal implementation here. This implementation can be used to train unconditional generative model of text, and also includes a small implementation of classifier guidance for conditional generation. Will be happy to answer questions about the implementation or diffusion models in general! Also includes a denoising/generative sampling loop visualization:

minimal-text-diffusion

ilovescience · October 15, 2022, 10:03am

Just read this paper yesterday. The idea of treating the “noise” that you use in diffusion models as a latent vector is a very surprising but powerful concept in my opinion. Lots to explore there…

Tamori · October 19, 2022, 6:33am

Another interesting application of diffusion to create different 3D views of an object from a single image + the object’s pose:
https://3d-diffusion.github.io/

neuraloverflow · October 19, 2022, 5:11pm

Would be nice to generate a mesh from a 3d point cloud input with SD

KarlH · October 19, 2022, 11:30pm

What’s the current SOTA resource for stable diffusion with respect to inference speed? Is there a resource somewhere that tried to keep up with “SOTA” versions of SD for things like inference speed, memory usage, etc?

ilovescience · October 21, 2022, 12:28am

This Computerphile video is good:

daspartho · October 21, 2022, 3:09am

Really good math explanation of diffusion models:

undertaker2077 · October 21, 2022, 4:30am

I have a plan to work on text generation. Glad to find your work as a starting point. I see a list of references in your repo. Which ones do you recommend to read first for the uninitiated?

amanmadaan · October 21, 2022, 5:48am

[2205.14217] Diffusion-LM Improves Controllable Text Generation might be a good starting point, but there are a ton of caveats that come with any approach that manipulates latent space for text generation (GANs/VAEs/Normalizing flows).

I plan to add a blog/tutorial in the next week or so as well.

Good luck and I’m looking forward to knowing more about what you find/come up with!

bencoman · October 21, 2022, 3:37pm

Ha! The video mentions the “enhance” meme. Only yesterday on the drive to work I was thinking… the awful thing about this ML denoising and generative upsizing, is that it belies all my previous vocal criticism of TV memes where technology can super-“enhance” poor security footage.

joshfp · October 29, 2022, 3:15pm

Random thought: Learning about Stable Diffusion, I’m surprised how accurately it resembles Michelangelo’s famous quote “The sculpture is already complete within the marble block, before I start my work. It is already there, I just have to chisel away the superfluous material.”. It can be paraphrased as “The image is already complete before I start my work. It is already there, I just have to remove the superfluous noise.”

sswam · October 29, 2022, 5:14pm

There are two new VAEs out from stability. These don’t affect the form of the image, only the final decode step where it scales up from the 64x64 latents to a 512x512 image. The new decoders give much better results, when it comes to details such as small faces, eyes, and letters.

barnacl · October 29, 2022, 5:38pm

replace this line:
vae = AutoencoderKL.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="vae")
with:
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-ema")

sswam · October 29, 2022, 5:40pm

Or stabilityai/sd-vae-ft-mse, which has been trained for more steps and seems to give better results.