I’m trying to implement the DiffEdit paper, and to do so, I need to add noise to the input image.
The following is how I’m doing it.
img = Image.open('/content/planet.png').resize((512, 512))
import torchvision.transforms as T
with torch.no_grad():
lat = vae.encode(T.ToTensor()(img).unsqueeze(0).half().to('cuda')*2-1)
lat = 0.18215 * lat.latent_dist.sample()
sched = LMSDiscreteScheduler(
beta_start=0.00085,
beta_end=0.012,
beta_schedule='scaled_linear',
num_train_timesteps=1000
)
sched.set_timesteps(15)
noise = torch.randn_like(lat)
ts = tensor([sched.timesteps[10]])
lat = sched.add_noise(lat, noise, timesteps=ts)
However, the last cell outputs the following error and I’m baffled as to why it’s occuring.
RuntimeError: a Tensor with 0 elements cannot be converted to Scalar
Full Traceback
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
[<ipython-input-24-8d2efdc445c5>](https://6umz6pprmf5-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230509-060147-RC00_530563781#) in <cell line: 3>() 1 noise = torch.randn_like(lat) 2 ts = 10 ----> 3 lat = sched.add_noise(lat, noise, timesteps=tensor([sched.timesteps[ts]]))
---
1 frames
---
[/usr/local/lib/python3.10/dist-packages/diffusers/schedulers/scheduling_lms_discrete.py](https://6umz6pprmf5-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230509-060147-RC00_530563781#) in add_noise(self, original_samples, noise, timesteps) 302 timesteps = timesteps.to(original_samples.device) 303 --> 304 step_indices = [(schedule_timesteps == t).nonzero().item() for t in timesteps] 305 306 sigma = sigmas[step_indices].flatten()
[/usr/local/lib/python3.10/dist-packages/diffusers/schedulers/scheduling_lms_discrete.py](https://6umz6pprmf5-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230509-060147-RC00_530563781#) in <listcomp>(.0) 302 timesteps = timesteps.to(original_samples.device) 303 --> 304 step_indices = [(schedule_timesteps == t).nonzero().item() for t in timesteps] 305 306 sigma = sigmas[step_indices].flatten()
RuntimeError: a Tensor with 0 elements cannot be converted to Scalar
I’ve thoroughly checked the tensors that are being used and none of them have 0 elements. I’ve also tried directly editing the add_noise method, but any changes I make to it don’t seem to be registering (e.g., doing a print statement causes the same error to be thrown, and the traceback says it’s occuring at that print statement); I’m doing this on Google Colab.
You can view the code on Colab here: Google Colab
The relevant code is under the “Add Noise to Image” header (there are two headers of the same name in the notebook; it’s the first one that is the relevant one).
I’d really appreciate help; I’m baffled.
I believe the example from johono’s notebook is because the images are progressively sort of mixed-in as they progress through the diffusion process. And I believe that’s what happens when you do the non-inpaiting for DiffEdit. Or rather, what I think is happening - you take only the parts of the image (in latent space) that are not masked and continue on the diffusion process for that part with the edited prompt. The masked part can either be diffused using the original prompt and/or replaced by the final latent result from the original prompt — depending on how you do things. (I’m just phrasing things here as I see how this might work — the paper probably has a specific way to do this but I don’t recall …)
?
