Pipei2i error: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor)

simonw · October 11, 2022, 8:42pm

Got this error running this notebook: diffusion-nbs/stable_diffusion.ipynb at master · fastai/diffusion-nbs · GitHub

At this step:

torch.manual_seed(1000)
prompt = "Wolf howling at the moon, photorealistic 4K"
images = pipei2i(prompt=prompt, num_images_per_prompt=3, init_image=init_image, strength=0.8, num_inference_steps=50).images
image_grid(images, rows=1, cols=3)

Error was:

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

Here’s a screenshot of the full trackback:

Anyone know how to fix this one?

I’m running on a https://lambdalabs.com/ A100 instance.

simonw · October 11, 2022, 8:43pm

I changed the first line in that notebook to the following to get a missing dependency:

%pip install diffusers transformers fastcore fastdownload

simonw · October 11, 2022, 8:49pm

Found the answer here: Lesson 9 official topic - #27

@bernhard_jung says:

You need a diffusers version higher than 0.4.1, because this patch is needed for fp16 to work.

simonw · October 11, 2022, 8:52pm

So running this in the first cell should work:

%pip install -U diffusers transformers fastcore fastdownload

It doesn’t though - that still gave me diffusers 0.4.1.

This DID work:

%pip install -U https://github.com/huggingface/diffusers/archive/refs/heads/main.zip

That installs the latest main branch of the diffusers GitHub repo.

simonw · October 11, 2022, 9:00pm

Better fix inspired by Lesson 9 official topic - #66

%pip install https://github.com/huggingface/diffusers/archive/e895952816dc4a9a04d1d82fb5f8bdee59341c06.zip

That grabs a specific commit which definitely includes the fix.

cjmills · October 11, 2022, 9:02pm

Same, had to install from GitHub as both pip and conda are still on 0.4.1

nmoran · October 12, 2022, 4:15pm

Seems default version from pip is now 0.4.2 so should work with fresh install or upgrade.

simonw · October 12, 2022, 5:03pm

Yeah here are their release notes, from 18 hours ago: Release v0.4.2: Patch release · huggingface/diffusers · GitHub

gsg · October 12, 2022, 7:53pm

@simonw Did it fix the problem?
I have the following versions:

diffusers                         0.4.2
fastai                            2.7.5
huggingface-hub                   0.10.1
transformers                      4.23.1

and still get the

Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

ste · October 13, 2022, 1:09am

Quick fix: change torch_dtype to float32 and it works.

from diffusers import StableDiffusionImg2ImgPipeline
from fastdownload import FastDownload

pipei2i = StableDiffusionImg2ImgPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16",
    torch_dtype=torch.float32, # Changed row
).to("cuda")

gsg · October 13, 2022, 2:23am

@ste,
Yes it does! Thanks!

strickvl · October 13, 2022, 5:14am

From what I understand of their docs here, this is mainly for places where you have memory constraints.

“Note: If you are limited by GPU memory and have less than 10GB of GPU RAM available, please make sure to load the StableDiffusionPipeline in float16 precision instead of the default float32 precision as done above.”

I imagine that most of us should probably be ok to use the float32, esp if using cloud instances, but I wonder if it’s also slower then to generate the images etc. Probably the way it is right now allows for faster iteration.

In any case, I used the requirements.txt (recently merged in) for my pip install, then restarted the kernel (while using JarvisLabs) and it worked for me with the notebook as is.

ste · October 13, 2022, 7:48am

That’s right @strickvl, both less memory usage and faster computation with fp16!

PalaashAgrawal · October 13, 2022, 10:08am

@jeremy Seems like there are a lot of issues related to the versions of packages. Do you want to keep this in the back of your mind and maybe release a stable configuration later on?

strickvl · October 13, 2022, 10:48am

A requirements.txt file was merged onto the repo (yesterday, I think), which should be what you’re looking for in terms of ‘stable’ versions of packages. Hopefully that can be updated as needed.

osanseviero · October 13, 2022, 8:26pm

Hey all, we just did a new release in diffusers. Please let us know if you still face this issue.

gsg · October 13, 2022, 9:49pm

Yes. I installed diffusers 0.5.1, and (after restarting the Kernel), the error went away.

Thanks!

Also, for the
‘’’
prompt = “a photograph of an astronaut riding a horse”
“”"

the horse also went away (at first)…

barnacl · October 13, 2022, 10:32pm

There seems to be similar issue with the inpainting pipeline.
Have to use autocast currently to get around it
If anyone wants to play around with it, here is the code

from diffusers import StableDiffusionInpaintPipeline
from PIL import ImageDraw

pipe_inpaint = StableDiffusionInpaintPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16",
    torch_dtype=torch.float16,
).to("cuda")
p = FastDownload().download("https://i.redd.it/hdxi9adiquw21.jpg")#credit-https://www.reddit.com/r/funny/comments/blw0hq/a_true_photobomb/
init_image = Image.open(p).convert("RGB").resize((485, 323))
#mask is black and white(region to be inpainted)
msk = init_image.resize((485, 323))
drawMsk = ImageDraw.Draw(msk)
drawMsk.rectangle([(0, 0), (485, 323)],outline=(0, 0, 0), fill=(0, 0, 0))#black bkg
drawMsk.rectangle([(350, 170), (480, 250)],outline=(255, 255, 255), fill=(255, 255, 255))#white
image_grid([init_image,msk],1,2)
torch.manual_seed(10)
from torch import autocast
prompt = "rocks in the water"
with autocast("cuda"):
    images = pipe_inpaint(prompt=prompt, init_image=init_image, mask_image=msk,strength=0.7,guidance_scale=8, num_inference_steps=70).images
images[0]

the-groot · October 14, 2022, 3:22pm

Hi, maintainer of diffusers here

Thanks for reporting it! This issue with image2image and inpainting pipeline has been fixed in v0.4.2 please upgrade your version of diffusers and then this should run in fp16 without any issues.

jeremy · October 14, 2022, 3:22pm

This course will cover a lot of cutting edge stuff - the idea of having a stable configuration isn’t really compatible with what we’re doing here… Expect sharp edges!

Also, a goal of this course is for students to get comfortable with cutting edge research - which includes dealing with all the little software issues which come up in research code.