I had the same issue for StableDiffusionImg2ImgPipeline. So I installed the diffusers library from the latest commit and the issue was solved for me.
pip install git+https://github.com/huggingface/diffusers.git@e895952816dc4a9a04d1d82fb5f8bdee59341c06
(If you are working on colab, restart the runtime after you run the above command)
Hope this solves your issue too.
Update: Diffusers version 0.4.2 has the fix for this issue. Install it by using the command pip install diffusers==0.4.2