Duplicate timesteps when using Karras sigmas can break Img2Img with low strength

[nhnt11]

Describe the bug

When using Karras sigmas, at high num_inference_steps (e.g. 100) and low strength (e.g. 0.1), image to image breaks with an error like a Tensor with 2 elements cannot be converted to Scalar.

The root cause seems to be duplicate timesteps. The final few sigmas are so small that they end up mapping to the same discrete timestep values, resulting in timesteps like [... 5 4 4 3 2 2 1 1 1 0 0].

This breaks logic in a few spots that assume that timesteps are unique. For example when inferring the step index in add_noise

Reproduction

import torch
from diffusers import StableDiffusionXLImg2ImgPipeline
from typing import cast
from diffusers import DPMSolverMultistepScheduler, StableDiffusionPipeline, 

sdxl_img2img_model = cast(StableDiffusionXLImg2ImgPipeline, StableDiffusionXLImg2ImgPipeline.from_pretrained(
 'stabilityai/stable-diffusion-xl-base-1.0',
 torch_dtype=torch.float16,
 use_safetensors=True,
 variant="fp16",
 revision="76d28af79639c28a79fa5c6c6468febd3490a37e",
)).to('cuda')

common_config = {'beta_start': 0.00085, 'beta_end': 0.012, 'beta_schedule': 'scaled_linear'}
scheduler = DPMSolverMultistepScheduler(**common_config, use_karras_sigmas=True)

init_img_url = 'https://storage.googleapis.com/pai-images/ca5dae9cef0541798c2d03f244365fde.jpeg'
from urllib.request import urlopen
from PIL import Image

init_img = Image.open(urlopen(init_img_url))
init_img = init_img.resize((1024, 1024))

generator = torch.Generator(device='cuda')
generator.manual_seed(12345)
img2img_params = {
 'prompt': ['a frog'],
 'negative_prompt': [''],
 "num_inference_steps": 100,
 "guidance_scale": 7,
 "image": init_img,
 "strength": 0.1
}

sdxl_img2img_model.scheduler = scheduler
sdxl_img2img_res = sdxl_img2img_model(**img2img_params, generator=generator, output_type='pil')

display(sdxl_img2img_res.images[0])

Logs

No response

System Info

• diffusers version: 0.22.1
• Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.31
• Python version: 3.11.5
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Huggingface_hub version: 0.17.1
• Transformers version: 4.34.0
• Accelerate version: 0.22.0
• xFormers version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@yiyixuxu @patrickvonplaten

[nhnt11]

This proof of concept commit unbreaks image to image for the testcase in the description: playgroundai-legacy@361c86f

With this change, we always use the first matching timestep when there are dupes. This means that different strength values that map to timestep indices that contain the same timestep value will effectively be coalesced and yield the same results.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

@yiyixuxu is this one solved?

[patrickvonplaten]

Gentle ping here again @yiyixuxu

[yiyixuxu]

@patrickvonplaten
I have proposed a solution here #5746

would this be ok?