Skip to content

Fix guess_mode in StableDiffusionXLControlNetPipeline - #4725

Closed
YashalKanungo wants to merge 1 commit into
huggingface:mainfrom
YashalKanungo:main
Closed

Fix guess_mode in StableDiffusionXLControlNetPipeline#4725
YashalKanungo wants to merge 1 commit into
huggingface:mainfrom
YashalKanungo:main

Conversation

@YashalKanungo

Copy link
Copy Markdown

Fix guess_mode in StableDiffusionXLControlNetPipeline

What does this PR do?

It fixes a bug due to which guess mode was not usable in StableDiffusionXLControlNetPipeline.

Reproducing the original bug

Following the example at https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0
if we set guess_mode=True

images=pipe(
prompt,
negative_prompt=negative_prompt,
image=image,
controlnet_conditioning_scale=controlnet_conditioning_scale,
guess_mode=True
).images

it throws the error,

RuntimeErrorTraceback (mostrecentcalllast)
CellIn[57], line2421image=np.concatenate([image, image, image], axis=2)
22image=Image.fromarray(image)
--->24images=pipe(
25prompt, negative_prompt=negative_prompt, image=image, controlnet_conditioning_scale=controlnet_conditioning_scale, guess_mode=True26 ).imagesRuntimeError: Thesizeoftensora (8192) mustmatchthesizeoftensorb (4096) atnon-singletondimension1

Fixes

During ControlNet inference for only the conditional batch, also only use the corresponding add_text_embeds and add_time_ids.

Tests

  1. Able to generate some images with guess_mode=True
  2. Ran test_controlnet_sdxl
pytest tests/pipelines/controlnet/test_controlnet_sdxl.py
...
============================================================= 62 passed, 3 skipped, 165 warnings in 97.70s (0:01:37) ==============================================================

Before submitting

Who can review?

Chunk and use single add_text_embeds and add_time_ids when using guess_mode and do_classifier_free_guidance
@sayakpaul

Copy link
Copy Markdown
Member

Being addressed in #4155. So, closing this one.

Thank you for your contribution!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@YashalKanungo@sayakpaul