[Feature] Dynamically adjust the guidance scale for ControlNet XL.

[pixeli99]

Before

prompt1="a couple watching a romantic sunset, 4k photo"generator=torch.manual_seed(123)
image=pipeline(
prompt1, num_images_per_prompt=4, generator=generator, image=pil_image_rgb, controlnet_conditioning_scale=0.5, guidance_scale=5.0,
).images

After

What

For the original parameters, Controlnetxl's output will darken the colors. This is particularly severe when the text prompt length is short. Dynamically selecting guidance scale can alleviate this phenomenon and allow for the use of larger control weight without deteriorating image quality.

How

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[yiyixuxu]

Wow! super cool. Thanks for your PR!
Do we have a reference for where this technique is coming from?

Also, can we see more examples? The vibrant color is really lovely for this sunset photo. It would also be interesting to see how it changes color across different contexts.

[sayakpaul]

I concur with @yiyixuxu's comments. Thanks for contributing this!

[pixeli99]

Wow! super cool. Thanks for your PR! Do we have a reference for where this technique is coming from?

Also, can we see more examples? The vibrant color is really lovely for this sunset photo. It would also be interesting to see how it changes color across different contexts.

In fact, I did not find any references. It was just an accidental discovery during my own training of controlnet. This calculation method may not be rigorous.

Fortunately, I found a Chinese blog that explains this issue, hoping it will be helpful. https://zhuanlan.zhihu.com/p/587049023

Please note that this adaptive method is not always effective and sometimes may result in oversaturated colors. However, in most cases, we don't need to worry about this.

[sayakpaul]

@pixeli99 let us know if you need any help to shape this PR in a mergeable state.

[patrickvonplaten]

To allow for more flexibility for future changes could we maybe add a parameter

Suggested change

	use_dynamic_scale: bool=False,
	guidance_scale_type="static"# dynamic,

[patrickvonplaten]

A bit worried about cluttering the call API too much with too many parameters. Would like to think a bit about the broader context of prompt embed tuning/tweaking as described here:
#4778

before moving too fast with this PR

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.