Control SDXL classifier-free guidance scale with step granularity

[jamarju]

What does this PR do?

Please consider this PR that allows step-wise granular control of StableDiffusionXLPipeline's classifier-free guidance scale by passing a list of num_inference_steps floats instead of a single float in the guidance_scale argument.

By allowing granular control over CFG scales you can, for example, disable CFG on the last denoising steps and save a considerable amount of compute without a noticeable degradation of the image in most of the cases, as depicted above.

The above images were generated with SDXL conditioned on 3 different prompts over 30 denoising steps. On each column, CFG was set to 6.0 during the first S steps and then to 0.0 during the last 30-S steps for S in (0, 5, 10, 15, 20, 25, 30) by passing guidance_scale = [6.0] * S + [0.0] * (30-S).

Every 5 steps in which CFG is turned off cut generation time down by 7% on A100 hardware so, as an example, if you disable CFG for the last 15 steps, you save 21% of compute.

@patrickvonplaten@sayakpaul

[vladmandic]

this would also be a no-op if my suggestion of having pipeline state passed back as reference in per-step callback so callback could modify such values as it deemed fit.

[patrickvonplaten]

This should also be solved by: #4778

@yiyixuxu can you try to prioritize this a bit?

[patrickvonplaten]

Fully agree that we need better support for a schedule-type of design, we should probably in a first step map out all the different designs here

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[vladmandic]

i think this should now be resolved.

[yiyixuxu]

you can achieve this with callback now - checkout our doc here https://huggingface.co/docs/diffusers/main/en/using-diffusers/callback