Analysis of Classifier-Free Guidance Weight Schedulers

[rootonchair]

Model/Pipeline/Scheduler description

The paper's authors performs an analysis and proposes 1 line change in order to make Classifier-Free Guidance looks better

I personally run some test to confirm

SD1.5 DDIM scheduler, 50 steps, "a photograph of an astronaut riding a horse", seed: 1024
guidance scale: 7.5
static

linear (proposed)

guidance scale: 14.0
static

linear (proposed)

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Paper: https://arxiv.org/abs/2404.13040

[rootonchair]

Update SDXL result
guidance scale: 14.0
static

linear

[DN6]

cc: @yiyixuxu and @asomoza for visibility.

[bghira]

what a neat way to make use of the knowledge already in the model!

[yiyixuxu]

cc @asomoza can we make a callback for this?

[asomoza]

yes but there's a lot of techniques about manipulating the CFG, most of them without papers, I added the cutout one because I know it's really popular, makes the generations faster, and as a kind of example on how to manipulate the CFG

Maybe we should let the community add these ones later on? Other more popular ones are automatic cfg and Dynamic Thresholding

[YunhoKim21]

Hi, interested on your work. Is there an explanation of WHY this phenomena happens?

[bghira]

i'm not an expert on this, but on a cursory glance it seems to be basing the strength of guidance by the position in the timestep schedule. this also likely works because there's two types of attention being used by the model, with earlier timesteps being cross-attn (heavily relying on text conditional input) and later timesteps being self-attn (practically ignoring the prompt)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

@yiyixuxu @asomoza @a-r-r-o-w this could be a good candidate for modular diffusers no?

[bghira]

modular diffusers?

[asomoza]

this could be a good candidate for modular diffusers no?

yes, actually all kind of variations the the CFG and also all the techniques that use the callbacks are the best initial candidates for the modular diffusers.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.