Handling cue word splicing in low-pass guidance

[lllzzz3]

if do_classifier_free_guidance and use_low_pass_guidance:
prompt_embeds_orig = prompt_embeds
prompt_embeds = torch.cat([negative_prompt_embeds, negative_prompt_embeds, prompt_embeds_orig], dim=0)
prompt_embeds_init = torch.cat([negative_prompt_embeds, prompt_embeds_orig], dim=0)
elif do_classifier_free_guidance:
prompt_embeds_orig = prompt_embeds
prompt_embeds_init = torch.cat([negative_prompt_embeds, prompt_embeds_orig], dim=0)
prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds_orig], dim=0)

Why does negative_prompt_embeds appear twice when handling low-pass guidance?

[choi403]

Hi @lllzzz3, thank you for your interest in our work.

As written in Eq.(3) in our paper, we use the original input image for the first term of the guidance equation.
This leads to requiring an extra forward pass (i.e., 3 distinct terms), unlike vanilla CFG, where only two terms need to be computed.
The additional negative prompt embedding you pointed out is the one we supply for this new term.
The two negative prompt embeddings should be identical in theory, but can be set to different values although the semantic meaning of this would become difficult to interpret.