[WIP][Core] feat: enable true cfg in hunyuanvideo.

[sayakpaul]

What does this PR do?

Similar to how it's done in Flux, this PR adds support for doing CFG in HunyuanVideo.

Here are results: https://wandb.ai/sayakpaul/hunyuanvideo_cfg/runs/41dhd69p

Code

importtorchfromdiffusersimportHunyuanVideoPipeline, HunyuanVideoTransformer3DModelfromdiffusers.utilsimportexport_to_videoimportargparseprompt="A cat walks on the grass, realistic"negative_prompt="worst quality, inconsistent motion, blurry, jittery, distorted"defload_pipeline():
model_id="hunyuanvideo-community/HunyuanVideo"transformer=HunyuanVideoTransformer3DModel.from_pretrained(
model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe=HunyuanVideoPipeline.from_pretrained(
model_id, transformer=transformer, torch_dtype=torch.float16
).to("cuda")
returnpipeif__name__=="__main__":
parser=argparse.ArgumentParser()
parser.add_argument("--true_cfg_scale", type=float, default=1.0)
args=parser.parse_args()
pipe=load_pipeline()
output=pipe(
prompt="A cat walks on the grass, realistic",
negative_prompt=negative_prompt,
true_cfg_scale=args.true_cfg_scale,
height=320,
width=512,
num_frames=61,
generator=torch.manual_seed(0),
).frames[0]
path=f"output_cfg@{args.true_cfg_scale}.mp4"export_to_video(output, path, fps=15)

TODOs

• Add docs
• Add support for negative embeds in __call__()
• Add tests

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

@a-r-r-o-w a gentle ping.

[a-r-r-o-w]

Hi, thanks for your contribution!

The results seem a little too burnt to me and not very promising. Could we see more examples to determine if there's an improvement to quality considering there's ~2x more time required for generation?

[a-r-r-o-w]

Ohh, the PR is from you as well. I thought you asked me for review on a contributors PR lol

[sayakpaul]

Here are some more results: https://wandb.ai/sayakpaul/hunyuanvideo_cfg/runs/w4xbb90g.

The cat seems to be fishy 🐟

I ran it another subject (rocket): https://wandb.ai/sayakpaul/hunyuanvideo_cfg/runs/kebbcmvi and the benefits seem visually better, especially for this one:

true_cfg_0_d53f2d11117d53cc601c.mp4

Compared to the no CFG case:

true_cfg_0_17717d7d8b10d0ab0944.mp4

WDYT? The effects seem to be dependent on the subjects and the scenes being generated.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.