Hunyuanvideo15

[yiyixuxu]

https://huggingface.co/collections/hunyuanvideo-community/hunyuanvideo-15

testing script

importtorchdtype=torch.bfloat16device="cuda:0"fromdiffusersimportHunyuanVideo15Pipeline, HunyuanVideo15ImageToVideoPipelinefromdiffusers.utilsimportexport_to_video, load_imaget2v_names= ["480p_t2v", "720p_t2v", "480p_t2v_distilled"]
num_frames=31# use a minimum number for testing, 121 is default# test t2vprompt="A close-up shot captures a scene on a polished, light-colored granite kitchen counter, illuminated by soft natural light from an unseen window. Initially, the frame focuses on a tall, clear glass filled with golden, translucent apple juice standing next to a single, shiny red apple with a green leaf still attached to its stem. The camera moves horizontally to the right. As the shot progresses, a white ceramic plate smoothly enters the frame, revealing a fresh arrangement of about seven or eight more apples, a mix of vibrant reds and greens, piled neatly upon it. A shallow depth of field keeps the focus sharply on the fruit and glass, while the kitchen backsplash in the background remains softly blurred. The scene is in a realistic style."seed=1fornameint2v_names:
print(f"Testing {name}...")
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
pipe=HunyuanVideo15Pipeline.from_pretrained(f"hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-{name}", torch_dtype=dtype)
pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()
generator=torch.Generator(device=device).manual_seed(seed)
video=pipe(
prompt=prompt,
generator=generator,
num_frames=num_frames,
num_inference_steps=50,
).frames[0]
export_to_video(video, f"yiyi_test_hy15_{name}_output.mp4", fps=24)
max_allocated=torch.cuda.max_memory_allocated() /1024**3# GBprint(f"Max Allocated Memory: {max_allocated:.2f} GB")
# test i2vi2v_names= ["480p_i2v", "720p_i2v", "480p_i2v_distilled", "720p_i2v_distilled"]
image=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/wan_i2v_input.JPG")
prompt="Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."seed=1fornameini2v_names:
print(f"Testing {name}...")
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
pipe=HunyuanVideo15ImageToVideoPipeline.from_pretrained(f"hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-{name}", torch_dtype=dtype)
pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()
generator=torch.Generator(device=device).manual_seed(seed)
video=pipe(
prompt=prompt,
generator=generator,
image=image,
num_frames=num_frames,
num_inference_steps=50,
).frames[0]
export_to_video(video, f"yiyi_test_hy15_{name}_output.mp4", fps=24)
max_allocated=torch.cuda.max_memory_allocated() /1024**3# GBprint(f"Max Allocated Memory: {max_allocated:.2f} GB")

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[tin2tin]

Thank you for working on this. Eagerly awaiting this one (Wan doesn't work for me).

[yiyixuxu]

@tin2tin
i will merge this soon, do you want to test it out? all the checkpoints are uploaded, you can find scripts in the PR description

[tin2tin]

I don't have time right now, but I'll definitely check it out later.

[tin2tin]

720p_t2v seems to be loading, but it's too heavy for me to run (using your example code) - it ended with a crash.

480p_t2v

HunyuanVideo 1.5 runs just fine on my setup in ComfyUI, so they must have found out how to optimize it seriously.

[yiyixuxu]

@tin2tin do you want to try with group offloading?

[tin2tin]

I only have 32 GB RAM, that's usually not enough for group offloading.

I did try using single file pre-quantized file, but I couldn't get it working.

[sayakpaul]

My comments are mostly nits! I can help with the tests.

[sayakpaul]

We could have the model class subclass from AutoencoderMixin to get rid of the common methods. Example:

diffusers/src/diffusers/models/autoencoders/autoencoder_kl.py

Line 38 in c3f4598

classAutoencoderKL(ModelMixin, AutoencoderMixin, ConfigMixin, FromOriginalModelMixin, PeftAdapterMixin):

[sayakpaul]

Not that important but we're still doing the AE decoding according to:

diffusers/src/diffusers/pipelines/hunyuan_video1_5/pipeline_hunyuan_video1_5.py

Line 826 in c3f4598

video=self.vae.decode(latents, return_dict=False)[0]

I got the impression that the DiT might directly predicting the pixel space.

[sayakpaul]

We can get rid of these common attention related methods if we subclass the model from AttentionMixin:

diffusers/src/diffusers/models/transformers/transformer_flux2.py

Lines 631 to 639 in 152f7ca

	classFlux2Transformer2DModel(
	ModelMixin,
	ConfigMixin,
	PeftAdapterMixin,
	FromOriginalModelMixin,
	FluxTransformer2DLoadersMixin,
	CacheMixin,
	AttentionMixin,
	):

[sayakpaul]

Interesting that for encoder_hidden_states_cond_emb, we do zeros_likes and for encoder_hidden_states_2_cond_emb, we do ones_like.

[sayakpaul]

(nit):

I would imagine that in case we're on pure T2V, image_embeds would be None. But if that's not the case (the current code suggests we might need image_embeds to be zeroes), we could change the type hint of image_embeds to image_embeds: torch.Tensor and make it a positional argument?

[yiyixuxu]

they actually create it all zero image_embeds for t2v

[sayakpaul]

I see. So, it's probably like:

• First condition embedding type: 0
• Second condition embedding type: 1
• Third condition embedding type: 2

[sayakpaul]

Started tests in #12759 :) Completing now.

[sayakpaul]

Okay I will work on adding the Hub variant for FA3 varlen so that we can ease the user-experience a bit here.

[sayakpaul]

(nit): We could recommend flash_hub and sage_hub here as backends instead to promote more the usage of the kernels-based backends. It will also keep things central to the Hub.

[yiyixuxu]

@bot /style

[github-actions]

Style bot fixed some files and pushed the changes.

[tin2tin]

Congratulations on the commit. Do you have any suggestions on what I could try to get it working on 24 GB VRAM (RTX 4090) and 32 GB RAM?