[docs] Model cards by stevhliu · Pull Request #11112 · huggingface/diffusers

stevhliu · 2025-03-18T20:47:05Z

~~🚧 WIP 🚧~~

Based on our discussions about making it easier to run video models by including some minimal code optimized for memory and inference speed, this PR refactors the model card (starting with CogVideoX, but eventually expanding to other models as well) to reflect that. This provides users with easy copy/paste code they can run.

Parallel to this effort is to also improve the generic video generation guide.

HuggingFaceDocBuilderDev · 2025-03-18T20:53:49Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hlky · 2025-04-03T06:00:26Z

Looks good on first impression, I will review it in depth later today, wanted to raise #10301 with you as it will help simplify the examples (in combination with #11130 for the quantization cases). Also, it would be cool to have the examples be configurable/update with options, to demonstrate here's an artist's (4o) impression of what it could look like:

sayakpaul

This is a very good start. Left some comments, let me know if they make sense.

sayakpaul · 2025-04-14T13:46:58Z

-|:---:|:---:|
-| [`THUDM/CogVideoX-5b-I2V`](https://huggingface.co/THUDM/CogVideoX-5b-I2V) | torch.bfloat16 |
-| [`THUDM/CogVideoX-1.5-5b-I2V`](https://huggingface.co/THUDM/CogVideoX-1.5-5b-I2V) | torch.bfloat16 |
+[CogVideoX](https://huggingface.co/papers/2408.06072) is a large diffusion transformer model - available in 2B and 5B parameters - designed to generate longer and more consistent videos from text. This model uses a 3D causal variational autoencoder to more efficiently process video data by reducing sequence length (and associated training compute) and preventing flickering in generated videos. An "expert" transformer with adaptive LayerNorm improves alignment between text and video, and 3D full attention helps accurately capture motion and time in generated videos.


This is okay but I would perhaps tackle the removal of the abstract section in a separate PR. Also, this does add an additional overload of coming up with a description for the paper. I would like to avoid that for now.

I think it'd be good to also tackle this now since for the new pipeline cards, we want to have a nice and complete example of what it should look like no?
Good point that adding a description of the paper adds additional overload, but I think its necessary, since we want to give users a version of the abstract that is more accessible (meaning using common everyday language) versus academic (inspired by @asomoza 's comment here)

I am a bit spread thin on this one. So, I will go with what the team prefers.

stevhliu · 2025-04-23T18:33:17Z

Thanks @hlky, those PRs look to be super nice for user experience and I'll update the code examples once it's merged! The configurable example is also really neat and maybe we can make a Space out of it and embed it in the docs? I'll probably have to follow up on this one in a separate PR though 😅

sayakpaul · 2025-04-28T15:30:23Z

@stevhliu sorry for the delay on my end. The changes look nice and I responded to some of the questions/comments you had. Perhaps after #11130, we could simplify the quantization examples a bit.

@a-r-r-o-w do we want to touch any other video models in this PR?

stevhliu · 2025-05-19T20:28:14Z

@sayakpaul, I simplified the examples with the new PipelineQuantizationConfig! Let me know if there are any other changes you'd like to see, otherwise I think we can merge!

sayakpaul

Looking much better and another round of feedback.

sayakpaul · 2025-05-20T02:56:11Z

+## Notes

-### Memory optimization
+- CogVideoX supports LoRAs with [`~loaders.CogVideoXLoraLoaderMixin.load_lora_weights`].


Do we need this separate note besides having the LoRA marker button at the top of the page?

I think it'd be nice to have a easy copy/paste example for users who want to use this specific model, will fold under collapsible section as suggested. I also added a link to the LoRA marker button at the top :)

sayakpaul · 2025-05-20T02:59:16Z

+<hfoption id="inference speed">

-Refer to the [Quantization](../../quantization/overview) overview to learn more about supported quantization backends and selecting a quantization backend that supports your use case. The example below demonstrates how to load a quantized [`HunyuanVideoPipeline`] for inference with bitsandbytes.
+Compilation is slow the first time but subsequent calls to the pipeline are faster.


For compilation, should we also refer the readers to our compilation guide?

Added link to the compile section in fp16.md and will combine torch2.0.md with it in a separate PR as discussed!

sayakpaul

Works for me. @DN6 @a-r-r-o-w do you want to review this as well? I think with the video models becoming more and more crucial these docs will be very important!

stevhliu · 2025-06-02T23:55:10Z

I'll merge for now and we can keep iterating on it if there is additional feedback! 🤗

* initial * update * hunyuanvideo * ltx * fix * wan * gen guide * feedback * feedback * pipeline-level quant config * feedback * ltx This reverts commit 2ebe9ca.

a-r-r-o-w · 2025-06-06T08:14:25Z

-export_to_video(output, "wan-i2v.mp4", fps=16)
-```
-
-### First and Last Frame Interpolation


Seems like this PR deleted the examples for different pipelines. Is this intended? I'm adding it back in the examples back in #11582 since we need it to be documented
Edit: Ah I see, it's the same as the example docstring, so makes sense. It's just the FLF2V documentation disappeared and I thought that some other things went missing too

hlky mentioned this pull request Mar 19, 2025
[Wan LoRAs] make T2V LoRAs compatible with Wan I2V #11107
Merged

stevhliu force-pushed the video branch 2 times, most recently from 9cda27b to b31ed8eCompare March 27, 2025 18:11

stevhliu force-pushed the video branch from 417c6a1 to b54c46bCompare March 31, 2025 23:05

stevhliu requested review from a-r-r-o-w, asomoza, hlky and sayakpaul March 31, 2025 23:38

sayakpaul reviewed Apr 14, 2025
View reviewed changes

stevhliu force-pushed the video branch from b54c46b to 2a4090bCompare April 23, 2025 18:14

stevhliu marked this pull request as ready for review April 23, 2025 18:21

stevhliu force-pushed the video branch from 09d9bb4 to 75e6a50Compare April 29, 2025 20:14

stevhliu force-pushed the video branch from ed7d298 to df85f84Compare May 19, 2025 18:15

sayakpaul reviewed May 20, 2025
View reviewed changes

stevhliu force-pushed the video branch from df85f84 to ddab9b4Compare May 20, 2025 16:42

sayakpaul approved these changes May 28, 2025
View reviewed changes

stevhliu added 9 commits June 2, 2025 15:08

initial
df1ed74

update
ef82096

hunyuanvideo
37a12e6

ltx
539c654

fix
3c6d331

wan
ba67eb2

gen guide
aa600ae

feedback
22d2c95

feedback
a8e720c

stevhliu added 3 commits June 2, 2025 15:09

pipeline-level quant config
2a15eee

feedback
d73f551

ltx
7384a76

stevhliu force-pushed the video branch from 8f0c9df to 7384a76Compare June 2, 2025 22:09

stevhliu merged commit c934720 into huggingface:mainJun 2, 2025

stevhliu deleted the video branch June 2, 2025 23:55

a-r-r-o-w reviewed Jun 6, 2025
View reviewed changes

Uh oh!

Conversation

stevhliu commented Mar 18, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Mar 18, 2025

Uh oh!

hlky commented Apr 3, 2025

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sayakpaulApr 14, 2025

Choose a reason for hiding this comment

Uh oh!

stevhliuApr 23, 2025

Choose a reason for hiding this comment

Uh oh!

sayakpaulApr 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stevhliu commented Apr 23, 2025

Uh oh!

sayakpaul commented Apr 28, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevhliu commented May 19, 2025

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaulMay 20, 2025

Choose a reason for hiding this comment

Uh oh!

stevhliuMay 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaulMay 20, 2025

Choose a reason for hiding this comment

Uh oh!

stevhliuMay 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

stevhliu commented Jun 2, 2025

Uh oh!

a-r-r-o-wJun 6, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

stevhliu commented Mar 18, 2025•
edited
Loading

sayakpaul commented Apr 28, 2025•
edited
Loading

a-r-r-o-wJun 6, 2025•
edited
Loading