feat: Add Motif-Video model and pipelines

[waitingcheung]

What does this PR do?

This PR adds support for Motif-Video - a text-to-video (T2V) and image-to-video (I2V) diffusion model from Motif Technologies. The implementation includes the transformer architecture, both pipeline variants, guiding configurations, and comprehensive documentation.

Changes

New Files

• Model: src/diffusers/models/transformers/transformer_motif_video.py - MotifVideoTransformer3DModel
• Pipelines:
  • src/diffusers/pipelines/motif_video/pipeline_motif_video.py - Text-to-Video
  • src/diffusers/pipelines/motif_video/pipeline_motif_video_image2video.py - Image-to-Video
• Output: src/diffusers/pipelines/motif_video/pipeline_output.py
• Tests:
  • tests/pipelines/motif_video/test_motif_video.py
  • tests/pipelines/motif_video/test_motif_video_image2video.py
• Documentation:
  • docs/source/en/api/models/motif_video_transformer_3d.md
  • docs/source/en/api/pipelines/motif_video.md

Key Features

• Architecture: DiT-based transformer with T5Gemma2Encoder for text encoding
• Flow Match: Uses FlowMatchEulerDiscreteScheduler
• Guiding: Supports ClassifierFreeGuidance, SkipLayerGuidance, and AdaptiveProjectedGuidance
• Video Processing: Wan-style VAE for video encoding/decoding

Version Requirements

• transformers>=5.1.0 - Required for T5Gemma2Encoder (critical bug fix in PR #43633)
• The pipeline includes a version check that raises a clear error with upgrade instructions if the transformers version is too old

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[waitingcheung]

@yiyixuxu@asomoza@sayakpaul

Quick ping for visibility. This PR adds Motif-Video (T2V/I2V + new transformer and pipelines).

Would appreciate your feedback, especially on dependency/version constraints:

• transformers>=5.1.0 for T5Gemma2Encoder (currently enforced via an assertion with an upgrade message)
• compel requiring transformers<5, which may conflict with diffusers usage

This is currently blocking some diffusers-side integration, so your input would help.

A working branch for this integration is available here.

[sayakpaul]

transformers>=5.1.0 for T5Gemma2Encoder (currently enforced via an assertion with an upgrade message)

I think we can guard the transformers import in the pipeline with something like is_transformers_version("<", "5.1.0")?

compel conflict is fine IMO.

[waitingcheung]

transformers>=5.1.0 for T5Gemma2Encoder (currently enforced via an assertion with an upgrade message)

I think we can guard the transformers import in the pipeline with something like is_transformers_version("<", "5.1.0")?

compel conflict is fine IMO.

We have something like this at the top of the pipeline code to guide the users to upgrade the transformers package before importing T5Gemma2Encoder

# Check transformers version before importing T5Gemma2Encoderifnotis_transformers_version(">=", "5.1.0"):
importtransformersraiseImportError(
f"MotifVideoPipeline requires transformers>=5.1.0. "f"Found: {transformers.__version__}. ""Please upgrade transformers: pip install transformers --upgrade"
)

[sayakpaul]

Then it will cut it.

[waitingcheung]

@dg845 , @yiyixuxu
I would appreciate your feedback on this PR when you have a moment.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

thanks, left some final comments/qustions
I think we can merge this soon

[yiyixuxu]

@claude-2-serge can you do a final review?

[github-actions]

🤗 Serge says:

⚠️ Review failed: HTTPError: 400 Client Error: Bad Request for url: https://api.anthropic.com/v1/chat/completions

[yiyixuxu]

@claude can you do a final review?

1. do not talk about "things that are done well" - only focus on TODOs if any
2. can you look up all the review activities here and check if there is any update needs to be made to https://github.com/huggingface/diffusers/tree/main/.ai

[github-actions]

Claude finished @yiyixuxu's task in 6m 6s —— View job

---

Final Review of PR #13551: Motif-Video

• Read .ai/ guidelines
• Review transformer model for remaining issues
• Review T2V pipeline for remaining issues
• Review I2V pipeline for remaining issues
• Review tests and supporting files
• Check unresolved review comments
• Check if .ai/ needs updates based on review activities
• Post findings

---

Remaining TODOs from open reviewer requests

1. _keep_in_fp32_modules missing — transformer_motif_video.py:768

@yiyixuxuasked to test this now that #13697 has merged. The model still lacks _keep_in_fp32_modules. Other DiT models (Wan, Helios, SkyReelsV2) all declare it. Suggested:

_keep_in_fp32_modules= ["time_text_embed", "norm"]

Once this is added, the ad hoc get_parameter_dtype + int8 guard at line 436–438 should be removable.

2. Inline _maybe_gradient_checkpoint_block — transformer_motif_video.py:887

@yiyixuxurequested inlining this function. The Wan transformer inlines gradient checkpointing directly in forward. Currently _maybe_gradient_checkpoint_block wraps a 4-line helper called in 3 separate loops (lines 983, 997, 1013). Inline the if torch.is_grad_enabled() and self.gradient_checkpointing: pattern directly in each loop, matching the Wan convention.

3. decoder_hidden_states = hidden_states.clone() is unconditional — transformer_motif_video.py:969

This clones the full hidden state tensor on every forward pass, but decoder_hidden_states is only used when self.num_decoder_layers > 0 (line 1008). Default config has num_decoder_layers=0, so this wastes memory. Move it inside the if self.num_decoder_layers > 0: block.

---

New findings

4. Inconsistent guider None-safety — pipeline_motif_video.py:613-751, pipeline_motif_video_image2video.py:762-876

Both pipelines check self.guider is not None in some places (lines 613–614, 714 in T2V) but call self.guider.set_state() unconditionally at line 728 (T2V) / 852 (I2V). If guider is None, the denoising loop crashes. Since guider is required for inference and not listed in _optional_components, remove the is not None guards and treat it as always-present — or add consistent None checks throughout. The current code is a partial mix that will confuse readers.

---

.ai/ updates

After tracing all review activities across this PR (3 rounds of review, ~50 inline comments from dg845/yiyixuxu/claude), I found the following patterns that surfaced repeatedly but are not currently documented in .ai/:

1. Missing guidance on separate cross-attention modules — .ai/models.md

The attention pattern section only documents the single self-attention case (MyModelAttention + MyModelAttnProcessor). This PR initially had cross-attention inlined in the block's forward, and two reviewers (yiyixuxu, dg845) independently asked for it to be refactored into a dedicated MotifVideoCrossAttention class with its own processor — routing through AttentionModuleMixin just like self-attention. This convention (separate attention class per attention type) is not documented anywhere.

Suggested addition to .ai/models.md attention pattern section:

When a model has both self-attention and cross-attention (or other attention variants), each attention type should have its own Attention-style class inheriting AttentionModuleMixin with its own processor. Don't inline cross-attention compute directly in the block's forward — route it through the processor system so custom attention backends and set_attn_processor apply uniformly.

2. Missing guidance on guider as a required component — .ai/pipelines.md

The guider abstraction is documented in modular.md but not in pipelines.md. This PR went through iterations of whether guider should be optional or required, with yiyixuxu settling on required (like scheduler). New pipeline authors would benefit from:

For pipelines using the guider abstraction, guider should be a required __init__ parameter (not in _optional_components). It can be stored in the model repo as a subfolder (like scheduler). See HunyuanVideo for the pattern.

These are suggestions — not blocking.

---

[dg845]

Thanks for your hard work on this PR! Left a few final comments :).

[sayakpaul]

@claude-2-serge can you do a final review?

[github-actions]

🤗 Serge says:

⚠️ Review failed: HTTPError: 400 Client Error: Bad Request for url: https://api.anthropic.com/v1/chat/completions