hunyuandit model/pipeline review

[hlky]

hunyuandit model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Reviewed: target model/pipeline files, lazy exports, config/loading paths, dtype/device/offload behavior, attention processors, docs, examples, and tests. Fast and slow tests exist for both base and ControlNet pipelines, but ControlNet has several no-op fast tests noted below. Local .venv snippets were run; full pytest collection was blocked because this .venv torch build is missing torch._C._distributed_c10d.

Duplicate search: searched GitHub Issues and PRs for hunyuandit, target class/file names, from_transformer transformer_num_layers, prompt_embeds num_images_per_prompt attention_mask, MultiControlNet controlnet_conditioning_scale, _no_split_modules, and HunyuanAttnProcessor2_0 set_attention_backend. No exact duplicates found. Related closed issue: #9142 covered older multi-ControlNet loading, not the runtime scale/validation failures below.

Issue 1: from_transformer() cannot build a matching HunyuanDiT ControlNet

Affected code:

diffusers/src/diffusers/models/controlnets/controlnet_hunyuan.py

Lines 177 to 209 in 0f1abc4

	config = transformer.config
	activation_fn = config.activation_fn
	attention_head_dim = config.attention_head_dim
	cross_attention_dim = config.cross_attention_dim
	cross_attention_dim_t5 = config.cross_attention_dim_t5
	hidden_size = config.hidden_size
	in_channels = config.in_channels
	mlp_ratio = config.mlp_ratio
	num_attention_heads = config.num_attention_heads
	patch_size = config.patch_size
	sample_size = config.sample_size
	text_len = config.text_len
	text_len_t5 = config.text_len_t5
	conditioning_channels = conditioning_channels
	transformer_num_layers = transformer_num_layers or config.transformer_num_layers
	controlnet = cls(
	conditioning_channels=conditioning_channels,
	transformer_num_layers=transformer_num_layers,
	activation_fn=activation_fn,
	attention_head_dim=attention_head_dim,
	cross_attention_dim=cross_attention_dim,
	cross_attention_dim_t5=cross_attention_dim_t5,
	hidden_size=hidden_size,
	in_channels=in_channels,
	mlp_ratio=mlp_ratio,
	num_attention_heads=num_attention_heads,
	patch_size=patch_size,
	sample_size=sample_size,
	text_len=text_len,
	text_len_t5=text_len_t5,
	)

Problem:
HunyuanDiT2DControlNetModel.from_transformer() reads config.transformer_num_layers, but HunyuanDiT2DModel stores num_layers. Even when transformer_num_layers is passed manually, the method does not copy pooled_projection_dim or use_style_cond_and_image_meta_size, so weight loading can hit size mismatches.

Impact:
The public constructor for deriving a ControlNet from a transformer is broken for normal HunyuanDiT transformers and can also create a ControlNet with incompatible conditioning embeddings.

Reproduction:

from diffusers import HunyuanDiT2DControlNetModel, HunyuanDiT2DModel

transformer = HunyuanDiT2DModel(
 sample_size=8, num_layers=4, patch_size=2,
 attention_head_dim=4, num_attention_heads=2, in_channels=4,
 cross_attention_dim=8, cross_attention_dim_t5=8,
 pooled_projection_dim=4, hidden_size=8, text_len=4, text_len_t5=4,
 use_style_cond_and_image_meta_size=False,
)

try:
 HunyuanDiT2DControlNetModel.from_transformer(transformer)
except Exception as e:
 print(type(e).__name__, str(e).splitlines()[0])

try:
 HunyuanDiT2DControlNetModel.from_transformer(transformer, transformer_num_layers=4)
except Exception as e:
 print(type(e).__name__, str(e).splitlines()[0])

Relevant precedent:

diffusers/src/diffusers/models/controlnets/controlnet_flux.py

Lines 135 to 142 in 0f1abc4

	config = dict(transformer.config)
	config["num_layers"] = num_layers
	config["num_single_layers"] = num_single_layers
	config["attention_head_dim"] = attention_head_dim
	config["num_attention_heads"] = num_attention_heads
	controlnet = cls.from_config(config)

diffusers/src/diffusers/models/controlnets/controlnet_sd3.py

Lines 257 to 260 in 0f1abc4

	config = transformer.config
	config["num_layers"] = num_layers or config.num_layers
	config["extra_conditioning_channels"] = num_extra_conditioning_channels
	controlnet = cls.from_config(config)

Suggested fix:

config = transformer.config
transformer_num_layers = transformer_num_layers or config.num_layers

controlnet = cls(
 conditioning_channels=conditioning_channels,
 transformer_num_layers=transformer_num_layers,
 activation_fn=config.activation_fn,
 attention_head_dim=config.attention_head_dim,
 cross_attention_dim=config.cross_attention_dim,
 cross_attention_dim_t5=config.cross_attention_dim_t5,
 hidden_size=config.hidden_size,
 in_channels=config.in_channels,
 mlp_ratio=config.mlp_ratio,
 num_attention_heads=config.num_attention_heads,
 patch_size=config.patch_size,
 sample_size=config.sample_size,
 pooled_projection_dim=config.pooled_projection_dim,
 text_len=config.text_len,
 text_len_t5=config.text_len_t5,
 use_style_cond_and_image_meta_size=config.use_style_cond_and_image_meta_size,
)

Issue 2: Precomputed prompt masks are not repeated for num_images_per_prompt

Affected code:

diffusers/src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

Lines 349 to 362 in 0f1abc4

	prompt_attention_mask = text_inputs.attention_mask.to(device)
	prompt_embeds = text_encoder(
	text_input_ids.to(device),
	attention_mask=prompt_attention_mask,
	)
	prompt_embeds = prompt_embeds[0]
	prompt_attention_mask = prompt_attention_mask.repeat(num_images_per_prompt, 1)
	prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)
	bs_embed, seq_len, _ = prompt_embeds.shape
	# duplicate text embeddings for each generation per prompt, using mps friendly method
	prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
	prompt_embeds = prompt_embeds.view(bs_embed * num_images_per_prompt, seq_len, -1)

diffusers/src/diffusers/pipelines/hunyuandit/pipeline_hunyuandit.py

Lines 394 to 409 in 0f1abc4

	negative_prompt_attention_mask = uncond_input.attention_mask.to(device)
	negative_prompt_embeds = text_encoder(
	uncond_input.input_ids.to(device),
	attention_mask=negative_prompt_attention_mask,
	)
	negative_prompt_embeds = negative_prompt_embeds[0]
	negative_prompt_attention_mask = negative_prompt_attention_mask.repeat(num_images_per_prompt, 1)
	if do_classifier_free_guidance:
	# duplicate unconditional embeddings for each generation per prompt, using mps friendly method
	seq_len = negative_prompt_embeds.shape[1]
	negative_prompt_embeds = negative_prompt_embeds.to(dtype=dtype, device=device)
	negative_prompt_embeds = negative_prompt_embeds.repeat(1, num_images_per_prompt, 1)
	negative_prompt_embeds = negative_prompt_embeds.view(batch_size * num_images_per_prompt, seq_len, -1)

diffusers/src/diffusers/pipelines/controlnet_hunyuandit/pipeline_hunyuandit_controlnet.py

Lines 377 to 390 in 0f1abc4

	prompt_attention_mask = text_inputs.attention_mask.to(device)
	prompt_embeds = text_encoder(
	text_input_ids.to(device),
	attention_mask=prompt_attention_mask,
	)
	prompt_embeds = prompt_embeds[0]
	prompt_attention_mask = prompt_attention_mask.repeat(num_images_per_prompt, 1)
	prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)
	bs_embed, seq_len, _ = prompt_embeds.shape
	# duplicate text embeddings for each generation per prompt, using mps friendly method
	prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
	prompt_embeds = prompt_embeds.view(bs_embed * num_images_per_prompt, seq_len, -1)

diffusers/src/diffusers/pipelines/controlnet_hunyuandit/pipeline_hunyuandit_controlnet.py

Lines 422 to 437 in 0f1abc4

	negative_prompt_attention_mask = uncond_input.attention_mask.to(device)
	negative_prompt_embeds = text_encoder(
	uncond_input.input_ids.to(device),
	attention_mask=negative_prompt_attention_mask,
	)
	negative_prompt_embeds = negative_prompt_embeds[0]
	negative_prompt_attention_mask = negative_prompt_attention_mask.repeat(num_images_per_prompt, 1)
	if do_classifier_free_guidance:
	# duplicate unconditional embeddings for each generation per prompt, using mps friendly method
	seq_len = negative_prompt_embeds.shape[1]
	negative_prompt_embeds = negative_prompt_embeds.to(dtype=dtype, device=device)
	negative_prompt_embeds = negative_prompt_embeds.repeat(1, num_images_per_prompt, 1)
	negative_prompt_embeds = negative_prompt_embeds.view(batch_size * num_images_per_prompt, seq_len, -1)

Problem:
When users pass prompt_embeds and attention masks directly, encode_prompt() repeats embeddings for num_images_per_prompt but leaves the provided masks at the original batch size. Generated masks are repeated, but caller-provided masks are not.

Impact:
Batched precomputed embeddings fail when num_images_per_prompt > 1; with batch size 1 the mask broadcasts, hiding the bug.

Reproduction:

import torch
from diffusers import AutoencoderKL, DDPMScheduler, HunyuanDiT2DModel, HunyuanDiTPipeline

transformer = HunyuanDiT2DModel(
 sample_size=16, num_layers=2, patch_size=2,
 attention_head_dim=8, num_attention_heads=3, in_channels=4,
 cross_attention_dim=32, cross_attention_dim_t5=32,
 pooled_projection_dim=16, hidden_size=24,
).eval()

pipe = HunyuanDiTPipeline(
 vae=AutoencoderKL(), text_encoder=None, tokenizer=None,
 transformer=transformer, scheduler=DDPMScheduler(),
 safety_checker=None, feature_extractor=None,
 text_encoder_2=None, tokenizer_2=None, requires_safety_checker=False,
)
pipe.set_progress_bar_config(disable=True)

pipe(
 prompt_embeds=torch.randn(2, 77, 32),
 prompt_attention_mask=torch.ones(2, 77, dtype=torch.long),
 prompt_embeds_2=torch.randn(2, 256, 32),
 prompt_attention_mask_2=torch.ones(2, 256, dtype=torch.long),
 num_images_per_prompt=2, guidance_scale=1.0,
 num_inference_steps=1, height=16, width=16,
 output_type="latent", use_resolution_binning=False,
)

Relevant precedent:
The same method is copied into the ControlNet pipeline, so the fix should be applied at the source and propagated.

Suggested fix:

# Keep generated masks unrepeated in the generation branch, then normalize both paths once.
if prompt_embeds is None:
 ...
 prompt_attention_mask = text_inputs.attention_mask.to(device)
 prompt_embeds = text_encoder(...)[0]

prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)
prompt_attention_mask = prompt_attention_mask.to(device).repeat(num_images_per_prompt, 1)

...
if do_classifier_free_guidance and negative_prompt_embeds is None:
 ...
 negative_prompt_attention_mask = uncond_input.attention_mask.to(device)
 negative_prompt_embeds = text_encoder(...)[0]

if do_classifier_free_guidance:
 negative_prompt_embeds = negative_prompt_embeds.to(dtype=dtype, device=device)
 negative_prompt_attention_mask = negative_prompt_attention_mask.to(device).repeat(num_images_per_prompt, 1)
 ...

Issue 3: Multi-ControlNet inputs are neither normalized nor length-checked

Affected code:

diffusers/src/diffusers/pipelines/controlnet_hunyuandit/pipeline_hunyuandit_controlnet.py

Line 642 in 0f1abc4

controlnet_conditioning_scale: float | list[float] = 1.0,

diffusers/src/diffusers/pipelines/controlnet_hunyuandit/pipeline_hunyuandit_controlnet.py

Lines 867 to 888 in 0f1abc4

	elif isinstance(self.controlnet, HunyuanDiT2DMultiControlNetModel):
	control_images = []
	for control_image_ in control_image:
	control_image_ = self.prepare_image(
	image=control_image_,
	width=width,
	height=height,
	batch_size=batch_size * num_images_per_prompt,
	num_images_per_prompt=num_images_per_prompt,
	device=device,
	dtype=self.dtype,
	do_classifier_free_guidance=self.do_classifier_free_guidance,
	guess_mode=False,
	)
	control_image_ = self.vae.encode(control_image_).latent_dist.sample()
	control_image_ = control_image_ * self.vae.config.scaling_factor
	control_images.append(control_image_)
	control_image = control_images

diffusers/src/diffusers/models/controlnets/controlnet_hunyuan.py

Lines 374 to 400 in 0f1abc4

	for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)):
	block_samples = controlnet(
	hidden_states=hidden_states,
	timestep=timestep,
	controlnet_cond=image,
	conditioning_scale=scale,
	encoder_hidden_states=encoder_hidden_states,
	text_embedding_mask=text_embedding_mask,
	encoder_hidden_states_t5=encoder_hidden_states_t5,
	text_embedding_mask_t5=text_embedding_mask_t5,
	image_meta_size=image_meta_size,
	style=style,
	image_rotary_emb=image_rotary_emb,
	return_dict=return_dict,
	)
	# merge samples
	if i == 0:
	control_block_samples = block_samples
	else:
	control_block_samples = [
	control_block_sample + block_sample
	for control_block_sample, block_sample in zip(control_block_samples[0], block_samples[0])
	]
	control_block_samples = (control_block_samples,)
	return control_block_samples

Problem:
For HunyuanDiT2DMultiControlNetModel, conditioning_scale is iterated directly. The default scalar 1.0 raises TypeError, and mismatched lengths between control_image, conditioning_scale, and self.nets are silently truncated by zip().

Impact:
Multiple ControlNets are fragile: the documented scalar default fails, and missing images/scales can silently skip ControlNets.

Reproduction:

import torch
from diffusers import HunyuanDiT2DMultiControlNetModel

multi = HunyuanDiT2DMultiControlNetModel([torch.nn.Identity(), torch.nn.Identity()])
try:
 multi(torch.zeros(1), torch.zeros(1), [torch.zeros(1), torch.zeros(1)], conditioning_scale=1.0)
except Exception as e:
 print(type(e).__name__, str(e))

class DummyControlNet(torch.nn.Module):
 def __init__(self, value):
 super().__init__()
 self.value = value
 self.calls = 0
 def forward(self, *args, **kwargs):
 self.calls += 1
 return ([torch.tensor(float(self.value))],)

first, second = DummyControlNet(1), DummyControlNet(2)
multi = HunyuanDiT2DMultiControlNetModel([first, second])
out = multi(torch.zeros(1), torch.zeros(1), [torch.zeros(1)], conditioning_scale=[1.0, 1.0], return_dict=False)
print(out[0][0].item(), first.calls, second.calls)

Relevant precedent:

diffusers/src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet.py

Lines 981 to 990 in 0f1abc4

	# align format for control guidance
	if not isinstance(control_guidance_start, list) and isinstance(control_guidance_end, list):
	control_guidance_start = len(control_guidance_end) * [control_guidance_start]
	elif not isinstance(control_guidance_end, list) and isinstance(control_guidance_start, list):
	control_guidance_end = len(control_guidance_start) * [control_guidance_end]
	elif not isinstance(control_guidance_start, list) and not isinstance(control_guidance_end, list):
	mult = len(self.controlnet.nets) if isinstance(self.controlnet, SD3MultiControlNetModel) else 1
	control_guidance_start, control_guidance_end = (
	mult * [control_guidance_start],
	mult * [control_guidance_end],

diffusers/src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet.py

Lines 1172 to 1178 in 0f1abc4

	if isinstance(controlnet_keep[i], list):
	cond_scale = [c * s for c, s in zip(controlnet_conditioning_scale, controlnet_keep[i])]
	else:
	controlnet_cond_scale = controlnet_conditioning_scale
	if isinstance(controlnet_cond_scale, list):
	controlnet_cond_scale = controlnet_cond_scale[0]
	cond_scale = controlnet_cond_scale * controlnet_keep[i]

Suggested fix:

if isinstance(self.controlnet, HunyuanDiT2DMultiControlNetModel):
 count = len(self.controlnet.nets)

 if not isinstance(controlnet_conditioning_scale, list):
 controlnet_conditioning_scale = [controlnet_conditioning_scale] * count

 if not isinstance(control_image, list) or len(control_image) != count:
 raise ValueError(f"`control_image` must be a list of {count} images for multiple ControlNets.")

 if len(controlnet_conditioning_scale) != count:
 raise ValueError(f"`controlnet_conditioning_scale` must contain {count} values.")

Issue 4: HunyuanDiT models do not support device_map="auto"

Affected code:

diffusers/src/diffusers/models/transformers/hunyuan_transformer_2d.py

Lines 201 to 246 in 0f1abc4

	class HunyuanDiT2DModel(ModelMixin, AttentionMixin, ConfigMixin):
	"""
	HunYuanDiT: Diffusion model with a Transformer backbone.
	Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers.
	Parameters:
	num_attention_heads (`int`, *optional*, defaults to 16):
	The number of heads to use for multi-head attention.
	attention_head_dim (`int`, *optional*, defaults to 88):
	The number of channels in each head.
	in_channels (`int`, *optional*):
	The number of channels in the input and output (specify if the input is **continuous**).
	patch_size (`int`, *optional*):
	The size of the patch to use for the input.
	activation_fn (`str`, *optional*, defaults to `"geglu"`):
	Activation function to use in feed-forward.
	sample_size (`int`, *optional*):
	The width of the latent images. This is fixed during training since it is used to learn a number of
	position embeddings.
	dropout (`float`, *optional*, defaults to 0.0):
	The dropout probability to use.
	cross_attention_dim (`int`, *optional*):
	The number of dimension in the clip text embedding.
	hidden_size (`int`, *optional*):
	The size of hidden layer in the conditioning embedding layers.
	num_layers (`int`, *optional*, defaults to 1):
	The number of layers of Transformer blocks to use.
	mlp_ratio (`float`, *optional*, defaults to 4.0):
	The ratio of the hidden layer size to the input size.
	learn_sigma (`bool`, *optional*, defaults to `True`):
	Whether to predict variance.
	cross_attention_dim_t5 (`int`, *optional*):
	The number dimensions in t5 text embedding.
	pooled_projection_dim (`int`, *optional*):
	The size of the pooled projection.
	text_len (`int`, *optional*):
	The length of the clip text embedding.
	text_len_t5 (`int`, *optional*):
	The length of the T5 text embedding.
	use_style_cond_and_image_meta_size (`bool`, *optional*):
	Whether or not to use style condition and image meta size. True for version <=1.1, False for version >= 1.2
	"""
	_skip_layerwise_casting_patterns = ["pos_embed", "norm", "pooler"]
	_supports_group_offloading = False

diffusers/src/diffusers/models/controlnets/controlnet_hunyuan.py

Lines 40 to 42 in 0f1abc4

	class HunyuanDiT2DControlNetModel(ModelMixin, ConfigMixin):
	@register_to_config
	def __init__(

Problem:
Neither HunyuanDiT2DModel nor HunyuanDiT2DControlNetModel defines _no_split_modules. Diffusers rejects device_map="auto" for model classes without this attribute.

Impact:
Large HunyuanDiT checkpoints cannot use automatic model-level device placement, unlike related transformer/controlnet families.

Reproduction:

from diffusers import HunyuanDiT2DControlNetModel, HunyuanDiT2DModel

common = dict(
 sample_size=8, patch_size=2, attention_head_dim=4, num_attention_heads=2,
 in_channels=4, cross_attention_dim=8, cross_attention_dim_t5=8,
 pooled_projection_dim=4, hidden_size=8, text_len=4, text_len_t5=4,
)

for model in [
 HunyuanDiT2DModel(num_layers=1, **common),
 HunyuanDiT2DControlNetModel(transformer_num_layers=4, **common),
]:
 try:
 print(model.__class__.__name__, model._get_no_split_modules("auto"))
 except Exception as e:
 print(model.__class__.__name__, type(e).__name__, str(e).splitlines()[0])

Relevant precedent:

diffusers/src/diffusers/models/transformers/transformer_flux.py

Lines 566 to 568 in 0f1abc4

	_no_split_modules = ["FluxTransformerBlock", "FluxSingleTransformerBlock"]
	_skip_layerwise_casting_patterns = ["pos_embed", "norm"]
	_repeated_blocks = ["FluxTransformerBlock", "FluxSingleTransformerBlock"]

diffusers/src/diffusers/models/controlnets/controlnet_sana.py

Lines 43 to 44 in 0f1abc4

	_no_split_modules = ["SanaTransformerBlock", "PatchEmbed"]
	_skip_layerwise_casting_patterns = ["patch_embed", "norm"]

Suggested fix:

class HunyuanDiT2DModel(...):
 _no_split_modules = ["HunyuanDiTBlock", "PatchEmbed"]
 _skip_layerwise_casting_patterns = ["pos_embed", "norm", "pooler"]

class HunyuanDiT2DControlNetModel(...):
 _no_split_modules = ["HunyuanDiTBlock", "PatchEmbed"]
 _skip_layerwise_casting_patterns = ["pos_embed", "norm", "pooler"]

Issue 5: Attention backend selection is a no-op

Affected code:

diffusers/src/diffusers/models/transformers/hunyuan_transformer_2d.py

Lines 20 to 21 in 0f1abc4

	from ..attention import AttentionMixin, FeedForward
	from ..attention_processor import Attention, FusedHunyuanAttnProcessor2_0, HunyuanAttnProcessor2_0

diffusers/src/diffusers/models/transformers/hunyuan_transformer_2d.py

Lines 111 to 120 in 0f1abc4

	self.attn1 = Attention(
	query_dim=dim,
	cross_attention_dim=None,
	dim_head=dim // num_attention_heads,
	heads=num_attention_heads,
	qk_norm="layer_norm" if qk_norm else None,
	eps=1e-6,
	bias=True,
	processor=HunyuanAttnProcessor2_0(),
	)

diffusers/src/diffusers/models/attention_processor.py

Lines 3124 to 3201 in 0f1abc4

	class HunyuanAttnProcessor2_0:
	r"""
	Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). This is
	used in the HunyuanDiT model. It applies a s normalization layer and rotary embedding on query and key vector.
	"""
	def __init__(self):
	if not hasattr(F, "scaled_dot_product_attention"):
	raise ImportError("AttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.")
	def __call__(
	self,
	attn: Attention,
	hidden_states: torch.Tensor,
	encoder_hidden_states: torch.Tensor | None = None,
	attention_mask: torch.Tensor | None = None,
	temb: torch.Tensor | None = None,
	image_rotary_emb: torch.Tensor | None = None,
	) -> torch.Tensor:
	from .embeddings import apply_rotary_emb
	residual = hidden_states
	if attn.spatial_norm is not None:
	hidden_states = attn.spatial_norm(hidden_states, temb)
	input_ndim = hidden_states.ndim
	if input_ndim == 4:
	batch_size, channel, height, width = hidden_states.shape
	hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)
	batch_size, sequence_length, _ = (
	hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape
	)
	if attention_mask is not None:
	attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)
	# scaled_dot_product_attention expects attention_mask shape to be
	# (batch, heads, source_length, target_length)
	attention_mask = attention_mask.view(batch_size, attn.heads, -1, attention_mask.shape[-1])
	if attn.group_norm is not None:
	hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)
	query = attn.to_q(hidden_states)
	if encoder_hidden_states is None:
	encoder_hidden_states = hidden_states
	elif attn.norm_cross:
	encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)
	key = attn.to_k(encoder_hidden_states)
	value = attn.to_v(encoder_hidden_states)
	inner_dim = key.shape[-1]
	head_dim = inner_dim // attn.heads
	query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
	key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
	value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
	if attn.norm_q is not None:
	query = attn.norm_q(query)
	if attn.norm_k is not None:
	key = attn.norm_k(key)
	# Apply RoPE if needed
	if image_rotary_emb is not None:
	query = apply_rotary_emb(query, image_rotary_emb)
	if not attn.is_cross_attention:
	key = apply_rotary_emb(key, image_rotary_emb)
	# the output of sdp = (batch, num_heads, seq_len, head_dim)
	# TODO: add support for attn.scale when we move to Torch 2.1
	hidden_states = F.scaled_dot_product_attention(
	query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
	)

Problem:
HunyuanDiT uses legacy HunyuanAttnProcessor2_0 from attention_processor.py. That processor calls F.scaled_dot_product_attention directly and has no _attention_backend, so model.set_attention_backend("native") silently leaves all processors unchanged.

Impact:
Users cannot select modern attention backends for HunyuanDiT despite the model exposing the generic attention backend API.

Reproduction:

from diffusers import HunyuanDiT2DModel

model = HunyuanDiT2DModel(
 sample_size=8, num_layers=1, patch_size=2,
 attention_head_dim=4, num_attention_heads=2, in_channels=4,
 cross_attention_dim=8, cross_attention_dim_t5=8,
 pooled_projection_dim=4, hidden_size=8, text_len=4, text_len_t5=4,
)

print([hasattr(p, "_attention_backend") for p in model.attn_processors.values()])
model.set_attention_backend("native")
print([getattr(p, "_attention_backend", None) for p in model.attn_processors.values()])

Relevant precedent:

diffusers/src/diffusers/models/transformers/transformer_flux.py

Lines 75 to 124 in 0f1abc4

	class FluxAttnProcessor:
	_attention_backend = None
	_parallel_config = None
	def __init__(self):
	if not hasattr(F, "scaled_dot_product_attention"):
	raise ImportError(f"{self.__class__.__name__} requires PyTorch 2.0. Please upgrade your pytorch version.")
	def __call__(
	self,
	attn: "FluxAttention",
	hidden_states: torch.Tensor,
	encoder_hidden_states: torch.Tensor = None,
	attention_mask: torch.Tensor | None = None,
	image_rotary_emb: torch.Tensor | None = None,
	) -> torch.Tensor:
	query, key, value, encoder_query, encoder_key, encoder_value = _get_qkv_projections(
	attn, hidden_states, encoder_hidden_states
	)
	query = query.unflatten(-1, (attn.heads, -1))
	key = key.unflatten(-1, (attn.heads, -1))
	value = value.unflatten(-1, (attn.heads, -1))
	query = attn.norm_q(query)
	key = attn.norm_k(key)
	if attn.added_kv_proj_dim is not None:
	encoder_query = encoder_query.unflatten(-1, (attn.heads, -1))
	encoder_key = encoder_key.unflatten(-1, (attn.heads, -1))
	encoder_value = encoder_value.unflatten(-1, (attn.heads, -1))
	encoder_query = attn.norm_added_q(encoder_query)
	encoder_key = attn.norm_added_k(encoder_key)
	query = torch.cat([encoder_query, query], dim=1)
	key = torch.cat([encoder_key, key], dim=1)
	value = torch.cat([encoder_value, value], dim=1)
	if image_rotary_emb is not None:
	query = apply_rotary_emb(query, image_rotary_emb, sequence_dim=1)
	key = apply_rotary_emb(key, image_rotary_emb, sequence_dim=1)
	hidden_states = dispatch_attention_fn(
	query,
	key,
	value,
	attn_mask=attention_mask,
	backend=self._attention_backend,
	parallel_config=self._parallel_config,

Suggested fix:
Move Hunyuan attention processors to the model-file pattern used by newer transformers, add _attention_backend = None and _parallel_config = None, and route attention through dispatch_attention_fn(...) instead of calling F.scaled_dot_product_attention directly.

Issue 6: ControlNet fast tests contain passing TODO stubs

Affected code:

diffusers/tests/pipelines/controlnet_hunyuandit/test_controlnet_hunyuandit.py

Lines 177 to 187 in 0f1abc4

	def test_sequential_cpu_offload_forward_pass(self):
	# TODO(YiYi) need to fix later
	pass
	def test_sequential_offload_forward_pass_twice(self):
	# TODO(YiYi) need to fix later
	pass
	def test_save_load_optional_components(self):
	# TODO(YiYi) need to fix later
	pass

Problem:
test_sequential_cpu_offload_forward_pass, test_sequential_offload_forward_pass_twice, and test_save_load_optional_components contain only TODO comments and pass, so pytest reports them as successful without checking anything.

Impact:
The target’s ControlNet offload and serialization coverage is weaker than it appears, and regressions in optional component save/load behavior are not exercised.

Reproduction:

from pathlib import Path

text = Path("tests/pipelines/controlnet_hunyuandit/test_controlnet_hunyuandit.py").read_text()
for name in [
 "test_sequential_cpu_offload_forward_pass",
 "test_sequential_offload_forward_pass_twice",
 "test_save_load_optional_components",
]:
 start = text.index(f"def {name}")
 end = text.find("\n def ", start + 1)
 block = text[start:end if end != -1 else len(text)]
 print(name, "TODO" in block and "pass" in block)

Relevant precedent:

diffusers/tests/pipelines/hunyuandit/test_hunyuan_dit.py

Lines 129 to 145 in 0f1abc4

	@unittest.skip("The HunyuanDiT Attention pooling layer does not support sequential CPU offloading.")
	def test_sequential_cpu_offload_forward_pass(self):
	# TODO(YiYi) need to fix later
	# This is because it instantiates it's attention layer from torch.nn.MultiheadAttention, which calls to
	# `torch.nn.functional.multi_head_attention_forward` with the weights and bias. Since the hook is never
	# triggered with a forward pass call, the weights stay on the CPU. There are more examples where we skip
	# this test because of MHA (example: HunyuanVideo Framepack)
	pass
	@unittest.skip("The HunyuanDiT Attention pooling layer does not support sequential CPU offloading.")
	def test_sequential_offload_forward_pass_twice(self):
	# TODO(YiYi) need to fix later
	# This is because it instantiates it's attention layer from torch.nn.MultiheadAttention, which calls to
	# `torch.nn.functional.multi_head_attention_forward` with the weights and bias. Since the hook is never
	# triggered with a forward pass call, the weights stay on the CPU. There are more examples where we skip
	# this test because of MHA (example: HunyuanVideo Framepack)
	pass

diffusers/tests/pipelines/hunyuandit/test_hunyuan_dit.py

Line 222 in 0f1abc4

def test_save_load_optional_components(self):

Suggested fix:
Implement the save/load optional component test for ControlNet, and either implement offload tests or explicitly skip them with the same unsupported-offload reason used by the base HunyuanDiT pipeline.