cogview4 model/pipeline review

[hlky]

cogview4 model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Duplicate-search status: gh search was API-rate-limited, so I used the GitHub connector searches for cogview4, affected classes/files, attention_mask, dispatch_attention_fn, sigmas, callback_on_step_end, CogView4PipelineOutput, CogView4ControlPipeline load_lora_weights, and test coverage. I found no exact duplicates for the findings below. Related: #10962 and #10966 fixed the same prompt/negative embed shape problem for the base pipeline, not the control pipeline. The cache-context part of Issue 3 is already broadly tracked by #12760.

Issue 1: Text attention masks do not actually mask tokens

Affected code:

diffusers/src/diffusers/models/transformers/transformer_cogview4.py

Lines 168 to 179 in 0f1abc4

	if attention_mask is not None:
	text_attn_mask = attention_mask
	assert text_attn_mask.dim() == 2, "the shape of text_attn_mask should be (batch_size, text_seq_length)"
	text_attn_mask = text_attn_mask.float().to(query.device)
	mix_attn_mask = torch.ones((batch_size, text_seq_length + image_seq_length), device=query.device)
	mix_attn_mask[:, :text_seq_length] = text_attn_mask
	mix_attn_mask = mix_attn_mask.unsqueeze(2)
	attn_mask_matrix = mix_attn_mask @ mix_attn_mask.transpose(1, 2)
	attention_mask = (attn_mask_matrix > 0).unsqueeze(1).to(query.dtype)
	hidden_states = F.scaled_dot_product_attention(
	query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Lines 624 to 648 in 0f1abc4

	with self.transformer.cache_context("cond"):
	noise_pred_cond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	)[0]
	# perform guidance
	if self.do_classifier_free_guidance:
	with self.transformer.cache_context("uncond"):
	noise_pred_uncond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=negative_prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	)[0]

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py

Lines 674 to 694 in 0f1abc4

	hidden_states=latent_model_input,
	encoder_hidden_states=prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	)[0]
	# perform guidance
	if self.do_classifier_free_guidance:
	noise_pred_uncond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=negative_prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,

diffusers/examples/cogview4-control/train_control_cogview4.py

Lines 1048 to 1055 in 0f1abc4

	attention_mask = tokenizer(
	prompts,
	padding="longest", # not use max length
	max_length=args.max_sequence_length,
	truncation=True,
	add_special_tokens=True,
	return_tensors="pt",
	).attention_mask.float()

Problem:
CogView4AttnProcessor converts the text mask to query.dtype before passing it to SDPA. Float SDPA masks are additive bias masks, so 0.0 does not block masked positions. The pipelines also never pass text masks to the transformer, even though _get_glm_embeds pads prompts to a multiple of 16. The training script separately builds an unpadded tokenizer mask, so its mask length can disagree with padded prompt_embeds.

Impact:
Padded or explicitly masked text tokens can affect image tokens. Batched prompts with different lengths can produce different results from single-prompt inference, and training can fail or train with invalid masks.

Reproduction:

import torch
from diffusers import CogView4Transformer2DModel

torch.manual_seed(0)
model = CogView4Transformer2DModel(
 patch_size=2, in_channels=4, out_channels=4, num_layers=1,
 attention_head_dim=4, num_attention_heads=2, text_embed_dim=8,
 time_embed_dim=8, condition_dim=4,
).eval()

hidden_states = torch.randn(1, 4, 8, 8)
encoder_hidden_states = torch.randn(1, 4, 8)
mask = torch.tensor([[0, 1, 1, 1]])

poisoned = encoder_hidden_states.clone()
poisoned[:, 0] = 1_000_000.0
cleaned = encoder_hidden_states.clone()
cleaned[:, 0] = 0.0

kwargs = dict(
 hidden_states=hidden_states,
 timestep=torch.tensor([1]),
 original_size=torch.tensor([[64, 64]]),
 target_size=torch.tensor([[64, 64]]),
 crop_coords=torch.tensor([[0, 0]]),
 attention_mask=mask,
 return_dict=False,
)

with torch.no_grad():
 out_poisoned = model(encoder_hidden_states=poisoned, **kwargs)[0]
 out_cleaned = model(encoder_hidden_states=cleaned, **kwargs)[0]

leak = (out_poisoned - out_cleaned).abs().max().item()
assert leak < 1e-6, f"masked token leaked into image output: {leak}"

Relevant precedent:

diffusers/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py

Lines 202 to 224 in 0f1abc4

	txt_tokens = self.tokenizer(
	txt, max_length=self.tokenizer_max_length + drop_idx, padding=True, truncation=True, return_tensors="pt"
	).to(device)
	encoder_hidden_states = self.text_encoder(
	input_ids=txt_tokens.input_ids,
	attention_mask=txt_tokens.attention_mask,
	output_hidden_states=True,
	)
	hidden_states = encoder_hidden_states.hidden_states[-1]
	split_hidden_states = self._extract_masked_hidden(hidden_states, txt_tokens.attention_mask)
	split_hidden_states = [e[drop_idx:] for e in split_hidden_states]
	attn_mask_list = [torch.ones(e.size(0), dtype=torch.long, device=e.device) for e in split_hidden_states]
	max_seq_len = max([e.size(0) for e in split_hidden_states])
	prompt_embeds = torch.stack(
	[torch.cat([u, u.new_zeros(max_seq_len - u.size(0), u.size(1))]) for u in split_hidden_states]
	)
	encoder_attention_mask = torch.stack(
	[torch.cat([u, u.new_zeros(max_seq_len - u.size(0))]) for u in attn_mask_list]
	)
	prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)
	return prompt_embeds, encoder_attention_mask

diffusers/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py

Lines 694 to 704 in 0f1abc4

	with self.transformer.cache_context("cond"):
	noise_pred = self.transformer(
	hidden_states=latents,
	timestep=timestep / 1000,
	guidance=guidance,
	encoder_hidden_states_mask=prompt_embeds_mask,
	encoder_hidden_states=prompt_embeds,
	img_shapes=img_shapes,
	attention_kwargs=self.attention_kwargs,
	return_dict=False,
	)[0]

diffusers/src/diffusers/models/transformers/transformer_qwenimage.py

Lines 946 to 952 in 0f1abc4

	if encoder_hidden_states_mask is not None:
	# Build joint mask: [text_mask, all_ones_for_image]
	batch_size, image_seq_len = hidden_states.shape[:2]
	image_mask = torch.ones((batch_size, image_seq_len), dtype=torch.bool, device=hidden_states.device)
	joint_attention_mask = torch.cat([encoder_hidden_states_mask, image_mask], dim=1)
	joint_attention_mask = joint_attention_mask[:, None, None, :]
	block_attention_kwargs["attention_mask"] = joint_attention_mask

Suggested fix:

# Keep this as bool. Do not cast to query.dtype.
attention_mask = (attn_mask_matrix > 0).unsqueeze(1)

Also return padded attention masks from _get_glm_embeds / encode_prompt, repeat them with num_images_per_prompt, and pass the conditional and unconditional masks to self.transformer(...). The training script should pad attention_mask with leading zeros using the same pad_length logic as _get_glm_embeds.

Issue 2: Attention backend selection is a no-op for CogView4

Affected code:

diffusers/src/diffusers/models/transformers/transformer_cogview4.py

Lines 114 to 124 in 0f1abc4

	class CogView4AttnProcessor:
	"""
	Processor for implementing scaled dot-product attention for the CogView4 model. It applies a rotary embedding on
	query and key vectors, but does not include spatial normalization.
	The processor supports passing an attention mask for text tokens. The attention mask should have shape (batch_size,
	text_seq_length) where 1 indicates a non-padded token and 0 indicates a padded token.
	"""
	def __init__(self):
	if not hasattr(F, "scaled_dot_product_attention"):

diffusers/src/diffusers/models/transformers/transformer_cogview4.py

Lines 178 to 179 in 0f1abc4

	hidden_states = F.scaled_dot_product_attention(
	query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False

diffusers/src/diffusers/models/transformers/transformer_cogview4.py

Lines 401 to 402 in 0f1abc4

	hidden_states = F.scaled_dot_product_attention(
	query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False

Problem:
The CogView4 processors call F.scaled_dot_product_attention directly and do not define _attention_backend / _parallel_config. model.set_attention_backend(...) therefore cannot steer CogView4 attention to the requested backend, contrary to the review rules.

Impact:
Users cannot opt into supported diffusers attention backends for CogView4, and context-parallel/backend validation cannot reason about this model correctly.

Reproduction:

from diffusers import CogView4Transformer2DModel

model = CogView4Transformer2DModel(
 patch_size=2, in_channels=4, out_channels=4, num_layers=1,
 attention_head_dim=4, num_attention_heads=2, text_embed_dim=8,
 time_embed_dim=8, condition_dim=4,
)
model.set_attention_backend("native")
processor = model.transformer_blocks[0].attn1.processor

assert hasattr(processor, "_attention_backend"), "CogView4 processor ignores set_attention_backend()"

Relevant precedent:

diffusers/src/diffusers/models/transformers/transformer_flux.py

Lines 75 to 123 in 0f1abc4

	class FluxAttnProcessor:
	_attention_backend = None
	_parallel_config = None
	def __init__(self):
	if not hasattr(F, "scaled_dot_product_attention"):
	raise ImportError(f"{self.__class__.__name__} requires PyTorch 2.0. Please upgrade your pytorch version.")
	def __call__(
	self,
	attn: "FluxAttention",
	hidden_states: torch.Tensor,
	encoder_hidden_states: torch.Tensor = None,
	attention_mask: torch.Tensor | None = None,
	image_rotary_emb: torch.Tensor | None = None,
	) -> torch.Tensor:
	query, key, value, encoder_query, encoder_key, encoder_value = _get_qkv_projections(
	attn, hidden_states, encoder_hidden_states
	)
	query = query.unflatten(-1, (attn.heads, -1))
	key = key.unflatten(-1, (attn.heads, -1))
	value = value.unflatten(-1, (attn.heads, -1))
	query = attn.norm_q(query)
	key = attn.norm_k(key)
	if attn.added_kv_proj_dim is not None:
	encoder_query = encoder_query.unflatten(-1, (attn.heads, -1))
	encoder_key = encoder_key.unflatten(-1, (attn.heads, -1))
	encoder_value = encoder_value.unflatten(-1, (attn.heads, -1))
	encoder_query = attn.norm_added_q(encoder_query)
	encoder_key = attn.norm_added_k(encoder_key)
	query = torch.cat([encoder_query, query], dim=1)
	key = torch.cat([encoder_key, key], dim=1)
	value = torch.cat([encoder_value, value], dim=1)
	if image_rotary_emb is not None:
	query = apply_rotary_emb(query, image_rotary_emb, sequence_dim=1)
	key = apply_rotary_emb(key, image_rotary_emb, sequence_dim=1)
	hidden_states = dispatch_attention_fn(
	query,
	key,
	value,
	attn_mask=attention_mask,
	backend=self._attention_backend,

diffusers/src/diffusers/models/transformers/transformer_qwenimage.py

Lines 490 to 569 in 0f1abc4

	class QwenDoubleStreamAttnProcessor2_0:
	"""
	Attention processor for Qwen double-stream architecture, matching DoubleStreamLayerMegatron logic. This processor
	implements joint attention computation where text and image streams are processed together.
	"""
	_attention_backend = None
	_parallel_config = None
	def __init__(self):
	if not hasattr(F, "scaled_dot_product_attention"):
	raise ImportError(
	"QwenDoubleStreamAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0."
	)
	def __call__(
	self,
	attn: Attention,
	hidden_states: torch.FloatTensor, # Image stream
	encoder_hidden_states: torch.FloatTensor = None, # Text stream
	encoder_hidden_states_mask: torch.FloatTensor = None,
	attention_mask: torch.FloatTensor | None = None,
	image_rotary_emb: torch.Tensor | None = None,
	) -> torch.FloatTensor:
	if encoder_hidden_states is None:
	raise ValueError("QwenDoubleStreamAttnProcessor2_0 requires encoder_hidden_states (text stream)")
	seq_txt = encoder_hidden_states.shape[1]
	# Compute QKV for image stream (sample projections)
	img_query = attn.to_q(hidden_states)
	img_key = attn.to_k(hidden_states)
	img_value = attn.to_v(hidden_states)
	# Compute QKV for text stream (context projections)
	txt_query = attn.add_q_proj(encoder_hidden_states)
	txt_key = attn.add_k_proj(encoder_hidden_states)
	txt_value = attn.add_v_proj(encoder_hidden_states)
	# Reshape for multi-head attention
	img_query = img_query.unflatten(-1, (attn.heads, -1))
	img_key = img_key.unflatten(-1, (attn.heads, -1))
	img_value = img_value.unflatten(-1, (attn.heads, -1))
	txt_query = txt_query.unflatten(-1, (attn.heads, -1))
	txt_key = txt_key.unflatten(-1, (attn.heads, -1))
	txt_value = txt_value.unflatten(-1, (attn.heads, -1))
	# Apply QK normalization
	if attn.norm_q is not None:
	img_query = attn.norm_q(img_query)
	if attn.norm_k is not None:
	img_key = attn.norm_k(img_key)
	if attn.norm_added_q is not None:
	txt_query = attn.norm_added_q(txt_query)
	if attn.norm_added_k is not None:
	txt_key = attn.norm_added_k(txt_key)
	# Apply RoPE
	if image_rotary_emb is not None:
	img_freqs, txt_freqs = image_rotary_emb
	img_query = apply_rotary_emb_qwen(img_query, img_freqs, use_real=False)
	img_key = apply_rotary_emb_qwen(img_key, img_freqs, use_real=False)
	txt_query = apply_rotary_emb_qwen(txt_query, txt_freqs, use_real=False)
	txt_key = apply_rotary_emb_qwen(txt_key, txt_freqs, use_real=False)
	# Concatenate for joint attention
	# Order: [text, image]
	joint_query = torch.cat([txt_query, img_query], dim=1)
	joint_key = torch.cat([txt_key, img_key], dim=1)
	joint_value = torch.cat([txt_value, img_value], dim=1)
	joint_hidden_states = dispatch_attention_fn(
	joint_query,
	joint_key,
	joint_value,
	attn_mask=attention_mask,
	dropout_p=0.0,
	is_causal=False,
	backend=self._attention_backend,

Suggested fix:

from ..attention_dispatch import dispatch_attention_fn

class CogView4AttnProcessor:
 _attention_backend = None
 _parallel_config = None

 ...
 hidden_states = dispatch_attention_fn(
 query,
 key,
 value,
 attn_mask=attention_mask,
 dropout_p=0.0,
 is_causal=False,
 backend=self._attention_backend,
 parallel_config=self._parallel_config,
 )

Apply the same pattern to CogView4TrainingAttnProcessor.

Issue 3: Custom sigmas are unusable

Affected code:

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Lines 104 to 110 in 0f1abc4

	if timesteps is not None and sigmas is not None:
	if not accepts_timesteps and not accepts_sigmas:
	raise ValueError(
	f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"
	f" timestep or sigma schedules. Please check whether you are using the correct scheduler."
	)
	scheduler.set_timesteps(timesteps=timesteps, sigmas=sigmas, device=device, **kwargs)

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Lines 587 to 605 in 0f1abc4

	timesteps = (
	np.linspace(self.scheduler.config.num_train_timesteps, 1.0, num_inference_steps)
	if timesteps is None
	else np.array(timesteps)
	)
	timesteps = timesteps.astype(np.int64).astype(np.float32)
	sigmas = timesteps / self.scheduler.config.num_train_timesteps if sigmas is None else sigmas
	mu = calculate_shift(
	image_seq_len,
	self.scheduler.config.get("base_image_seq_len", 256),
	self.scheduler.config.get("base_shift", 0.25),
	self.scheduler.config.get("max_shift", 0.75),
	)
	if XLA_AVAILABLE:
	timestep_device = "cpu"
	else:
	timestep_device = device
	timesteps, num_inference_steps = retrieve_timesteps(
	self.scheduler, num_inference_steps, timestep_device, timesteps, sigmas, mu=mu

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py

Lines 637 to 655 in 0f1abc4

	timesteps = (
	np.linspace(self.scheduler.config.num_train_timesteps, 1.0, num_inference_steps)
	if timesteps is None
	else np.array(timesteps)
	)
	timesteps = timesteps.astype(np.int64).astype(np.float32)
	sigmas = timesteps / self.scheduler.config.num_train_timesteps if sigmas is None else sigmas
	mu = calculate_shift(
	image_seq_len,
	self.scheduler.config.get("base_image_seq_len", 256),
	self.scheduler.config.get("base_shift", 0.25),
	self.scheduler.config.get("max_shift", 0.75),
	)
	if XLA_AVAILABLE:
	timestep_device = "cpu"
	else:
	timestep_device = device
	timesteps, num_inference_steps = retrieve_timesteps(
	self.scheduler, num_inference_steps, timestep_device, timesteps, sigmas, mu=mu

Problem:
When users pass sigmas, the pipelines still synthesize a default timesteps array of length num_inference_steps and pass both arrays to retrieve_timesteps. A short custom sigma schedule therefore fails before inference.

Impact:
The public sigmas argument is effectively broken unless users also arrange matching timesteps and num_inference_steps, which contradicts the docstring.

Reproduction:

import numpy as np
from diffusers import FlowMatchEulerDiscreteScheduler
from diffusers.pipelines.cogview4.pipeline_cogview4 import retrieve_timesteps

scheduler = FlowMatchEulerDiscreteScheduler(use_dynamic_shifting=True)
num_inference_steps = 50
sigmas = [1.0, 0.5]
timesteps = np.linspace(scheduler.config.num_train_timesteps, 1.0, num_inference_steps)
timesteps = timesteps.astype(np.int64).astype(np.float32)

retrieve_timesteps(scheduler, num_inference_steps, "cpu", timesteps, sigmas, mu=0.5)

Relevant precedent:

diffusers/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py

Lines 67 to 105 in 0f1abc4

	b = base_shift - m * base_seq_len
	mu = image_seq_len * m + b
	return mu
	# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.retrieve_timesteps
	def retrieve_timesteps(
	scheduler,
	num_inference_steps: int | None = None,
	device: str | torch.device | None = None,
	timesteps: list[int] | None = None,
	sigmas: list[float] | None = None,
	**kwargs,
	):
	r"""
	Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles
	custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`.
	Args:
	scheduler (`SchedulerMixin`):
	The scheduler to get timesteps from.
	num_inference_steps (`int`):
	The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps`
	must be `None`.
	device (`str` or `torch.device`, *optional*):
	The device to which the timesteps should be moved to. If `None`, the timesteps are not moved.
	timesteps (`list[int]`, *optional*):
	Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed,
	`num_inference_steps` and `sigmas` must be `None`.
	sigmas (`list[float]`, *optional*):
	Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed,
	`num_inference_steps` and `timesteps` must be `None`.
	Returns:
	`tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the
	second element is the number of inference steps.
	"""
	if timesteps is not None and sigmas is not None:
	raise ValueError("Only one of `timesteps` or `sigmas` can be passed. Please choose one to set custom values")

Suggested fix:

if timesteps is None and sigmas is None:
 timesteps = np.linspace(self.scheduler.config.num_train_timesteps, 1.0, num_inference_steps)
 timesteps = timesteps.astype(np.int64).astype(np.float32)
elif timesteps is not None:
 timesteps = np.array(timesteps).astype(np.int64).astype(np.float32)

if sigmas is None and timesteps is not None:
 sigmas = timesteps / self.scheduler.config.num_train_timesteps

timesteps, num_inference_steps = retrieve_timesteps(
 self.scheduler, num_inference_steps, timestep_device, timesteps, sigmas, mu=mu
)

Issue 4: Step-end callbacks receive sigma instead of timestep

Affected code:

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Lines 656 to 660 in 0f1abc4

	if callback_on_step_end is not None:
	callback_kwargs = {}
	for k in callback_on_step_end_tensor_inputs:
	callback_kwargs[k] = locals()[k]
	callback_outputs = callback_on_step_end(self, i, self.scheduler.sigmas[i], callback_kwargs)

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py

Lines 703 to 707 in 0f1abc4

	if callback_on_step_end is not None:
	callback_kwargs = {}
	for k in callback_on_step_end_tensor_inputs:
	callback_kwargs[k] = locals()[k]
	callback_outputs = callback_on_step_end(self, i, self.scheduler.sigmas[i], callback_kwargs)

Problem:
Both pipelines call callback_on_step_end(self, i, self.scheduler.sigmas[i], ...). The callback API and other pipelines pass the current timestep t.

Impact:
User callbacks that inspect timestep values, implement custom stopping, or coordinate with scheduler timesteps receive the wrong quantity.

Reproduction:

import torch
from diffusers import AutoencoderKL, CogView4Pipeline, CogView4Transformer2DModel, FlowMatchEulerDiscreteScheduler

transformer = CogView4Transformer2DModel(
 patch_size=2, in_channels=4, out_channels=4, num_layers=1,
 attention_head_dim=4, num_attention_heads=2, text_embed_dim=8,
 time_embed_dim=8, condition_dim=4, sample_size=8,
)
vae = AutoencoderKL(
 block_out_channels=[32, 64], in_channels=3, out_channels=3,
 down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
 up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"],
 latent_channels=4, sample_size=32,
)
scheduler = FlowMatchEulerDiscreteScheduler(use_dynamic_shifting=True)
pipe = CogView4Pipeline(None, None, vae, transformer, scheduler)
pipe.set_progress_bar_config(disable=True)

seen = []
def callback(pipe, i, t, kwargs):
 seen.append((float(t), float(pipe.scheduler.timesteps[i])))
 return kwargs

pipe(
 prompt_embeds=torch.randn(1, 4, 8),
 num_inference_steps=2,
 guidance_scale=1.0,
 height=16,
 width=16,
 output_type="latent",
 callback_on_step_end=callback,
)
assert seen[0][0] == seen[0][1], seen

Relevant precedent:

diffusers/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py

Lines 731 to 737 in 0f1abc4

	latents = latents.to(latents_dtype)
	if callback_on_step_end is not None:
	callback_kwargs = {}
	for k in callback_on_step_end_tensor_inputs:
	callback_kwargs[k] = locals()[k]
	callback_outputs = callback_on_step_end(self, i, t, callback_kwargs)

diffusers/src/diffusers/pipelines/flux/pipeline_flux.py

Lines 983 to 989 in 0f1abc4

	latents = latents.to(latents_dtype)
	if callback_on_step_end is not None:
	callback_kwargs = {}
	for k in callback_on_step_end_tensor_inputs:
	callback_kwargs[k] = locals()[k]
	callback_outputs = callback_on_step_end(self, i, t, callback_kwargs)

Suggested fix:

callback_outputs = callback_on_step_end(self, i, t, callback_kwargs)

Issue 5: Control pipeline is missing base CogView4 pipeline capabilities

Affected code:

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Line 137 in 0f1abc4

class CogView4Pipeline(DiffusionPipeline, CogView4LoraLoaderMixin):

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py

Line 139 in 0f1abc4

class CogView4ControlPipeline(DiffusionPipeline):

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py

Lines 399 to 404 in 0f1abc4

	if prompt_embeds is not None and negative_prompt_embeds is not None:
	if prompt_embeds.shape != negative_prompt_embeds.shape:
	raise ValueError(
	"`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but"
	f" got: `prompt_embeds` {prompt_embeds.shape} != `negative_prompt_embeds`"
	f" {negative_prompt_embeds.shape}."

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py

Lines 674 to 694 in 0f1abc4

	hidden_states=latent_model_input,
	encoder_hidden_states=prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	)[0]
	# perform guidance
	if self.do_classifier_free_guidance:
	noise_pred_uncond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=negative_prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,

diffusers/examples/cogview4-control/README.md

Lines 99 to 123 in 0f1abc4

	from diffusers import CogView4ControlPipeline
	from diffusers.utils import load_image
	from PIL import Image
	import numpy as np
	import torch
	pipe = CogView4ControlPipeline.from_pretrained("THUDM/CogView4-6B", torch_dtype=torch.bfloat16).to("cuda")
	pipe.load_lora_weights("...") # change this.
	open_pose = OpenposeDetector.from_pretrained("lllyasviel/Annotators")
	# prepare pose condition.
	url = "https://huggingface.co/Adapter/t2iadapter/resolve/main/people.jpg"
	image = load_image(url)
	image = open_pose(image, detect_resolution=512, image_resolution=1024)
	image = np.array(image)[:, :, ::-1]
	image = Image.fromarray(np.uint8(image))
	prompt = "A couple, 4k photo, highly detailed"
	gen_images = pipe(
	prompt=prompt,
	control_image=image,
	num_inference_steps=50,
	joint_attention_kwargs={"scale": 0.9},

Problem:
CogView4ControlPipeline does not inherit CogView4LoraLoaderMixin, although the control README shows pipe.load_lora_weights(...). It also retains the old strict prompt/negative embed shape check, even though CogView4 runs cond/uncond in separate transformer calls and the base pipeline was already relaxed. Finally, the control denoising loop does not set "cond" / "uncond" cache contexts; that cache-context facet is already tracked generally by issue #12760.

Impact:
Control LoRA loading is unavailable, split text-encoding workflows fail for valid prompt/negative embeddings with different sequence lengths, and cache hooks can share state between conditional and unconditional passes.

Reproduction:

import inspect
import torch
from diffusers import CogView4ControlPipeline

print("has load_lora_weights:", hasattr(CogView4ControlPipeline, "load_lora_weights"))

pipe = object.__new__(CogView4ControlPipeline)
pipe._callback_tensor_inputs = ["latents", "prompt_embeds", "negative_prompt_embeds"]
try:
 pipe.check_inputs(
 prompt=None,
 height=16,
 width=16,
 negative_prompt=None,
 callback_on_step_end_tensor_inputs=["latents"],
 prompt_embeds=torch.randn(1, 12, 8),
 negative_prompt_embeds=torch.randn(1, 4, 8),
 )
except Exception as e:
 print(type(e).__name__, e)

source = inspect.getsource(CogView4ControlPipeline.__call__)
print("uses cache_context:", 'cache_context("cond")' in source and 'cache_context("uncond")' in source)

Relevant precedent:

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Line 137 in 0f1abc4

class CogView4Pipeline(DiffusionPipeline, CogView4LoraLoaderMixin):

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Lines 624 to 648 in 0f1abc4

	with self.transformer.cache_context("cond"):
	noise_pred_cond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	)[0]
	# perform guidance
	if self.do_classifier_free_guidance:
	with self.transformer.cache_context("uncond"):
	noise_pred_uncond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=negative_prompt_embeds,
	timestep=timestep,
	original_size=original_size,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	)[0]

Related prior base-pipeline fix: #10966
Duplicate for cache contexts: #12760

Suggested fix:

from ...loaders import CogView4LoraLoaderMixin

class CogView4ControlPipeline(DiffusionPipeline, CogView4LoraLoaderMixin):
 ...

if prompt_embeds is not None and negative_prompt_embeds is not None:
 if prompt_embeds.shape[0] != negative_prompt_embeds.shape[0]:
 raise ValueError(...)
 if prompt_embeds.shape[-1] != negative_prompt_embeds.shape[-1]:
 raise ValueError(...)

with self.transformer.cache_context("cond"):
 noise_pred_cond = self.transformer(...)

with self.transformer.cache_context("uncond"):
 noise_pred_uncond = self.transformer(...)

Issue 6: Lazy export advertises the wrong output class

Affected code:

diffusers/src/diffusers/pipelines/cogview4/__init__.py

Line 15 in 0f1abc4

_import_structure = {"pipeline_output": ["CogView4PlusPipelineOutput"]}

diffusers/src/diffusers/pipelines/cogview4/pipeline_output.py

Lines 10 to 12 in 0f1abc4

	class CogView4PipelineOutput(BaseOutput):
	"""
	Output class for CogView3 pipelines.

Problem:
pipelines/cogview4/__init__.py exports CogView4PlusPipelineOutput, but pipeline_output.py defines CogView4PipelineOutput. The output docstring also says CogView3.

Impact:
from diffusers.pipelines.cogview4 import CogView4PipelineOutput fails, and autodoc/lazy import metadata is wrong.

Reproduction:

from diffusers.pipelines.cogview4 import CogView4PipelineOutput
print(CogView4PipelineOutput)

Relevant precedent:

diffusers/src/diffusers/pipelines/cogview3/__init__.py

Line 15 in 0f1abc4

_import_structure = {"pipeline_output": ["CogView3PlusPipelineOutput"]}

diffusers/src/diffusers/pipelines/cogview3/pipeline_output.py

Lines 10 to 12 in 0f1abc4

	class CogView3PipelineOutput(BaseOutput):
	"""
	Output class for CogView3 pipelines.

Suggested fix:

_import_structure = {"pipeline_output": ["CogView4PipelineOutput"]}

Also update the output docstring to say CogView4.

Issue 7: Control example documentation/script has runnable breakages

Affected code:

diffusers/examples/cogview4-control/README.md

Line 19 in 0f1abc4

accelerate launch train_control_lora_cogview4.py \

diffusers/examples/cogview4-control/README.md

Line 123 in 0f1abc4

joint_attention_kwargs={"scale": 0.9},

diffusers/examples/cogview4-control/train_control_cogview4.py

Lines 821 to 827 in 0f1abc4

	if args.only_target_transformer_blocks:
	cogview4_transformer.patch_embed.proj.requires_grad_(True)
	for name, module in cogview4_transformer.named_modules():
	if "transformer_blocks" in name:
	module.requires_grad_(True)
	else:
	module.requirs_grad_(False)

Problem:
The README’s primary LoRA command references train_control_lora_cogview4.py, but that file is not present. The inference snippet passes joint_attention_kwargs, while the pipeline argument is attention_kwargs. The full fine-tuning script has module.requirs_grad_(False), which fails when --only_target_transformer_blocks is used.

Impact:
Users following the example cannot launch the documented LoRA training path, can pass an unsupported inference kwarg, and can hit an AttributeError in the full fine-tuning script.

Reproduction:

from pathlib import Path
import torch.nn as nn

assert Path("examples/cogview4-control/train_control_lora_cogview4.py").exists()
nn.Linear(1, 1).requirs_grad_(False)

Relevant precedent:
The README itself says inference is performed with CogView4ControlPipeline, whose signature exposes attention_kwargs, not joint_attention_kwargs.

Suggested fix:

# train_control_cogview4.py
module.requires_grad_(False)

Add the missing LoRA script or remove/update that section, and change the README inference kwarg to:

attention_kwargs={"scale": 0.9}

Issue 8: Slow tests and ControlPipeline tests are missing

Affected code:

diffusers/tests/pipelines/cogview4/test_cogview4.py

Line 32 in 0f1abc4

class CogView4PipelineFastTests(PipelineTesterMixin, unittest.TestCase):

Problem:
The CogView4 test file only defines CogView4PipelineFastTests. There is no @slow CogView4 integration test, and there is no fast or slow coverage for CogView4ControlPipeline.

Impact:
The control pipeline can regress independently from the base pipeline, and real checkpoint loading/inference is untested. This also leaves the slow-test requirement for the target family unmet.

Reproduction:

from pathlib import Path

test_files = list(Path("tests").rglob("*cogview4*.py"))
text = "\n".join(path.read_text(encoding="utf-8") for path in test_files)

print("test files:", [str(p) for p in test_files])
print("@slow present:", "@slow" in text)
print("CogView4ControlPipeline tested:", "CogView4ControlPipeline" in text)

Relevant precedent:

diffusers/tests/pipelines/cogview3/test_cogview3plus.py

Lines 241 to 264 in 0f1abc4

	@slow
	@require_torch_accelerator
	class CogView3PlusPipelineIntegrationTests(unittest.TestCase):
	prompt = "A painting of a squirrel eating a burger."
	def setUp(self):
	super().setUp()
	gc.collect()
	backend_empty_cache(torch_device)
	def tearDown(self):
	super().tearDown()
	gc.collect()
	backend_empty_cache(torch_device)
	def test_cogview3plus(self):
	generator = torch.Generator("cpu").manual_seed(0)
	pipe = CogView3PlusPipeline.from_pretrained("THUDM/CogView3Plus-3b", torch_dtype=torch.float16)
	pipe.enable_model_cpu_offload(device=torch_device)
	prompt = self.prompt
	images = pipe(
	prompt=prompt,

Suggested fix:
Add a CogView4ControlPipelineFastTests class with tiny synthetic components/control images, and add at least one @slow smoke test for the released CogView4 checkpoint family, including the control pipeline when the checkpoint is available to CI.