glm_image model/pipeline review

[hlky]

glm_image model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Duplicate-search status: searched huggingface/diffusers Issues and PRs for glm_image, GLM-Image, GlmImagePipeline, GlmImageTransformer2DModel, attention_mask, prompt_embeds dtype device, check_inputs width, transformer version gating, and slow-test coverage. I found no duplicates for the findings below. Related but not duplicates: PR #12974 adjusted GLM transformer-version gating, PR #13007 added batch support, PR #13344 added model tests, and issue #13227 tracks an MPS loading corruption issue.

Issue 1: Attention masks do not actually mask padded text tokens

Affected code:

diffusers/src/diffusers/models/transformers/transformer_glm_image.py

Lines 317 to 325 in 0f1abc4

	if attention_mask is not None:
	text_attn_mask = attention_mask
	assert text_attn_mask.dim() == 2, "the shape of text_attn_mask should be (batch_size, text_seq_length)"
	text_attn_mask = text_attn_mask.float().to(query.device)
	mix_attn_mask = torch.ones((batch_size, text_seq_length + image_seq_length), device=query.device)
	mix_attn_mask[:, :text_seq_length] = text_attn_mask
	mix_attn_mask = mix_attn_mask.unsqueeze(2)
	attn_mask_matrix = mix_attn_mask @ mix_attn_mask.transpose(1, 2)
	attention_mask = (attn_mask_matrix > 0).unsqueeze(1).to(query.dtype)

diffusers/src/diffusers/pipelines/glm_image/pipeline_glm_image.py

Lines 518 to 543 in 0f1abc4

	attention_mask = torch.tensor(
	[[1] * len(input_ids_) + [0] * (max_length - len(input_ids_)) for input_ids_ in input_ids],
	device=device,
	)
	input_ids = torch.tensor(
	[
	input_ids_ + [self.tokenizer.pad_token_id] * (max_length - len(input_ids_))
	for input_ids_ in input_ids
	],
	device=device,
	)
	outputs = self.text_encoder(input_ids, attention_mask=attention_mask)
	glyph_embeds = outputs.last_hidden_state[attention_mask.bool()].unsqueeze(0)
	all_glyph_embeds.append(glyph_embeds)
	# Pad to same sequence length and stack (use left padding to match transformers)
	max_seq_len = max(emb.size(1) for emb in all_glyph_embeds)
	padded_embeds = []
	for emb in all_glyph_embeds:
	if emb.size(1) < max_seq_len:
	pad = torch.zeros(emb.size(0), max_seq_len - emb.size(1), emb.size(2), device=device, dtype=emb.dtype)
	emb = torch.cat([pad, emb], dim=1) # left padding
	padded_embeds.append(emb)
	glyph_embeds = torch.cat(padded_embeds, dim=0)
	return glyph_embeds.to(device=device, dtype=dtype)

diffusers/src/diffusers/pipelines/glm_image/pipeline_glm_image.py

Lines 972 to 999 in 0f1abc4

	noise_pred_cond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=prompt_embeds,
	prior_token_id=prior_token_ids,
	prior_token_drop=prior_token_drop_cond,
	timestep=timestep,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	kv_caches=kv_caches,
	)[0].float()
	# perform guidance
	if self.do_classifier_free_guidance:
	if prior_token_image_ids_per_sample is not None:
	kv_caches.set_mode("skip")
	noise_pred_uncond = self.transformer(
	hidden_states=latent_model_input,
	encoder_hidden_states=negative_prompt_embeds,
	prior_token_id=prior_token_ids,
	prior_token_drop=prior_token_drop_uncond,
	timestep=timestep,
	target_size=target_size,
	crop_coords=crops_coords_top_left,
	attention_kwargs=attention_kwargs,
	return_dict=False,
	kv_caches=kv_caches,

Problem:
GlmImageAttnProcessor converts a boolean padding mask into a dense float 0/1 tensor. SDPA treats float masks as additive attention bias, so 0 does not block a token. The pipeline also discards the glyph encoder padding mask after constructing padded glyph embeddings, so batched variable-length glyph prompts have no valid text mask passed to the transformer.

Impact:
Masked or padded text tokens can still affect image tokens. This can make batched outputs differ from equivalent single-prompt outputs and makes the public attention_mask argument misleading. It also prevents the bool-mask varlen path described in the review rules.

Reproduction:

import torch
from diffusers.models.attention_processor import Attention
from diffusers.models.transformers.transformer_glm_image import GlmImageAttnProcessor

attn = Attention(query_dim=4, heads=1, dim_head=4, out_dim=4, bias=False, processor=GlmImageAttnProcessor())
with torch.no_grad():
 attn.to_q.weight.zero_()
 attn.to_k.weight.zero_()
 attn.to_v.weight.copy_(torch.eye(4))
 attn.to_out[0].weight.copy_(torch.eye(4))

encoder_hidden_states = torch.tensor([[[0., 0., 0., 0.], [1000., 0., 0., 0.]]])
hidden_states = torch.zeros(1, 1, 4)
attention_mask = torch.tensor([[True, False]])

image_out, _ = attn(
 hidden_states=hidden_states,
 encoder_hidden_states=encoder_hidden_states,
 attention_mask=attention_mask,
)
print(image_out[0, 0, 0].item()) # current: ~155.66; expected: 0 if token 2 is masked

Relevant precedent:

diffusers/src/diffusers/models/transformers/transformer_qwenimage.py

Lines 946 to 952 in 0f1abc4

	if encoder_hidden_states_mask is not None:
	# Build joint mask: [text_mask, all_ones_for_image]
	batch_size, image_seq_len = hidden_states.shape[:2]
	image_mask = torch.ones((batch_size, image_seq_len), dtype=torch.bool, device=hidden_states.device)
	joint_attention_mask = torch.cat([encoder_hidden_states_mask, image_mask], dim=1)
	joint_attention_mask = joint_attention_mask[:, None, None, :]
	block_attention_kwargs["attention_mask"] = joint_attention_mask

diffusers/src/diffusers/pipelines/qwenimage/pipeline_qwenimage.py

Lines 695 to 704 in 0f1abc4

	noise_pred = self.transformer(
	hidden_states=latents,
	timestep=timestep / 1000,
	guidance=guidance,
	encoder_hidden_states_mask=prompt_embeds_mask,
	encoder_hidden_states=prompt_embeds,
	img_shapes=img_shapes,
	attention_kwargs=self.attention_kwargs,
	return_dict=False,
	)[0]

Suggested fix:

# In GlmImageAttnProcessor.__call__, keep a bool key mask instead of a float QK matrix.
if attention_mask is not None:
 attention_mask = attention_mask.to(device=query.device, dtype=torch.bool)
 cached_seq_length = key.shape[1] - text_seq_length - image_seq_length
 cache_mask = torch.ones((batch_size, cached_seq_length), device=query.device, dtype=torch.bool)
 image_mask = torch.ones((batch_size, image_seq_length), device=query.device, dtype=torch.bool)
 attention_mask = torch.cat([cache_mask, attention_mask, image_mask], dim=1)

Also return a glyph padding mask from _get_glyph_embeds and pass it through the conditional and unconditional transformer calls.

Issue 2: Width validation accepts invalid resolutions and silently truncates latents

Affected code:

diffusers/src/diffusers/pipelines/glm_image/pipeline_glm_image.py

Lines 633 to 642 in 0f1abc4

	if (
	height is not None
	and height % (self.vae_scale_factor * self.transformer.config.patch_size * 2) != 0
	or width is not None
	and width % (self.transformer.config.patch_size * 2) != 0
	):
	# GLM-Image uses 32× downsampling, so the image dimensions must be multiples of 32.
	raise ValueError(
	f"`height` and `width` have to be divisible by {self.vae_scale_factor * 4} but are {height} and {width}."
	)

Problem:
check_inputs validates height with vae_scale_factor * patch_size * 2, but validates width only with patch_size * 2. With default GLM settings this accepts widths divisible by 4, even though the pipeline later floors latent width by width // vae_scale_factor.

Impact:
Invalid widths pass validation and produce latents for a smaller decoded width than requested.

Reproduction:

import torch
from diffusers import GlmImagePipeline

class Config:
 patch_size = 2

class Transformer:
 config = Config()

pipe = object.__new__(GlmImagePipeline)
pipe.vae_scale_factor = 8
pipe.transformer = Transformer()
pipe._callback_tensor_inputs = ["latents", "prompt_embeds"]

pipe.check_inputs(prompt="x", height=32, width=20, callback_on_step_end_tensor_inputs=["latents"])
latents = pipe.prepare_latents(1, 4, 32, 20, torch.float32, torch.device("cpu"), torch.Generator().manual_seed(0))
print(latents.shape[-1] * pipe.vae_scale_factor) # 16, not requested width 20

Relevant precedent:

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Lines 328 to 329 in 0f1abc4

	if height % 16 != 0 or width % 16 != 0:
	raise ValueError(f"`height` and `width` have to be divisible by 16 but are {height} and {width}.")

Suggested fix:

multiple_of = self.vae_scale_factor * self.transformer.config.patch_size * 2
if (height is not None and height % multiple_of != 0) or (width is not None and width % multiple_of != 0):
 raise ValueError(f"`height` and `width` have to be divisible by {multiple_of} but are {height} and {width}.")

Issue 3: Precomputed conditioning tensors are not moved or cast

Affected code:

diffusers/src/diffusers/pipelines/glm_image/pipeline_glm_image.py

Lines 584 to 600 in 0f1abc4

	if prompt_embeds is None:
	prompt_embeds = self._get_glyph_embeds(prompt, max_sequence_length, device, dtype)
	# Repeat embeddings for num_images_per_prompt
	if num_images_per_prompt > 1:
	prompt_embeds = prompt_embeds.repeat_interleave(num_images_per_prompt, dim=0)
	# For GLM-Image, negative_prompt must be "" instead of None
	if do_classifier_free_guidance and negative_prompt_embeds is None:
	negative_prompt = ""
	negative_prompt = batch_size * [negative_prompt] if isinstance(negative_prompt, str) else negative_prompt
	negative_prompt_embeds = self._get_glyph_embeds(negative_prompt, max_sequence_length, device, dtype)
	if num_images_per_prompt > 1:
	negative_prompt_embeds = negative_prompt_embeds.repeat_interleave(num_images_per_prompt, dim=0)
	return prompt_embeds, negative_prompt_embeds

diffusers/src/diffusers/pipelines/glm_image/pipeline_glm_image.py

Lines 833 to 836 in 0f1abc4

	else:
	# User provided prior_token_ids directly (from generate_prior_tokens)
	prior_token_image_ids_per_sample = prior_token_image_ids
	source_image_grid_thw_per_sample = source_image_grid_thw

Problem:
When users pass prompt_embeds, negative_prompt_embeds, or prior token tensors directly, the pipeline returns/uses them as-is instead of normalizing them to the execution device and dtype.

Impact:
Precomputed embeddings can fail at the transformer with dtype or device mismatches. Prior token tensors generated on CPU can also fail when the pipeline is on an accelerator.

Reproduction:

import torch
from diffusers import GlmImagePipeline

pipe = object.__new__(GlmImagePipeline)
prompt_embeds = torch.randn(1, 2, 4, dtype=torch.float64)

out, _ = pipe.encode_prompt(
 prompt=None,
 do_classifier_free_guidance=False,
 prompt_embeds=prompt_embeds,
 device=torch.device("cpu"),
 dtype=torch.float32,
)
print(out.dtype, out is prompt_embeds) # torch.float64 True

Relevant precedent:

diffusers/src/diffusers/pipelines/cogview4/pipeline_cogview4.py

Lines 216 to 218 in 0f1abc4

	prompt_embeds = self.text_encoder(text_input_ids.to(device), output_hidden_states=True).hidden_states[-2]
	prompt_embeds = prompt_embeds.to(dtype=dtype, device=device)

Suggested fix:

if prompt_embeds is None:
 prompt_embeds = self._get_glyph_embeds(prompt, max_sequence_length, device, dtype)
else:
 prompt_embeds = prompt_embeds.to(device=device, dtype=dtype)

if negative_prompt_embeds is not None:
 negative_prompt_embeds = negative_prompt_embeds.to(device=device, dtype=dtype)

if prior_token_ids is not None:
 prior_token_ids = prior_token_ids.to(device=device)
if prior_token_image_ids is not None:
 prior_token_image_ids = [x.to(device=device) for x in prior_token_image_ids]
if source_image_grid_thw is not None:
 source_image_grid_thw = [x.to(device=device) for x in source_image_grid_thw]

Issue 4: Transformers version gates are inconsistent with required GLM classes

Affected code:

diffusers/src/diffusers/pipelines/glm_image/pipeline_glm_image.py

Lines 36 to 40 in 0f1abc4

	# Because it's not released in stable as of 13/01/2026. So this is just a proxy.
	GlmImageProcessor = ProcessorMixin
	GlmImageForConditionalGeneration = PreTrainedModel
	if is_transformers_version(">=", "5.0.0.dev0"):
	from transformers import GlmImageForConditionalGeneration, GlmImageProcessor

diffusers/src/diffusers/pipelines/glm_image/__init__.py

Lines 20 to 27 in 0f1abc4

	if is_transformers_available() and is_transformers_version(">=", "4.57.4"):
	try:
	from transformers import GlmImageForConditionalGeneration, GlmImageProcessor
	_additional_imports["GlmImageForConditionalGeneration"] = GlmImageForConditionalGeneration
	_additional_imports["GlmImageProcessor"] = GlmImageProcessor
	except ImportError:
	pass

diffusers/tests/pipelines/glm_image/test_glm_image.py

Lines 29 to 37 in 0f1abc4

	if is_transformers_version(">=", "5.0.0.dev0"):
	from transformers import GlmImageConfig, GlmImageForConditionalGeneration, GlmImageProcessor
	enable_full_determinism()
	@require_transformers_version_greater("4.57.4")
	@require_torch_accelerator

Problem:
The pipeline file only imports real GlmImageProcessor / GlmImageForConditionalGeneration for transformers >= 5.0.0.dev0, but the package init tries >= 4.57.4 and the fast tests require only > 4.57.4.

Impact:
With transformers==4.57.6, the pipeline is importable but uses ProcessorMixin / PreTrainedModel placeholders, and the test decorator would allow tests whose GLM classes are not imported.

Reproduction:

import transformers
from diffusers.utils import is_transformers_version
from diffusers.pipelines.glm_image.pipeline_glm_image import GlmImageProcessor, GlmImageForConditionalGeneration
from transformers import ProcessorMixin, PreTrainedModel

print(transformers.__version__)
print(is_transformers_version(">", "4.57.4")) # True in this env
print(is_transformers_version(">=", "5.0.0.dev0")) # False
print(hasattr(transformers, "GlmImageProcessor")) # False
print(GlmImageProcessor is ProcessorMixin, GlmImageForConditionalGeneration is PreTrainedModel)

Relevant precedent:
Related prior version-gating PR: #12974

Suggested fix:

GLM_IMAGE_TRANSFORMERS_MIN_VERSION = "5.0.0.dev0" # or the first released transformers version with these classes

if is_transformers_available() and is_transformers_version(">=", GLM_IMAGE_TRANSFORMERS_MIN_VERSION):
 from transformers import GlmImageForConditionalGeneration, GlmImageProcessor
else:
 raise OptionalDependencyNotAvailable()

Use the same predicate in pipeline lazy imports and tests.

Issue 5: Slow tests are missing, and current fast coverage is not portable

Affected code:

diffusers/tests/pipelines/glm_image/test_glm_image.py

Lines 36 to 38 in 0f1abc4

	@require_transformers_version_greater("4.57.4")
	@require_torch_accelerator
	class GlmImagePipelineFastTests(PipelineTesterMixin, unittest.TestCase):

diffusers/tests/pipelines/glm_image/test_glm_image.py

Line 90 in 0f1abc4

processor = GlmImageProcessor.from_pretrained("zai-org/GLM-Image", subfolder="processor")

diffusers/docs/source/en/api/models/glm_image_transformer2d.md

Line 14 in 0f1abc4

A Diffusion Transformer model for 2D data from [GlmImageTransformer2DModel] (TODO).

diffusers/src/diffusers/pipelines/glm_image/pipeline_output.py

Lines 10 to 12 in 0f1abc4

	class GlmImagePipelineOutput(BaseOutput):
	"""
	Output class for CogView3 pipelines.

Problem:
There are no GLM Image slow tests. The pipeline “fast” tests are decorated with @require_torch_accelerator even though they run on CPU, and they load the processor from zai-org/GLM-Image instead of an internal tiny fixture. The model docs still contain a TODO, and the pipeline output docstring says CogView3.

Impact:
CPU CI can skip the pipeline fast suite, slow end-to-end loading of zai-org/GLM-Image is untested, and docs have stale placeholders.

Reproduction:

from pathlib import Path

test_text = Path("tests/pipelines/glm_image/test_glm_image.py").read_text(encoding="utf-8")
model_doc = Path("docs/source/en/api/models/glm_image_transformer2d.md").read_text(encoding="utf-8")

print("@slow present:", "@slow" in test_text)
print("fast class requires accelerator:", "@require_torch_accelerator" in test_text)
print("fast test downloads full GLM processor:", "zai-org/GLM-Image" in test_text)
print("model docs still contain TODO:", "TODO" in model_doc)

Relevant precedent:

diffusers/tests/pipelines/qwenimage/test_qwenimage.py

Line 117 in 0f1abc4

tokenizer = Qwen2Tokenizer.from_pretrained("hf-internal-testing/tiny-random-Qwen2VLForConditionalGeneration")

Suggested fix:
Use an hf-internal-testing tiny GLM processor fixture for fast tests, remove the accelerator requirement from CPU-only fast tests, add at least one @slow smoke test for GlmImagePipeline.from_pretrained("zai-org/GLM-Image", ...), and replace the stale TODO/CogView3 doc text.

Test status: a tiny CPU GlmImageTransformer2DModel forward pass succeeded. Targeted pytest collection for GLM model/pipeline tests failed in this local .venv because the installed Torch build lacks torch._C._distributed_c10d, so I could not use pytest results as signal for this audit.