ltx model/pipeline review

[hlky]

ltx model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules. Reviewed the listed LTX pipelines, modular pipeline blocks, LTX VAE, LTX transformer, imports/lazy-loading, tests, docs references, and duplicate GitHub Issues/PRs.

Duplicate search: checked broad ltx plus the specific class names and failure modes. Related but not exact duplicates: #10565 is a broad LTX I2V quality issue, #11104 is an older closed offload issue for LTXPipeline, #13121 fixed a related LTX2 num_videos_per_prompt bug, #13254 only refactors transformer tests, and #13378 introduced the modular LTX pipeline.

Issue 1: T5 prompt encoding builds a padding mask but does not pass it to T5

Affected code:

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx.py

Line 269 in 0f1abc4

prompt_embeds = self.text_encoder(text_input_ids.to(device))[0]

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_image2video.py

Line 292 in 0f1abc4

prompt_embeds = self.text_encoder(text_input_ids.to(device))[0]

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_i2v_long_multi_prompt.py

Line 538 in 0f1abc4

prompt_embeds = self.text_encoder(text_input_ids.to(device))[0]

diffusers/src/diffusers/modular_pipelines/ltx/encoders.py

Line 52 in 0f1abc4

prompt_embeds = components.text_encoder(text_input_ids.to(device))[0]

Problem:
These paths create prompt_attention_mask but call T5 as text_encoder(input_ids) instead of passing attention_mask=prompt_attention_mask. Padding tokens therefore participate in prompt encoding. LTXConditionPipeline already uses the mask, so prompt encoding is inconsistent within the LTX family.

Impact:
Short prompts padded to max_sequence_length get different embeddings from the masked T5 encoding. This can affect text adherence and may be related to broad I2V quality reports such as #10565, though I did not find an exact duplicate for this failure mode.

Reproduction:

import torch
from transformers import AutoTokenizer, T5EncoderModel
from diffusers import LTXPipeline

model_id = "hf-internal-testing/tiny-random-t5"
tokenizer = AutoTokenizer.from_pretrained(model_id)
text_encoder = T5EncoderModel.from_pretrained(model_id).eval()
pipe = LTXPipeline(None, None, text_encoder, tokenizer, None)

with torch.no_grad():
 got, mask = pipe._get_t5_prompt_embeds(["short"], max_sequence_length=16, device=torch.device("cpu"))
 inputs = tokenizer(["short"], padding="max_length", max_length=16, truncation=True, return_tensors="pt")
 expected = text_encoder(inputs.input_ids, attention_mask=inputs.attention_mask.bool())[0]

print(mask.tolist())
print((got - expected).abs().max().item())

Relevant precedent:

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py

Line 355 in 0f1abc4

prompt_embeds = self.text_encoder(text_input_ids.to(device), attention_mask=prompt_attention_mask)[0]

diffusers/src/diffusers/pipelines/mochi/pipeline_mochi.py

Line 240 in 0f1abc4

prompt_embeds = self.text_encoder(text_input_ids.to(device), attention_mask=prompt_attention_mask)[0]

Suggested fix:

prompt_embeds = self.text_encoder(text_input_ids.to(device), attention_mask=prompt_attention_mask)[0]
# modular:
prompt_embeds = components.text_encoder(text_input_ids.to(device), attention_mask=prompt_attention_mask)[0]

Issue 2: use_framewise_encoding never enables temporal tiled encoding

Affected code:

diffusers/src/diffusers/models/autoencoders/autoencoder_kl_ltx.py

Lines 1220 to 1224 in 0f1abc4

	def _encode(self, x: torch.Tensor) -> torch.Tensor:
	batch_size, num_channels, num_frames, height, width = x.shape
	if self.use_framewise_decoding and num_frames > self.tile_sample_min_num_frames:
	return self._temporal_tiled_encode(x)

Problem:
AutoencoderKLLTXVideo._encode checks self.use_framewise_decoding instead of self.use_framewise_encoding. Setting use_framewise_encoding = True does nothing, while setting use_framewise_decoding = True also changes encode behavior.

Impact:
Users cannot enable framewise encoding to reduce VAE encode memory for long videos, and enabling framewise decoding unexpectedly changes encode behavior too.

Reproduction:

import torch
from unittest import mock
from diffusers import AutoencoderKLLTXVideo

vae = AutoencoderKLLTXVideo(
 in_channels=3, out_channels=3, latent_channels=4,
 block_out_channels=(8, 8, 8, 8), decoder_block_out_channels=(8, 8, 8, 8),
 layers_per_block=(1, 1, 1, 1, 1), decoder_layers_per_block=(1, 1, 1, 1, 1),
 spatio_temporal_scaling=(True, True, False, False),
 decoder_spatio_temporal_scaling=(True, True, False, False),
 decoder_inject_noise=(False, False, False, False, False),
 upsample_residual=(False, False, False, False), upsample_factor=(1, 1, 1, 1),
 patch_size=1, patch_size_t=1, encoder_causal=True, decoder_causal=False,
)
vae.tile_sample_min_num_frames = 1
x = torch.randn(1, 3, 3, 32, 32)

vae.use_framewise_encoding = True
vae.use_framewise_decoding = False
with mock.patch.object(vae, "_temporal_tiled_encode", side_effect=RuntimeError("called")):
 vae.encode(x)
 print("framewise encoding did not call temporal tiled encode")

Relevant precedent:

diffusers/src/diffusers/models/autoencoders/autoencoder_kl_hunyuan_video.py

Lines 767 to 811 in 0f1abc4

	if self.use_framewise_encoding and num_frames > self.tile_sample_min_num_frames:
	return self._temporal_tiled_encode(x)
	if self.use_tiling and (width > self.tile_sample_min_width or height > self.tile_sample_min_height):
	return self.tiled_encode(x)
	x = self.encoder(x)
	enc = self.quant_conv(x)
	return enc
	@apply_forward_hook
	def encode(
	self, x: torch.Tensor, return_dict: bool = True
	) -> AutoencoderKLOutput | tuple[DiagonalGaussianDistribution]:
	r"""
	Encode a batch of images into latents.
	Args:
	x (`torch.Tensor`): Input batch of images.
	return_dict (`bool`, *optional*, defaults to `True`):
	Whether to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple.
	Returns:
	The latent representations of the encoded videos. If `return_dict` is True, a
	[`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned.
	"""
	if self.use_slicing and x.shape[0] > 1:
	encoded_slices = [self._encode(x_slice) for x_slice in x.split(1)]
	h = torch.cat(encoded_slices)
	else:
	h = self._encode(x)
	posterior = DiagonalGaussianDistribution(h)
	if not return_dict:
	return (posterior,)
	return AutoencoderKLOutput(latent_dist=posterior)
	def _decode(self, z: torch.Tensor, return_dict: bool = True) -> DecoderOutput | torch.Tensor:
	batch_size, num_channels, num_frames, height, width = z.shape
	tile_latent_min_height = self.tile_sample_min_height // self.spatial_compression_ratio
	tile_latent_min_width = self.tile_sample_min_width // self.spatial_compression_ratio
	tile_latent_min_num_frames = self.tile_sample_min_num_frames // self.temporal_compression_ratio
	if self.use_framewise_decoding and num_frames > tile_latent_min_num_frames:

Suggested fix:

if self.use_framewise_encoding and num_frames > self.tile_sample_min_num_frames:
 return self._temporal_tiled_encode(x)

Issue 3: LTXImageToVideoPipeline crashes for generator lists with multiple videos per prompt

Affected code:

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_image2video.py

Lines 534 to 548 in 0f1abc4

	if isinstance(generator, list):
	if len(generator) != batch_size:
	raise ValueError(
	f"You have passed a list of generators of length {len(generator)}, but requested an effective batch"
	f" size of {batch_size}. Make sure the batch size matches the length of the generators."
	)
	init_latents = [
	retrieve_latents(self.vae.encode(image[i].unsqueeze(0).unsqueeze(2)), generator[i])
	for i in range(batch_size)
	]
	else:
	init_latents = [
	retrieve_latents(self.vae.encode(img.unsqueeze(0).unsqueeze(2)), generator) for img in image
	]

Problem:
When num_videos_per_prompt > 1, the effective batch size is larger than the image batch. The non-list generator path broadcasts one encoded image, but the list-generator path indexes image[i] for every effective batch item and crashes.

Impact:
Users cannot generate multiple deterministic I2V samples per prompt with a list of generators.

Reproduction:

import torch
from types import SimpleNamespace
from diffusers import LTXImageToVideoPipeline

pipe = object.__new__(LTXImageToVideoPipeline)
pipe.vae_spatial_compression_ratio = 1
pipe.vae_temporal_compression_ratio = 1
pipe.transformer_spatial_patch_size = 1
pipe.transformer_temporal_patch_size = 1
pipe.vae = SimpleNamespace(
 latents_mean=torch.zeros(8),
 latents_std=torch.ones(8),
 encode=lambda x: SimpleNamespace(latents=torch.zeros(1, 8, 1, 4, 4)),
)

pipe.prepare_latents(
 image=torch.zeros(1, 3, 4, 4),
 batch_size=2,
 num_channels_latents=8,
 height=4,
 width=4,
 num_frames=2,
 dtype=torch.float32,
 device=torch.device("cpu"),
 generator=[torch.Generator().manual_seed(0), torch.Generator().manual_seed(1)],
)

Relevant precedent:

diffusers/src/diffusers/modular_pipelines/ltx/before_denoise.py

Lines 366 to 369 in 0f1abc4

	init_latents = block_state.image_latents.to(device=device, dtype=torch.float32)
	if init_latents.shape[0] < batch_size:
	init_latents = init_latents.repeat_interleave(batch_size // init_latents.shape[0], dim=0)
	init_latents = init_latents.repeat(1, 1, num_frames, 1, 1)

Related LTX2 coverage precedent: #13121

Suggested fix:
Repeat encoded image latents to the effective batch before sampling per-generator, or index source images modulo the image batch:

source_index = i % image.shape[0]
retrieve_latents(self.vae.encode(image[source_index].unsqueeze(0).unsqueeze(2)), generator[i])

Issue 4: Legacy multi-condition arguments silently drop conditions

Affected code:

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py

Lines 1029 to 1044 in 0f1abc4

	elif image is not None or video is not None:
	if not isinstance(image, list):
	image = [image]
	num_conditions = 1
	elif isinstance(image, list):
	num_conditions = len(image)
	if not isinstance(video, list):
	video = [video]
	num_conditions = 1
	elif isinstance(video, list):
	num_conditions = len(video)
	if not isinstance(frame_index, list):
	frame_index = [frame_index] * num_conditions
	if not isinstance(strength, list):
	strength = [strength] * num_conditions

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py

Lines 1074 to 1076 in 0f1abc4

	for condition_image, condition_video, condition_frame_index, condition_strength in zip(
	image, video, frame_index, strength
	):

Problem:
If image is a list and video is None, the code later rewrites video to [None] and resets num_conditions = 1. The subsequent zip(image, video, frame_index, strength) processes only the first image. The same issue happens for video lists when image is None.

Impact:
Users passing multiple image/frame_index/strength values through the legacy arguments get only the first condition applied, with no error.

Reproduction:

# Mirrors LTXConditionPipeline.__call__ normalization at lines 1029-1044.
image = ["image0", "image1"]
video = None
frame_index = [0, 8]
strength = [1.0, 1.0]

if not isinstance(image, list):
 image = [image]
 num_conditions = 1
elif isinstance(image, list):
 num_conditions = len(image)
if not isinstance(video, list):
 video = [video]
 num_conditions = 1

print(list(zip(image, video, frame_index, strength)))
# Only [('image0', None, 0, 1.0)] is processed.

Relevant precedent:
The conditions=[LTXVideoCondition(...), ...] path preserves list length:

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py

Lines 1021 to 1028 in 0f1abc4

	if conditions is not None:
	if not isinstance(conditions, list):
	conditions = [conditions]
	strength = [condition.strength for condition in conditions]
	frame_index = [condition.frame_index for condition in conditions]
	image = [condition.image for condition in conditions]
	video = [condition.video for condition in conditions]

Suggested fix:
Normalize the absent modality to the detected condition count instead of resetting the count:

if image is None:
 image = [None] * len(video)
elif not isinstance(image, list):
 image = [image]

if video is None:
 video = [None] * len(image)
elif not isinstance(video, list):
 video = [video]

num_conditions = max(len(image), len(video))

Issue 5: Latent upsample pipeline has no model CPU offload sequence

Affected code:

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_latent_upsample.py

Line 45 in 0f1abc4

model_cpu_offload_seq = ""

Problem:
LTXLatentUpsamplePipeline.model_cpu_offload_seq is an empty string even though the pipeline runs both vae and latent_upsampler.

Impact:
enable_model_cpu_offload() cannot use an explicit component order for the upsample pipeline, which is exactly the kind of memory-sensitive path users are likely to offload. The old closed issue #11104 is about a different LTX offload failure, not this specific pipeline sequence gap.

Reproduction:

from diffusers import LTXLatentUpsamplePipeline, LTX2LatentUpsamplePipeline

print(repr(LTXLatentUpsamplePipeline.model_cpu_offload_seq))
print(repr(LTX2LatentUpsamplePipeline.model_cpu_offload_seq))

Relevant precedent:

diffusers/src/diffusers/pipelines/ltx2/pipeline_ltx2_latent_upsample.py

Line 105 in 0f1abc4

model_cpu_offload_seq = "vae->latent_upsampler"

Suggested fix:

model_cpu_offload_seq = "vae->latent_upsampler"

Issue 6: Slow tests are missing, and LTXI2VLongMultiPromptPipeline has no fast test

Affected code:

diffusers/src/diffusers/pipelines/ltx/__init__.py

Lines 28 to 43 in 0f1abc4

	_import_structure["pipeline_ltx_i2v_long_multi_prompt"] = ["LTXI2VLongMultiPromptPipeline"]
	_import_structure["pipeline_ltx_image2video"] = ["LTXImageToVideoPipeline"]
	_import_structure["pipeline_ltx_latent_upsample"] = ["LTXLatentUpsamplePipeline"]
	if TYPE_CHECKING or DIFFUSERS_SLOW_IMPORT:
	try:
	if not (is_transformers_available() and is_torch_available()):
	raise OptionalDependencyNotAvailable()
	except OptionalDependencyNotAvailable:
	from ...utils.dummy_torch_and_transformers_objects import *
	else:
	from .modeling_latent_upsampler import LTXLatentUpsamplerModel
	from .pipeline_ltx import LTXPipeline
	from .pipeline_ltx_condition import LTXConditionPipeline
	from .pipeline_ltx_i2v_long_multi_prompt import LTXI2VLongMultiPromptPipeline

Problem:
The LTX family has fast tests for the main T2V/I2V/condition/latent-upsample pipelines, models, LoRA, and modular assembly, but I found no @slow LTX tests. I also found no test file for LTXI2VLongMultiPromptPipeline.

Impact:
Real checkpoint loading, docs examples, offload behavior, and the long multi-prompt windowing path are not covered. Open PR #13254 touches only transformer tests and does not cover this gap.

Reproduction:

from pathlib import Path

ltx_tests = list(Path("tests").rglob("*ltx*.py"))
print("slow hits:", [str(p) for p in ltx_tests if "@slow" in p.read_text(encoding="utf-8")])
print("long pipeline test exists:", Path("tests/pipelines/ltx/test_ltx_i2v_long_multi_prompt.py").exists())

Relevant precedent:
Pipeline families usually carry at least one slow test for real checkpoint loading/inference when a public pipeline is documented.

Suggested fix:
Add a slow test for a published LTX checkpoint, and add a focused fast test for LTXI2VLongMultiPromptPipeline using tiny local components and output_type="latent".

Issue 7: Modular LTX fast test uses a contributor-owned model repo

Affected code:

diffusers/tests/modular_pipelines/ltx/test_modular_pipeline_ltx.py

Lines 45 to 49 in 0f1abc4

	class TestLTXModularPipelineFast(ModularPipelineTesterMixin):
	pipeline_class = LTXModularPipeline
	pipeline_blocks_class = LTXAutoBlocks
	pretrained_model_name_or_path = "akshan-main/tiny-ltx-modular-pipe"

Problem:
The modular review rules require tiny test models under hf-internal-testing/, but the LTX modular test uses akshan-main/tiny-ltx-modular-pipe.

Impact:
CI and contributors depend on a personal namespace for a required fast test fixture. PR #13378 introduced the modular pipeline, but I found no follow-up issue/PR moving this fixture.

Reproduction:

from pathlib import Path

text = Path("tests/modular_pipelines/ltx/test_modular_pipeline_ltx.py").read_text()
print("akshan-main/tiny-ltx-modular-pipe" in text)

Relevant precedent:
The local modular rule modular.md explicitly says tiny test models must live under hf-internal-testing/.

Suggested fix:
Move/copy the tiny modular fixture to hf-internal-testing/tiny-ltx-modular-pipe and update the test constant.

Issue 8: Modular generated docstrings still contain TODO placeholders

Affected code:

diffusers/src/diffusers/modular_pipelines/ltx/modular_blocks_ltx.py

Lines 33 to 66 in 0f1abc4

	class LTXCoreDenoiseStep(SequentialPipelineBlocks):
	"""
	Denoise block that takes encoded conditions and runs the denoising process.
	Components:
	scheduler (`FlowMatchEulerDiscreteScheduler`) pachifier (`LTXVideoPachifier`) guider
	(`ClassifierFreeGuidance`) transformer (`LTXVideoTransformer3DModel`)
	Inputs:
	num_videos_per_prompt (`int`, *optional*, defaults to 1):
	The number of images to generate per prompt.
	prompt_embeds (`Tensor`):
	text embeddings used to guide the image generation. Can be generated from text_encoder step.
	prompt_attention_mask (`Tensor`):
	mask for the text embeddings. Can be generated from text_encoder step.
	negative_prompt_embeds (`Tensor`, *optional*):
	negative text embeddings used to guide the image generation. Can be generated from text_encoder step.
	negative_prompt_attention_mask (`Tensor`, *optional*):
	mask for the negative text embeddings. Can be generated from text_encoder step.
	num_inference_steps (`int`, *optional*, defaults to 50):
	The number of denoising steps.
	timesteps (`Tensor`, *optional*):
	Timesteps for the denoising process.
	sigmas (`list`, *optional*):
	Custom sigmas for the denoising process.
	height (`int`, *optional*, defaults to 512):
	The height in pixels of the generated image.
	width (`int`, *optional*, defaults to 704):
	The width in pixels of the generated image.
	num_frames (`int`, *optional*, defaults to 161):
	TODO: Add description.
	frame_rate (`int`, *optional*, defaults to 25):
	TODO: Add description.
	latents (`Tensor`, *optional*):

Problem:
modular_blocks_ltx.py contains many generated TODO: Add description. entries. The modular review rules require running utils/modular_auto_docstring.py --fix_and_overwrite and verifying no TODO placeholders remain.

Impact:
The modular pipeline public docs are incomplete and the block IO contract is less usable for users composing blocks directly.

Reproduction:

from pathlib import Path

for i, line in enumerate(Path("src/diffusers/modular_pipelines/ltx/modular_blocks_ltx.py").read_text().splitlines(), 1):
 if "TODO: Add description" in line:
 print(i, line.strip())

Relevant precedent:
The local modular rules require accurate InputParam/OutputParam descriptions and generated docstrings without TODO placeholders.

Suggested fix:
Add explicit descriptions for the unresolved block inputs/outputs, then run:

python utils/modular_auto_docstring.py --fix_and_overwrite