helios model/pipeline review

[hlky]

helios model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Duplicate search status: searched GitHub Issues/PRs for helios, HeliosPipeline latents, Helios num_videos_per_prompt, helios ftfy, helios timestep, prepare_video_latents first_frame_latent, and HeliosPyramidPipeline test. No specific duplicate issue/PR was found. PR #13218 only skips a Helios float16 save/load test and is not a duplicate.

Issue 1: num_videos_per_prompt breaks Helios generation

Affected code:

diffusers/src/diffusers/pipelines/helios/pipeline_helios.py

Lines 599 to 613 in 0f1abc4

	prompt_embeds, negative_prompt_embeds = self.encode_prompt(
	prompt=prompt,
	negative_prompt=negative_prompt,
	do_classifier_free_guidance=self.do_classifier_free_guidance,
	num_videos_per_prompt=num_videos_per_prompt,
	prompt_embeds=prompt_embeds,
	negative_prompt_embeds=negative_prompt_embeds,
	max_sequence_length=max_sequence_length,
	device=device,
	)
	transformer_dtype = self.transformer.dtype
	prompt_embeds = prompt_embeds.to(transformer_dtype)
	if negative_prompt_embeds is not None:
	negative_prompt_embeds = negative_prompt_embeds.to(transformer_dtype)

diffusers/src/diffusers/pipelines/helios/pipeline_helios.py

Lines 703 to 711 in 0f1abc4

	history_latents = torch.zeros(
	batch_size,
	num_channels_latents,
	num_history_latent_frames,
	height // self.vae_scale_factor_spatial,
	width // self.vae_scale_factor_spatial,
	device=device,
	dtype=torch.float32,
	)

diffusers/src/diffusers/pipelines/helios/pipeline_helios.py

Lines 792 to 802 in 0f1abc4

	latents = self.prepare_latents(
	batch_size,
	num_channels_latents,
	height,
	width,
	window_num_frames,
	dtype=torch.float32,
	device=device,
	generator=generator,
	latents=None,
	)

diffusers/src/diffusers/modular_pipelines/helios/before_denoise.py

Lines 100 to 116 in 0f1abc4

	block_state.batch_size = block_state.prompt_embeds.shape[0]
	block_state.dtype = block_state.prompt_embeds.dtype
	_, seq_len, _ = block_state.prompt_embeds.shape
	block_state.prompt_embeds = block_state.prompt_embeds.repeat(1, block_state.num_videos_per_prompt, 1)
	block_state.prompt_embeds = block_state.prompt_embeds.view(
	block_state.batch_size * block_state.num_videos_per_prompt, seq_len, -1
	)
	if block_state.negative_prompt_embeds is not None:
	_, seq_len, _ = block_state.negative_prompt_embeds.shape
	block_state.negative_prompt_embeds = block_state.negative_prompt_embeds.repeat(
	1, block_state.num_videos_per_prompt, 1
	)
	block_state.negative_prompt_embeds = block_state.negative_prompt_embeds.view(
	block_state.batch_size * block_state.num_videos_per_prompt, seq_len, -1
	)

diffusers/src/diffusers/modular_pipelines/helios/before_denoise.py

Lines 666 to 688 in 0f1abc4

	block_state.latent_shape = (
	batch_size,
	num_channels_latents,
	block_state.num_latent_frames_per_chunk,
	h_latent,
	w_latent,
	)
	# Set outputs
	block_state.history_sizes = history_sizes
	block_state.indices_hidden_states = indices_hidden_states.unsqueeze(0)
	block_state.indices_latents_history_short = indices_latents_history_short.unsqueeze(0)
	block_state.indices_latents_history_mid = indices_latents_history_mid.unsqueeze(0)
	block_state.indices_latents_history_long = indices_latents_history_long.unsqueeze(0)
	block_state.history_latents = torch.zeros(
	batch_size,
	num_channels_latents,
	sum(history_sizes),
	h_latent,
	w_latent,
	device=device,
	dtype=torch.float32,
	)

Problem:
encode_prompt expands prompt embeddings to batch_size * num_videos_per_prompt, but latent/history tensors are still allocated with the original prompt batch size. The first transformer call then receives hidden states with batch size 1 and prompt embeddings with batch size 2.

Impact:
Users cannot request multiple videos per prompt. The modular pipeline has the same contract problem: HeliosTextInputStep expands embeddings but leaves batch_size as the pre-expansion value used by history and latent prep.

Reproduction:

import torch
from transformers import AutoConfig, AutoTokenizer, T5EncoderModel
from diffusers import AutoencoderKLWan, HeliosPipeline, HeliosScheduler, HeliosTransformer3DModel

vae = AutoencoderKLWan(base_dim=3, z_dim=16, dim_mult=[1, 1, 1, 1], num_res_blocks=1, temperal_downsample=[False, True, True])
scheduler = HeliosScheduler(stage_range=[0, 1], stages=1, use_dynamic_shifting=True)
config = AutoConfig.from_pretrained("hf-internal-testing/tiny-random-t5")
pipe = HeliosPipeline(
 transformer=HeliosTransformer3DModel(
 patch_size=(1, 2, 2), num_attention_heads=2, attention_head_dim=12, in_channels=16, out_channels=16,
 text_dim=32, freq_dim=256, ffn_dim=32, num_layers=2, rope_dim=(4, 4, 4),
 ),
 vae=vae,
 scheduler=scheduler,
 text_encoder=T5EncoderModel(config),
 tokenizer=AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-t5"),
).to("cpu")
pipe.set_progress_bar_config(disable=True)

pipe(
 prompt="dance monkey",
 negative_prompt="negative",
 generator=torch.Generator("cpu").manual_seed(0),
 num_inference_steps=1,
 guidance_scale=1.0,
 height=16,
 width=16,
 num_frames=9,
 max_sequence_length=16,
 output_type="latent",
 num_videos_per_prompt=2,
)

Relevant precedent:
WanPipeline and other video pipelines propagate the effective batch size into latent prep after prompt/image expansion.

Suggested fix:

effective_batch_size = batch_size * num_videos_per_prompt

history_latents = torch.zeros(
 effective_batch_size,
 num_channels_latents,
 num_history_latent_frames,
 height // self.vae_scale_factor_spatial,
 width // self.vae_scale_factor_spatial,
 device=device,
 dtype=torch.float32,
)

latents = self.prepare_latents(
 effective_batch_size,
 num_channels_latents,
 height,
 width,
 window_num_frames,
 dtype=torch.float32,
 device=device,
 generator=generator,
 latents=chunk_latents,
)

For modular Helios, set block_state.batch_size to the expanded effective batch size or add a separate effective_batch_size output and use it for history/latent allocation.

Issue 2: Public latents input is ignored

Affected code:

diffusers/src/diffusers/pipelines/helios/pipeline_helios.py

Lines 792 to 802 in 0f1abc4

	latents = self.prepare_latents(
	batch_size,
	num_channels_latents,
	height,
	width,
	window_num_frames,
	dtype=torch.float32,
	device=device,
	generator=generator,
	latents=None,
	)

diffusers/src/diffusers/pipelines/helios/pipeline_helios_pyramid.py

Lines 844 to 854 in 0f1abc4

	latents = self.prepare_latents(
	batch_size,
	num_channels_latents,
	height,
	width,
	window_num_frames,
	dtype=torch.float32,
	device=device,
	generator=generator,
	latents=None,
	)

diffusers/src/diffusers/modular_pipelines/helios/denoise.py

Lines 243 to 265 in 0f1abc4

	class HeliosChunkNoiseGenStep(ModularPipelineBlocks):
	"""Generates noise latents for a chunk using randn_tensor."""
	model_name = "helios"
	@property
	def description(self) -> str:
	return "Generates random noise latents at full resolution for a single chunk."
	@property
	def inputs(self) -> list[InputParam]:
	return [
	InputParam("latent_shape", required=True, type_hint=tuple),
	InputParam.template("generator"),
	]
	@torch.no_grad()
	def __call__(self, components: HeliosModularPipeline, block_state: BlockState, k: int):
	device = components._execution_device
	block_state.latents = randn_tensor(
	block_state.latent_shape, generator=block_state.generator, device=device, dtype=torch.float32
	)
	return components, block_state

diffusers/src/diffusers/modular_pipelines/helios/denoise.py

Lines 268 to 313 in 0f1abc4

	class HeliosPyramidChunkNoiseGenStep(ModularPipelineBlocks):
	"""Generates noise latents and downsamples to smallest pyramid level."""
	model_name = "helios-pyramid"
	@property
	def description(self) -> str:
	return (
	"Generates random noise at full resolution, then downsamples to the smallest "
	"pyramid level via bilinear interpolation."
	)
	@property
	def inputs(self) -> list[InputParam]:
	return [
	InputParam("latent_shape", required=True, type_hint=tuple),
	InputParam(
	"pyramid_num_inference_steps_list",
	default=[10, 10, 10],
	type_hint=list,
	description="Number of denoising steps per pyramid stage.",
	),
	InputParam.template("generator"),
	]
	@torch.no_grad()
	def __call__(self, components: HeliosModularPipeline, block_state: BlockState, k: int):
	device = components._execution_device
	batch_size, num_channels_latents, num_latent_frames, h_latent, w_latent = block_state.latent_shape
	latents = randn_tensor(
	block_state.latent_shape, generator=block_state.generator, device=device, dtype=torch.float32
	)
	# Downsample to smallest pyramid level
	h, w = h_latent, w_latent
	latents = latents.permute(0, 2, 1, 3, 4).reshape(batch_size * num_latent_frames, num_channels_latents, h, w)
	for _ in range(len(block_state.pyramid_num_inference_steps_list) - 1):
	h //= 2
	w //= 2
	latents = F.interpolate(latents, size=(h, w), mode="bilinear") * 2
	block_state.latents = latents.reshape(batch_size, num_latent_frames, num_channels_latents, h, w).permute(
	0, 2, 1, 3, 4
	)
	return components, block_state

Problem:
Both standard pipelines accept latents, document it as pre-generated noisy latents, and list it in required optional params, but the denoise loop always calls prepare_latents(..., latents=None). The modular noise blocks also always sample new noise.

Impact:
Users cannot reproduce or edit a generation by supplying their own initial noise. Tests can miss this because the signature exists and generation still succeeds.

Reproduction:

import torch
from transformers import AutoConfig, AutoTokenizer, T5EncoderModel
from diffusers import AutoencoderKLWan, HeliosPipeline, HeliosScheduler, HeliosTransformer3DModel

def make_pipe():
 vae = AutoencoderKLWan(base_dim=3, z_dim=16, dim_mult=[1, 1, 1, 1], num_res_blocks=1, temperal_downsample=[False, True, True])
 config = AutoConfig.from_pretrained("hf-internal-testing/tiny-random-t5")
 return HeliosPipeline(
 transformer=HeliosTransformer3DModel(
 patch_size=(1, 2, 2), num_attention_heads=2, attention_head_dim=12, in_channels=16, out_channels=16,
 text_dim=32, freq_dim=256, ffn_dim=32, num_layers=2, rope_dim=(4, 4, 4),
 ),
 vae=vae,
 scheduler=HeliosScheduler(stage_range=[0, 1], stages=1, use_dynamic_shifting=True),
 text_encoder=T5EncoderModel(config),
 tokenizer=AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-t5"),
 ).to("cpu")

kwargs = dict(prompt="dance monkey", negative_prompt="negative", num_inference_steps=1, guidance_scale=1.0, height=16, width=16, num_frames=9, max_sequence_length=16, output_type="latent")
shape = (1, 16, 9, 2, 2)

pipe = make_pipe(); pipe.set_progress_bar_config(disable=True)
a = pipe(**kwargs, generator=torch.Generator("cpu").manual_seed(123), latents=torch.zeros(shape)).frames
pipe = make_pipe(); pipe.set_progress_bar_config(disable=True)
b = pipe(**kwargs, generator=torch.Generator("cpu").manual_seed(123), latents=torch.ones(shape)).frames

print((a - b).abs().max().item()) # 0.0: supplied latents were ignored

Relevant precedent:
FluxPipeline.prepare_latents uses the provided latents when present and validates shape before sampling new noise.

Suggested fix:

if latents is not None:
 if isinstance(latents, (list, tuple)):
 if len(latents) != num_latent_chunk:
 raise ValueError("`latents` must contain one tensor per Helios chunk.")
 chunk_latents = latents[k]
 elif num_latent_chunk == 1:
 chunk_latents = latents
 else:
 raise ValueError("For multi-chunk Helios generation, pass `latents` as a list of chunk tensors.")
else:
 chunk_latents = None

latents = self.prepare_latents(..., latents=chunk_latents)

Issue 3: Non-pyramid Helios passes fractional float timesteps while pyramid and modular cast to int64

Affected code:

diffusers/src/diffusers/pipelines/helios/pipeline_helios.py

Line 815 in 0f1abc4

timestep = t.expand(latents.shape[0])

diffusers/src/diffusers/pipelines/helios/pipeline_helios_pyramid.py

Lines 950 to 952 in 0f1abc4

	for i, t in enumerate(timesteps):
	timestep = t.expand(latents.shape[0]).to(torch.int64)

diffusers/src/diffusers/modular_pipelines/helios/denoise.py

Lines 421 to 424 in 0f1abc4

	with tqdm(total=num_inference_steps) as progress_bar:
	for i, t in enumerate(timesteps):
	timestep = t.expand(latents.shape[0]).to(torch.int64)
	latent_model_input = latents.to(transformer_dtype)

Problem:
HeliosPipeline forwards scheduler timesteps directly. With its default sigma path, those timesteps are fractional float64 values. HeliosPyramidPipeline and modular Helios cast the same value to torch.int64 before calling the transformer.

Impact:
The standard non-pyramid pipeline is numerically inconsistent with the rest of the Helios family and with the model tests, which use integer timesteps. The timestep embedding changes measurably.

Reproduction:

import torch
from diffusers import HeliosTransformer3DModel
from diffusers.utils.torch_utils import randn_tensor

model = HeliosTransformer3DModel(
 patch_size=(1, 2, 2), num_attention_heads=2, attention_head_dim=12, in_channels=4, out_channels=4,
 text_dim=16, freq_dim=256, ffn_dim=32, num_layers=1, rope_dim=(4, 4, 4),
).eval()
g = torch.Generator("cpu").manual_seed(0)
kwargs = dict(
 hidden_states=randn_tensor((1, 4, 2, 16, 16), generator=g),
 encoder_hidden_states=randn_tensor((1, 12, 16), generator=g),
 indices_hidden_states=torch.ones((1, 2)),
 indices_latents_history_short=torch.ones((1, 1)),
 indices_latents_history_mid=torch.ones((1, 1)),
 indices_latents_history_long=torch.ones((1, 4)),
 latents_history_short=randn_tensor((1, 4, 1, 16, 16), generator=g),
 latents_history_mid=randn_tensor((1, 4, 1, 16, 16), generator=g),
 latents_history_long=randn_tensor((1, 4, 4, 16, 16), generator=g),
 return_dict=False,
)
with torch.no_grad():
 out_float = model(timestep=torch.tensor([499.5], dtype=torch.float64), **kwargs)[0]
 out_int = model(timestep=torch.tensor([499], dtype=torch.int64), **kwargs)[0]
print((out_float - out_int).abs().max().item()) # about 5e-2

Relevant precedent:
HeliosPyramidPipeline and HeliosChunkDenoiseInner both cast timesteps to torch.int64 before transformer invocation.

Suggested fix:

timestep = t.expand(latents.shape[0]).to(torch.int64)

Issue 4: Precomputed V2V latents hit UnboundLocalError

Affected code:

diffusers/src/diffusers/pipelines/helios/pipeline_helios.py

Lines 380 to 419 in 0f1abc4

	def prepare_video_latents(
	self,
	video: torch.Tensor,
	latents_mean: torch.Tensor,
	latents_std: torch.Tensor,
	num_latent_frames_per_chunk: int,
	dtype: torch.dtype | None = None,
	device: torch.device | None = None,
	generator: torch.Generator | list[torch.Generator] | None = None,
	latents: torch.Tensor | None = None,
	) -> torch.Tensor:
	device = device or self._execution_device
	video = video.to(device=device, dtype=self.vae.dtype)
	if latents is None:
	num_frames = video.shape[2]
	min_frames = (num_latent_frames_per_chunk - 1) * self.vae_scale_factor_temporal + 1
	num_chunks = num_frames // min_frames
	if num_chunks == 0:
	raise ValueError(
	f"Video must have at least {min_frames} frames "
	f"(got {num_frames} frames). "
	f"Required: (num_latent_frames_per_chunk - 1) * {self.vae_scale_factor_temporal} + 1 = ({num_latent_frames_per_chunk} - 1) * {self.vae_scale_factor_temporal} + 1 = {min_frames}"
	)
	total_valid_frames = num_chunks * min_frames
	start_frame = num_frames - total_valid_frames
	first_frame = video[:, :, 0:1, :, :]
	first_frame_latent = self.vae.encode(first_frame).latent_dist.sample(generator=generator)
	first_frame_latent = (first_frame_latent - latents_mean) * latents_std
	latents_chunks = []
	for i in range(num_chunks):
	chunk_start = start_frame + i * min_frames
	chunk_end = chunk_start + min_frames
	video_chunk = video[:, :, chunk_start:chunk_end, :, :]
	chunk_latents = self.vae.encode(video_chunk).latent_dist.sample(generator=generator)
	chunk_latents = (chunk_latents - latents_mean) * latents_std
	latents_chunks.append(chunk_latents)
	latents = torch.cat(latents_chunks, dim=2)
	return first_frame_latent.to(device=device, dtype=dtype), latents.to(device=device, dtype=dtype)

diffusers/src/diffusers/pipelines/helios/pipeline_helios_pyramid.py

Lines 402 to 441 in 0f1abc4

	def prepare_video_latents(
	self,
	video: torch.Tensor,
	latents_mean: torch.Tensor,
	latents_std: torch.Tensor,
	num_latent_frames_per_chunk: int,
	dtype: torch.dtype | None = None,
	device: torch.device | None = None,
	generator: torch.Generator | list[torch.Generator] | None = None,
	latents: torch.Tensor | None = None,
	) -> torch.Tensor:
	device = device or self._execution_device
	video = video.to(device=device, dtype=self.vae.dtype)
	if latents is None:
	num_frames = video.shape[2]
	min_frames = (num_latent_frames_per_chunk - 1) * self.vae_scale_factor_temporal + 1
	num_chunks = num_frames // min_frames
	if num_chunks == 0:
	raise ValueError(
	f"Video must have at least {min_frames} frames "
	f"(got {num_frames} frames). "
	f"Required: (num_latent_frames_per_chunk - 1) * {self.vae_scale_factor_temporal} + 1 = ({num_latent_frames_per_chunk} - 1) * {self.vae_scale_factor_temporal} + 1 = {min_frames}"
	)
	total_valid_frames = num_chunks * min_frames
	start_frame = num_frames - total_valid_frames
	first_frame = video[:, :, 0:1, :, :]
	first_frame_latent = self.vae.encode(first_frame).latent_dist.sample(generator=generator)
	first_frame_latent = (first_frame_latent - latents_mean) * latents_std
	latents_chunks = []
	for i in range(num_chunks):
	chunk_start = start_frame + i * min_frames
	chunk_end = chunk_start + min_frames
	video_chunk = video[:, :, chunk_start:chunk_end, :, :]
	chunk_latents = self.vae.encode(video_chunk).latent_dist.sample(generator=generator)
	chunk_latents = (chunk_latents - latents_mean) * latents_std
	latents_chunks.append(chunk_latents)
	latents = torch.cat(latents_chunks, dim=2)
	return first_frame_latent.to(device=device, dtype=dtype), latents.to(device=device, dtype=dtype)

Problem:
prepare_video_latents only defines first_frame_latent inside if latents is None, but returns it unconditionally. Passing precomputed video_latents with a raw video therefore crashes.

Impact:
The exposed video_latents skip-encoding path is unusable unless callers also avoid this helper entirely and provide all derived companion latents themselves.

Reproduction:

from types import SimpleNamespace
import torch
from diffusers import HeliosPipeline

pipe = SimpleNamespace(_execution_device=torch.device("cpu"), vae=SimpleNamespace(dtype=torch.float32), vae_scale_factor_temporal=4)
video = torch.zeros(1, 3, 33, 16, 16)
video_latents = torch.zeros(1, 16, 9, 2, 2)

HeliosPipeline.prepare_video_latents(
 pipe,
 video=video,
 latents_mean=torch.zeros(1, 16, 1, 1, 1),
 latents_std=torch.ones(1, 16, 1, 1, 1),
 num_latent_frames_per_chunk=9,
 dtype=torch.float32,
 device=torch.device("cpu"),
 latents=video_latents,
)

Relevant precedent:
Video-to-video pipelines that accept precomputed latents either validate companion inputs or still derive required first-frame latents from the raw video.

Suggested fix:

first_frame = video[:, :, 0:1, :, :]
first_frame_latent = self.vae.encode(first_frame).latent_dist.sample(generator=generator)
first_frame_latent = (first_frame_latent - latents_mean) * latents_std

if latents is None:
 # existing chunk encoding path
 ...
return first_frame_latent.to(device=device, dtype=dtype), latents.to(device=device, dtype=dtype)

Also validate direct video_latents calls without raw video: require image_latents alongside video_latents, or raise a clear ValueError.

Issue 5: Prompt cleaning crashes without optional ftfy

Affected code:

diffusers/src/diffusers/pipelines/helios/pipeline_helios.py

Lines 44 to 90 in 0f1abc4

	if is_ftfy_available():
	import ftfy
	EXAMPLE_DOC_STRING = """
	Examples:
	```python
	>>> import torch
	>>> from diffusers.utils import export_to_video
	>>> from diffusers import AutoencoderKLWan, HeliosPipeline
	>>> # Available models: BestWishYsh/Helios-Base, BestWishYsh/Helios-Mid, BestWishYsh/Helios-Distilled
	>>> model_id = "BestWishYsh/Helios-Base"
	>>> vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
	>>> pipe = HeliosPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16)
	>>> pipe.to("cuda")
	>>> prompt = "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window."
	>>> negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"
	>>> output = pipe(
	... prompt=prompt,
	... negative_prompt=negative_prompt,
	... height=384,
	... width=640,
	... num_frames=132,
	... guidance_scale=5.0,
	... ).frames[0]
	>>> export_to_video(output, "output.mp4", fps=24)
	```
	"""
	def basic_clean(text):
	text = ftfy.fix_text(text)
	text = html.unescape(html.unescape(text))
	return text.strip()
	def whitespace_clean(text):
	text = re.sub(r"\s+", " ", text)
	text = text.strip()
	return text
	def prompt_clean(text):
	text = whitespace_clean(basic_clean(text))

diffusers/src/diffusers/pipelines/helios/pipeline_helios_pyramid.py

Lines 45 to 103 in 0f1abc4

	if is_ftfy_available():
	import ftfy
	EXAMPLE_DOC_STRING = """
	Examples:
	```python
	>>> import torch
	>>> from diffusers.utils import export_to_video
	>>> from diffusers import AutoencoderKLWan, HeliosPyramidPipeline
	>>> # Available models: BestWishYsh/Helios-Base, BestWishYsh/Helios-Mid, BestWishYsh/Helios-Distilled
	>>> model_id = "BestWishYsh/Helios-Base"
	>>> vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
	>>> pipe = HeliosPyramidPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16)
	>>> pipe.to("cuda")
	>>> prompt = "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window."
	>>> negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"
	>>> output = pipe(
	... prompt=prompt,
	... negative_prompt=negative_prompt,
	... height=384,
	... width=640,
	... num_frames=132,
	... guidance_scale=5.0,
	... ).frames[0]
	>>> export_to_video(output, "output.mp4", fps=24)
	```
	"""
	def optimized_scale(positive_flat, negative_flat):
	positive_flat = positive_flat.float()
	negative_flat = negative_flat.float()
	# Calculate dot production
	dot_product = torch.sum(positive_flat * negative_flat, dim=1, keepdim=True)
	# Squared norm of uncondition
	squared_norm = torch.sum(negative_flat**2, dim=1, keepdim=True) + 1e-8
	# st_star = v_cond^T * v_uncond / ||v_uncond||^2
	st_star = dot_product / squared_norm
	return st_star
	def basic_clean(text):
	text = ftfy.fix_text(text)
	text = html.unescape(html.unescape(text))
	return text.strip()
	def whitespace_clean(text):
	text = re.sub(r"\s+", " ", text)
	text = text.strip()
	return text
	def prompt_clean(text):
	text = whitespace_clean(basic_clean(text))

diffusers/src/diffusers/modular_pipelines/helios/encoders.py

Lines 31 to 51 in 0f1abc4

	if is_ftfy_available():
	import ftfy
	logger = logging.get_logger(__name__) # pylint: disable=invalid-name
	def basic_clean(text):
	text = ftfy.fix_text(text)
	text = html.unescape(html.unescape(text))
	return text.strip()
	def whitespace_clean(text):
	text = re.sub(r"\s+", " ", text)
	text = text.strip()
	return text
	def prompt_clean(text):
	text = whitespace_clean(basic_clean(text))

Problem:
The modules conditionally import ftfy, but basic_clean calls ftfy.fix_text unconditionally. ftfy is not in install_requires, so minimal installs can import Helios but fail when encoding any string prompt.

Impact:
Text-to-video generation crashes at prompt encoding in environments that install only core diffusers plus torch/transformers.

Reproduction:

import diffusers.pipelines.helios.pipeline_helios as standard
import diffusers.pipelines.helios.pipeline_helios_pyramid as pyramid
import diffusers.modular_pipelines.helios.encoders as modular

for module in (standard, pyramid, modular):
 if hasattr(module, "ftfy"):
 delattr(module, "ftfy")
 print(module.prompt_clean("hello & world"))

Relevant precedent:

diffusers/src/diffusers/pipelines/wan/pipeline_wan.py

Lines 78 to 92 in 0f1abc4

	def basic_clean(text):
	if is_ftfy_available():
	text = ftfy.fix_text(text)
	text = html.unescape(html.unescape(text))
	return text.strip()
	def whitespace_clean(text):
	text = re.sub(r"\s+", " ", text)
	text = text.strip()
	return text
	def prompt_clean(text):
	text = whitespace_clean(basic_clean(text))

Suggested fix:

def basic_clean(text):
 if is_ftfy_available():
 text = ftfy.fix_text(text)
 text = html.unescape(html.unescape(text))
 return text.strip()

Issue 6: Slow/integration coverage is effectively missing

Affected code:

diffusers/tests/pipelines/helios/test_helios.py

Lines 37 to 154 in 0f1abc4

	class HeliosPipelineFastTests(PipelineTesterMixin, unittest.TestCase):
	pipeline_class = HeliosPipeline
	params = TEXT_TO_IMAGE_PARAMS - {"cross_attention_kwargs"}
	batch_params = TEXT_TO_IMAGE_BATCH_PARAMS
	image_params = TEXT_TO_IMAGE_IMAGE_PARAMS
	image_latents_params = TEXT_TO_IMAGE_IMAGE_PARAMS
	required_optional_params = frozenset(
	[
	"num_inference_steps",
	"generator",
	"latents",
	"return_dict",
	"callback_on_step_end",
	"callback_on_step_end_tensor_inputs",
	]
	)
	test_xformers_attention = False
	supports_dduf = False
	def get_dummy_components(self):
	torch.manual_seed(0)
	vae = AutoencoderKLWan(
	base_dim=3,
	z_dim=16,
	dim_mult=[1, 1, 1, 1],
	num_res_blocks=1,
	temperal_downsample=[False, True, True],
	)
	torch.manual_seed(0)
	scheduler = HeliosScheduler(stage_range=[0, 1], stages=1, use_dynamic_shifting=True)
	config = AutoConfig.from_pretrained("hf-internal-testing/tiny-random-t5")
	text_encoder = T5EncoderModel(config)
	tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-t5")
	torch.manual_seed(0)
	transformer = HeliosTransformer3DModel(
	patch_size=(1, 2, 2),
	num_attention_heads=2,
	attention_head_dim=12,
	in_channels=16,
	out_channels=16,
	text_dim=32,
	freq_dim=256,
	ffn_dim=32,
	num_layers=2,
	cross_attn_norm=True,
	qk_norm="rms_norm_across_heads",
	rope_dim=(4, 4, 4),
	has_multi_term_memory_patch=True,
	guidance_cross_attn=True,
	zero_history_timestep=True,
	is_amplify_history=False,
	)
	components = {
	"transformer": transformer,
	"vae": vae,
	"scheduler": scheduler,
	"text_encoder": text_encoder,
	"tokenizer": tokenizer,
	}
	return components
	def get_dummy_inputs(self, device, seed=0):
	if str(device).startswith("mps"):
	generator = torch.manual_seed(seed)
	else:
	generator = torch.Generator(device=device).manual_seed(seed)
	inputs = {
	"prompt": "dance monkey",
	"negative_prompt": "negative",
	"generator": generator,
	"num_inference_steps": 2,
	"guidance_scale": 1.0,
	"height": 16,
	"width": 16,
	"num_frames": 9,
	"max_sequence_length": 16,
	"output_type": "pt",
	}
	return inputs
	def test_inference(self):
	device = "cpu"
	components = self.get_dummy_components()
	pipe = self.pipeline_class(**components)
	pipe.to(device)
	pipe.set_progress_bar_config(disable=None)
	inputs = self.get_dummy_inputs(device)
	video = pipe(**inputs).frames
	generated_video = video[0]
	self.assertEqual(generated_video.shape, (33, 3, 16, 16))
	# fmt: off
	expected_slice = torch.tensor([0.4529, 0.4527, 0.4499, 0.4542, 0.4528, 0.4524, 0.4531, 0.4534, 0.5328,
	0.5340, 0.5012, 0.5135, 0.5322, 0.5203, 0.5144, 0.5101])
	# fmt: on
	generated_slice = generated_video.flatten()
	generated_slice = torch.cat([generated_slice[:8], generated_slice[-8:]])
	self.assertTrue(torch.allclose(generated_slice, expected_slice, atol=1e-3))
	@unittest.skip("Helios uses a lot of mixed precision internally, which is not suitable for this test case")
	def test_save_load_float16(self):
	pass
	@unittest.skip("Test not supported")
	def test_attention_slicing_forward_pass(self):
	pass
	@unittest.skip("Optional components not applicable for Helios")
	def test_save_load_optional_components(self):
	pass

diffusers/tests/pipelines/helios/test_helios.py

Lines 155 to 172 in 0f1abc4

	@slow
	@require_torch_accelerator
	class HeliosPipelineIntegrationTests(unittest.TestCase):
	prompt = "A painting of a squirrel eating a burger."
	def setUp(self):
	super().setUp()
	gc.collect()
	backend_empty_cache(torch_device)
	def tearDown(self):
	super().tearDown()
	gc.collect()
	backend_empty_cache(torch_device)
	@unittest.skip("TODO: test needs to be implemented")
	def test_helios(self):
	pass

diffusers/tests/modular_pipelines/helios/test_modular_pipeline_helios.py

Lines 127 to 166 in 0f1abc4

	class TestHeliosPyramidModularPipelineFast(ModularPipelineTesterMixin):
	pipeline_class = HeliosPyramidModularPipeline
	pipeline_blocks_class = HeliosPyramidAutoBlocks
	pretrained_model_name_or_path = "hf-internal-testing/tiny-helios-pyramid-modular-pipe"
	params = frozenset(["prompt", "height", "width", "num_frames"])
	batch_params = frozenset(["prompt"])
	optional_params = frozenset(["pyramid_num_inference_steps_list", "num_videos_per_prompt", "latents"])
	output_name = "videos"
	expected_workflow_blocks = HELIOS_PYRAMID_WORKFLOWS
	def get_dummy_inputs(self, seed=0):
	generator = self.get_generator(seed)
	inputs = {
	"prompt": "A painting of a squirrel eating a burger",
	"generator": generator,
	"pyramid_num_inference_steps_list": [2, 2],
	"height": 64,
	"width": 64,
	"num_frames": 9,
	"max_sequence_length": 16,
	"output_type": "pt",
	}
	return inputs
	def test_inference_batch_single_identical(self):
	# Pyramid pipeline injects noise at each stage, so batch vs single can differ more
	super().test_inference_batch_single_identical(expected_max_diff=5e-1)
	@pytest.mark.skip(reason="Pyramid multi-stage noise makes offload comparison unreliable with tiny models")
	def test_components_auto_cpu_offload_inference_consistent(self):
	pass
	@pytest.mark.skip(reason="Pyramid multi-stage noise makes save/load comparison unreliable with tiny models")
	def test_save_from_pretrained(self):
	pass
	@pytest.mark.skip(reason="num_videos_per_prompt")
	def test_num_images_per_prompt(self):
	pass

Problem:
The only standard pipeline slow test is skipped with TODO: test needs to be implemented. There is no fast standard test class for HeliosPyramidPipeline, and no modular test coverage for HeliosPyramidDistilledModularPipeline.

Impact:
The exact public paths with the most branching - pyramid, distilled, slow checkpoint loading, and integration generation - can regress without CI signal. This also hides the runtime issues above.

Reproduction:

from pathlib import Path

standard = Path("tests/pipelines/helios/test_helios.py").read_text()
modular = Path("tests/modular_pipelines/helios/test_modular_pipeline_helios.py").read_text()

print("slow class present:", "class HeliosPipelineIntegrationTests" in standard)
print("slow test skipped:", '@unittest.skip("TODO: test needs to be implemented")' in standard)
print("standard pyramid fast test present:", "HeliosPyramidPipeline" in standard)
print("distilled modular test present:", "HeliosPyramidDistilled" in modular)

Relevant precedent:
Existing video pipeline suites generally include a real slow test for the public checkpoint and fast tests for each exported pipeline variant.

Suggested fix:
Add non-skipped slow tests for HeliosPipeline and HeliosPyramidPipeline using small generation settings, plus fast tests for HeliosPyramidPipeline and HeliosPyramidDistilledModularPipeline. Add explicit assertions for latents, num_videos_per_prompt, precomputed I2V/V2V latents, and standard-vs-modular timestep parity.

Note: I attempted to run tests/pipelines/helios/test_helios.py::HeliosPipelineFastTests::test_inference with .venv, but collection fails in this environment because the installed Torch build lacks torch._C._distributed_c10d, imported through shared test utilities.