latent_diffusion model/pipeline review

[hlky]

latent_diffusion model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Duplicate search: searched GitHub Issues/PRs for latent_diffusion, LDMTextToImagePipeline, LDMSuperResolutionPipeline, LDMBertConfig, and the specific failure modes below. I found no matching duplicates; old issues #170/#211 are unrelated historical failures.

Issue 1: guidance_scale=0 ignores the prompt

Affected code:

diffusers/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py

Lines 153 to 207 in 0f1abc4

	# get unconditional embeddings for classifier free guidance
	if guidance_scale != 1.0:
	uncond_input = self.tokenizer(
	[""] * batch_size, padding="max_length", max_length=77, truncation=True, return_tensors="pt"
	)
	negative_prompt_embeds = self.bert(uncond_input.input_ids.to(self._execution_device))[0]
	# get prompt text embeddings
	text_input = self.tokenizer(prompt, padding="max_length", max_length=77, truncation=True, return_tensors="pt")
	prompt_embeds = self.bert(text_input.input_ids.to(self._execution_device))[0]
	# get the initial random noise unless the user supplied it
	latents_shape = (batch_size, self.unet.config.in_channels, height // 8, width // 8)
	if isinstance(generator, list) and len(generator) != batch_size:
	raise ValueError(
	f"You have passed a list of generators of length {len(generator)}, but requested an effective batch"
	f" size of {batch_size}. Make sure the batch size matches the length of the generators."
	)
	if latents is None:
	latents = randn_tensor(
	latents_shape, generator=generator, device=self._execution_device, dtype=prompt_embeds.dtype
	)
	else:
	if latents.shape != latents_shape:
	raise ValueError(f"Unexpected latents shape, got {latents.shape}, expected {latents_shape}")
	latents = latents.to(self._execution_device)
	self.scheduler.set_timesteps(num_inference_steps)
	# prepare extra kwargs for the scheduler step, since not all schedulers have the same signature
	accepts_eta = "eta" in set(inspect.signature(self.scheduler.step).parameters.keys())
	extra_kwargs = {}
	if accepts_eta:
	extra_kwargs["eta"] = eta
	for t in self.progress_bar(self.scheduler.timesteps):
	if guidance_scale == 1.0:
	# guidance_scale of 1 means no guidance
	latents_input = latents
	context = prompt_embeds
	else:
	# For classifier free guidance, we need to do two forward passes.
	# Here we concatenate the unconditional and text embeddings into a single batch
	# to avoid doing two forward passes
	latents_input = torch.cat([latents] * 2)
	context = torch.cat([negative_prompt_embeds, prompt_embeds])
	# predict the noise residual
	noise_pred = self.unet(latents_input, t, encoder_hidden_states=context).sample
	# perform guidance
	if guidance_scale != 1.0:
	noise_pred_uncond, noise_prediction_text = noise_pred.chunk(2)
	noise_pred = noise_pred_uncond + guidance_scale * (noise_prediction_text - noise_pred_uncond)

Problem:
CFG is enabled whenever guidance_scale != 1.0. Diffusers pipelines document and implement CFG as enabled when guidance_scale > 1; at 0 or 0.5, users expect no CFG, not unconditional-prompt interpolation. With the current branch, guidance_scale=0 returns the unconditional prediction and removes prompt influence.

Impact:
Low guidance values silently produce prompt-insensitive or under-conditioned results.

Reproduction:

import torch
from types import SimpleNamespace
from diffusers import DDIMScheduler, LDMTextToImagePipeline

class M(torch.nn.Module):
 @property
 def device(self): return next(self.parameters()).device
 @property
 def dtype(self): return next(self.parameters()).dtype

class Tok:
 def __call__(self, prompt, max_length=None, **_):
 prompt = [prompt] if isinstance(prompt, str) else prompt
 ids = torch.zeros(len(prompt), max_length, dtype=torch.long)
 for i, p in enumerate(prompt):
 ids[i, 0] = sum(map(ord, p)) % 100 + 1 if p else 0
 return SimpleNamespace(input_ids=ids)

class Text(M):
 def __init__(self): super().__init__(); self.p = torch.nn.Parameter(torch.ones(()))
 def forward(self, ids): return (ids.float().unsqueeze(-1).repeat(1, 1, 32) / 100,)

class UNet(M):
 def __init__(self):
 super().__init__(); self.p = torch.nn.Parameter(torch.ones(()))
 self.config = SimpleNamespace(in_channels=4, sample_size=8)
 def forward(self, sample, timestep, encoder_hidden_states=None):
 v = encoder_hidden_states[:, 0, 0].view(-1, 1, 1, 1).to(sample)
 return SimpleNamespace(sample=v.expand_as(sample))

class VAE(M):
 def __init__(self):
 super().__init__(); self.p = torch.nn.Parameter(torch.ones(()))
 self.config = SimpleNamespace(block_out_channels=(1,1,1,1), scaling_factor=1.0)
 def decode(self, latents):
 return SimpleNamespace(sample=latents[:, :3].repeat_interleave(8, -1).repeat_interleave(8, -2))

pipe = LDMTextToImagePipeline(VAE(), Text(), Tok(), UNet(), DDIMScheduler())
pipe.set_progress_bar_config(disable=True)
latents = torch.zeros(1, 4, 8, 8)
run = lambda p, gs: pipe(p, latents=latents.clone(), num_inference_steps=1, guidance_scale=gs, output_type="np").images
print(abs(run("cat", 0.0) - run("dog", 0.0)).max()) # 0.0: prompt ignored
print(abs(run("cat", 1.0) - run("dog", 1.0)).max()) # prompt affects output

Relevant precedent:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Lines 763 to 764 in 0f1abc4

	def do_classifier_free_guidance(self):
	return self._guidance_scale > 1 and self.unet.config.time_cond_proj_dim is None

Suggested fix:

do_classifier_free_guidance = guidance_scale > 1.0
if do_classifier_free_guidance:
 ...
if do_classifier_free_guidance:
 latents_input = torch.cat([latents] * 2)
 context = torch.cat([negative_prompt_embeds, prompt_embeds])
else:
 latents_input = latents
 context = prompt_embeds

Issue 2: text-to-image hard-codes latent scale factor 8

Affected code:

diffusers/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py

Lines 139 to 165 in 0f1abc4

	# 0. Default height and width to unet
	height = height or self.unet.config.sample_size * self.vae_scale_factor
	width = width or self.unet.config.sample_size * self.vae_scale_factor
	if isinstance(prompt, str):
	batch_size = 1
	elif isinstance(prompt, list):
	batch_size = len(prompt)
	else:
	raise ValueError(f"`prompt` has to be of type `str` or `list` but is {type(prompt)}")
	if height % 8 != 0 or width % 8 != 0:
	raise ValueError(f"`height` and `width` have to be divisible by 8 but are {height} and {width}.")
	# get unconditional embeddings for classifier free guidance
	if guidance_scale != 1.0:
	uncond_input = self.tokenizer(
	[""] * batch_size, padding="max_length", max_length=77, truncation=True, return_tensors="pt"
	)
	negative_prompt_embeds = self.bert(uncond_input.input_ids.to(self._execution_device))[0]
	# get prompt text embeddings
	text_input = self.tokenizer(prompt, padding="max_length", max_length=77, truncation=True, return_tensors="pt")
	prompt_embeds = self.bert(text_input.input_ids.to(self._execution_device))[0]
	# get the initial random noise unless the user supplied it
	latents_shape = (batch_size, self.unet.config.in_channels, height // 8, width // 8)

Problem:
self.vae_scale_factor is computed in __init__, but validation and latent shape still use hard-coded 8. Tiny VAEs or any compatible AutoencoderKL/VQModel with a different scale factor generate the wrong output size or reject valid dimensions.

Impact:
The pipeline is inconsistent with its own config-derived scale factor and with fast-test-sized components.

Reproduction:

# Same tiny component pattern as above, but VAE has scale factor 2.
# The pipeline requests default height 16, but creates latents at height//8 and decodes to 4.
print(pipe.vae_scale_factor)
print(pipe.unet.config.sample_size * pipe.vae_scale_factor)
print(pipe("x", num_inference_steps=1, guidance_scale=1.0, output_type="np").images.shape)
# observed shape: (1, 4, 4, 3), expected height/width: 16

Relevant precedent:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Lines 694 to 700 in 0f1abc4

	def prepare_latents(self, batch_size, num_channels_latents, height, width, dtype, device, generator, latents=None):
	shape = (
	batch_size,
	num_channels_latents,
	int(height) // self.vae_scale_factor,
	int(width) // self.vae_scale_factor,
	)

Suggested fix:

if height % self.vae_scale_factor != 0 or width % self.vae_scale_factor != 0:
 raise ValueError(
 f"`height` and `width` have to be divisible by {self.vae_scale_factor} but are {height} and {width}."
 )

latents_shape = (
 batch_size,
 self.unet.config.in_channels,
 height // self.vae_scale_factor,
 width // self.vae_scale_factor,
)

Issue 3: LMS scheduler support is incomplete in text-to-image

Affected code:

diffusers/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py

Lines 172 to 210 in 0f1abc4

	if latents is None:
	latents = randn_tensor(
	latents_shape, generator=generator, device=self._execution_device, dtype=prompt_embeds.dtype
	)
	else:
	if latents.shape != latents_shape:
	raise ValueError(f"Unexpected latents shape, got {latents.shape}, expected {latents_shape}")
	latents = latents.to(self._execution_device)
	self.scheduler.set_timesteps(num_inference_steps)
	# prepare extra kwargs for the scheduler step, since not all schedulers have the same signature
	accepts_eta = "eta" in set(inspect.signature(self.scheduler.step).parameters.keys())
	extra_kwargs = {}
	if accepts_eta:
	extra_kwargs["eta"] = eta
	for t in self.progress_bar(self.scheduler.timesteps):
	if guidance_scale == 1.0:
	# guidance_scale of 1 means no guidance
	latents_input = latents
	context = prompt_embeds
	else:
	# For classifier free guidance, we need to do two forward passes.
	# Here we concatenate the unconditional and text embeddings into a single batch
	# to avoid doing two forward passes
	latents_input = torch.cat([latents] * 2)
	context = torch.cat([negative_prompt_embeds, prompt_embeds])
	# predict the noise residual
	noise_pred = self.unet(latents_input, t, encoder_hidden_states=context).sample
	# perform guidance
	if guidance_scale != 1.0:
	noise_pred_uncond, noise_prediction_text = noise_pred.chunk(2)
	noise_pred = noise_pred_uncond + guidance_scale * (noise_prediction_text - noise_pred_uncond)
	# compute the previous noisy sample x_t -> x_t-1
	latents = self.scheduler.step(noise_pred, t, latents, **extra_kwargs).prev_sample

Problem:
The docstring advertises LMSDiscreteScheduler, but __call__ never scales initial noise by scheduler.init_noise_sigma and never calls scheduler.scale_model_input(...). LMSDiscreteScheduler.step() emits the standard warning that scale_model_input was skipped.

Impact:
Schedulers whose model input scaling is non-noop are driven outside their expected contract, causing incorrect denoising.

Reproduction:

import warnings
# Build the same tiny pipeline as Issue 1, but use LMSDiscreteScheduler.
from diffusers import LMSDiscreteScheduler
pipe.scheduler = LMSDiscreteScheduler()
with warnings.catch_warnings(record=True) as caught:
 warnings.simplefilter("always")
 pipe("x", num_inference_steps=4, guidance_scale=1.0, output_type="np")
print([str(w.message) for w in caught if "scale_model_input" in str(w.message)])

Relevant precedent:

diffusers/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

Lines 161 to 175 in 0f1abc4

	latents = latents * self.scheduler.init_noise_sigma
	# prepare extra kwargs for the scheduler step, since not all schedulers have the same signature.
	# eta (η) is only used with the DDIMScheduler, it will be ignored for other schedulers.
	# eta corresponds to η in DDIM paper: https://huggingface.co/papers/2010.02502
	# and should be between [0, 1]
	accepts_eta = "eta" in set(inspect.signature(self.scheduler.step).parameters.keys())
	extra_kwargs = {}
	if accepts_eta:
	extra_kwargs["eta"] = eta
	for t in self.progress_bar(timesteps_tensor):
	# concat latents and low resolution image in the channel dimension.
	latents_input = torch.cat([latents, image], dim=1)
	latents_input = self.scheduler.scale_model_input(latents_input, t)

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Line 713 in 0f1abc4

latents = latents * self.scheduler.init_noise_sigma

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Lines 1037 to 1039 in 0f1abc4

	latent_model_input = torch.cat([latents] * 2) if self.do_classifier_free_guidance else latents
	if hasattr(self.scheduler, "scale_model_input"):
	latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)

Suggested fix:

self.scheduler.set_timesteps(num_inference_steps, device=self._execution_device)
...
latents = latents * self.scheduler.init_noise_sigma
...
latents_input = self.scheduler.scale_model_input(latents_input, t)
noise_pred = self.unet(latents_input, t, encoder_hidden_states=context).sample

Issue 4: super-resolution cannot use model CPU offload

Affected code:

diffusers/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

Lines 39 to 69 in 0f1abc4

	class LDMSuperResolutionPipeline(DiffusionPipeline):
	r"""
	A pipeline for image super-resolution using latent diffusion.
	This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods
	implemented for all pipelines (downloading, saving, running on a particular device, etc.).
	Parameters:
	vqvae ([`VQModel`]):
	Vector-quantized (VQ) model to encode and decode images to and from latent representations.
	unet ([`UNet2DModel`]):
	A `UNet2DModel` to denoise the encoded image.
	scheduler ([`SchedulerMixin`]):
	A scheduler to be used in combination with `unet` to denoise the encoded image latens. Can be one of
	[`DDIMScheduler`], [`LMSDiscreteScheduler`], [`EulerDiscreteScheduler`],
	[`EulerAncestralDiscreteScheduler`], [`DPMSolverMultistepScheduler`], or [`PNDMScheduler`].
	"""
	def __init__(
	self,
	vqvae: VQModel,
	unet: UNet2DModel,
	scheduler: DDIMScheduler
	| PNDMScheduler
	| LMSDiscreteScheduler
	| EulerDiscreteScheduler
	| EulerAncestralDiscreteScheduler
	| DPMSolverMultistepScheduler,
	):
	super().__init__()
	self.register_modules(vqvae=vqvae, unet=unet, scheduler=scheduler)

diffusers/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion_superresolution.py

Lines 152 to 157 in 0f1abc4

	latents = randn_tensor(latents_shape, generator=generator, device=self.device, dtype=latents_dtype)
	image = image.to(device=self.device, dtype=latents_dtype)
	# set timesteps and move to the correct device
	self.scheduler.set_timesteps(num_inference_steps, device=self.device)

Problem:
LDMSuperResolutionPipeline does not set model_cpu_offload_seq, so enable_model_cpu_offload() always raises. The runtime path also uses self.device instead of _execution_device, which would create tensors on CPU even after offload hooks are added.

Impact:
A large published super-resolution pipeline cannot use the normal low-memory offload path.

Reproduction:

from diffusers import DDIMScheduler, LDMSuperResolutionPipeline, UNet2DModel, VQModel

unet = UNet2DModel(sample_size=32, in_channels=6, out_channels=3, block_out_channels=(32, 64))
vqvae = VQModel(in_channels=3, out_channels=3, latent_channels=3, block_out_channels=(32, 64))
pipe = LDMSuperResolutionPipeline(vqvae=vqvae, unet=unet, scheduler=DDIMScheduler())

try:
 pipe.enable_model_cpu_offload(device="cpu")
except Exception as e:
 print(type(e).__name__, e)

Relevant precedent:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py

Line 114 in 0f1abc4

model_cpu_offload_seq = "text_encoder->unet->vae"

Suggested fix:

class LDMSuperResolutionPipeline(DiffusionPipeline):
 model_cpu_offload_seq = "unet->vqvae"

 ...
 device = self._execution_device
 latents = randn_tensor(latents_shape, generator=generator, device=device, dtype=latents_dtype)
 image = image.to(device=device, dtype=latents_dtype)
 self.scheduler.set_timesteps(num_inference_steps, device=device)

Issue 5: LDMBertConfig is defined but not exported by lazy imports

Affected code:

diffusers/src/diffusers/pipelines/latent_diffusion/__init__.py

Lines 24 to 37 in 0f1abc4

	_import_structure["pipeline_latent_diffusion"] = ["LDMBertModel", "LDMTextToImagePipeline"]
	_import_structure["pipeline_latent_diffusion_superresolution"] = ["LDMSuperResolutionPipeline"]
	if TYPE_CHECKING or DIFFUSERS_SLOW_IMPORT:
	try:
	if not (is_transformers_available() and is_torch_available()):
	raise OptionalDependencyNotAvailable()
	except OptionalDependencyNotAvailable:
	from ...utils.dummy_torch_and_transformers_objects import *
	else:
	from .pipeline_latent_diffusion import LDMBertModel, LDMTextToImagePipeline
	from .pipeline_latent_diffusion_superresolution import LDMSuperResolutionPipeline

diffusers/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py

Lines 252 to 296 in 0f1abc4

	class LDMBertConfig(PretrainedConfig):
	model_type = "ldmbert"
	keys_to_ignore_at_inference = ["past_key_values"]
	attribute_map = {"num_attention_heads": "encoder_attention_heads", "hidden_size": "d_model"}
	def __init__(
	self,
	vocab_size=30522,
	max_position_embeddings=77,
	encoder_layers=32,
	encoder_ffn_dim=5120,
	encoder_attention_heads=8,
	head_dim=64,
	encoder_layerdrop=0.0,
	activation_function="gelu",
	d_model=1280,
	dropout=0.1,
	attention_dropout=0.0,
	activation_dropout=0.0,
	init_std=0.02,
	classifier_dropout=0.0,
	scale_embedding=False,
	use_cache=True,
	pad_token_id=0,
	**kwargs,
	):
	self.vocab_size = vocab_size
	self.max_position_embeddings = max_position_embeddings
	self.d_model = d_model
	self.encoder_ffn_dim = encoder_ffn_dim
	self.encoder_layers = encoder_layers
	self.encoder_attention_heads = encoder_attention_heads
	self.head_dim = head_dim
	self.dropout = dropout
	self.attention_dropout = attention_dropout
	self.activation_dropout = activation_dropout
	self.activation_function = activation_function
	self.init_std = init_std
	self.encoder_layerdrop = encoder_layerdrop
	self.classifier_dropout = classifier_dropout
	self.use_cache = use_cache
	self.num_hidden_layers = encoder_layers
	self.scale_embedding = scale_embedding # scale factor will be sqrt(d_model) if True
	super().__init__(pad_token_id=pad_token_id, **kwargs)

Problem:
LDMBertModel is exported from diffusers.pipelines.latent_diffusion, but its matching LDMBertConfig is not. Users and conversion utilities must import from the private file path instead of the package.

Impact:
Public import behavior is inconsistent for the model/config pair.

Reproduction:

from diffusers.pipelines.latent_diffusion import LDMBertModel
print(LDMBertModel.__name__)

from diffusers.pipelines.latent_diffusion import LDMBertConfig
# ImportError: cannot import name 'LDMBertConfig'

Relevant precedent:
Model/config pairs elsewhere in diffusers are exported together from their owning package.

Suggested fix:

_import_structure["pipeline_latent_diffusion"] = ["LDMBertConfig", "LDMBertModel", "LDMTextToImagePipeline"]

...
from .pipeline_latent_diffusion import LDMBertConfig, LDMBertModel, LDMTextToImagePipeline

Issue 6: no @slow coverage for the family

Affected code:

diffusers/tests/pipelines/latent_diffusion/test_latent_diffusion.py

Lines 139 to 182 in 0f1abc4

	@nightly
	@require_torch_accelerator
	class LDMTextToImagePipelineSlowTests(unittest.TestCase):
	def setUp(self):
	super().setUp()
	gc.collect()
	backend_empty_cache(torch_device)
	def tearDown(self):
	super().tearDown()
	gc.collect()
	backend_empty_cache(torch_device)
	def get_inputs(self, device, dtype=torch.float32, seed=0):
	generator = torch.manual_seed(seed)
	latents = np.random.RandomState(seed).standard_normal((1, 4, 32, 32))
	latents = torch.from_numpy(latents).to(device=device, dtype=dtype)
	inputs = {
	"prompt": "A painting of a squirrel eating a burger",
	"latents": latents,
	"generator": generator,
	"num_inference_steps": 3,
	"guidance_scale": 6.0,
	"output_type": "np",
	}
	return inputs
	def test_ldm_default_ddim(self):
	pipe = LDMTextToImagePipeline.from_pretrained("CompVis/ldm-text2im-large-256").to(torch_device)
	pipe.set_progress_bar_config(disable=None)
	inputs = self.get_inputs(torch_device)
	image = pipe(**inputs).images
	image_slice = image[0, -3:, -3:, -1].flatten()
	assert image.shape == (1, 256, 256, 3)
	expected_slice = np.array([0.51825, 0.52850, 0.52543, 0.54258, 0.52304, 0.52569, 0.54363, 0.55276, 0.56878])
	max_diff = np.abs(expected_slice - image_slice).max()
	assert max_diff < 1e-3
	@nightly
	@require_torch_accelerator
	class LDMTextToImagePipelineNightlyTests(unittest.TestCase):

diffusers/tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py

Lines 119 to 133 in 0f1abc4

	@nightly
	@require_torch
	class LDMSuperResolutionPipelineIntegrationTests(unittest.TestCase):
	def test_inference_superresolution(self):
	init_image = load_image(
	"https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main"
	"/vq_diffusion/teddy_bear_pool.png"
	)
	init_image = init_image.resize((64, 64), resample=PIL_INTERPOLATION["lanczos"])
	ldm = LDMSuperResolutionPipeline.from_pretrained("duongna/ldm-super-resolution")
	ldm.set_progress_bar_config(disable=None)
	generator = torch.manual_seed(0)
	image = ldm(image=init_image, generator=generator, num_inference_steps=20, output_type="np").images

Problem:
The target has fast tests and nightly tests, but no @slow tests. LDMTextToImagePipelineSlowTests is named “Slow” but decorated with @nightly, and super-resolution only has @nightly integration coverage.

Impact:
RUN_SLOW does not exercise either published checkpoint path, despite the family having slow/nightly-only behavior not covered by fast tests.

Reproduction:

from pathlib import Path

for path in [
 "tests/pipelines/latent_diffusion/test_latent_diffusion.py",
 "tests/pipelines/latent_diffusion/test_latent_diffusion_superresolution.py",
]:
 text = Path(path).read_text()
 print(path, "@slow" in text, "@nightly" in text)

Relevant precedent:
Other pipeline integration suites use @slow for normal checkpoint regression tests and reserve @nightly for heavier/full-output cases.

Suggested fix:

from ...testing_utils import slow

@slow
@require_torch_accelerator
class LDMTextToImagePipelineSlowTests(unittest.TestCase):
 ...

@slow
@require_torch
class LDMSuperResolutionPipelineSlowTests(unittest.TestCase):
 ...

Verification: LDMSuperResolutionPipelineFastTests::test_inference_superresolution passed locally. The text-to-image fast test could not be collected in this .venv because the local Torch build is missing torch._C._distributed_c10d, imported through the shared pipeline test mixin.