t2i_adapter model/pipeline review

[hlky]

t2i_adapter model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Reviewed: target model/pipeline files, public exports/lazy imports, serialization/loading, dtype/device/offload paths, related SD/SDXL precedents, fast/slow tests, docs, and examples. Public imports and lazy-loading registration look correct.

Duplicate searches run with gh search issues/prs for t2i_adapter, affected class names, MultiAdapter, adapter_conditioning_scale, iteration over a 0-d tensor, SDXL list adapters, latent output, PathLike save/load, docs scheduler typo, and slow coverage.

Issue 1: MultiAdapter still breaks on the pipeline default scale

Affected code:

diffusers/src/diffusers/models/adapter.py

Lines 88 to 94 in 0f1abc4

	if adapter_weights is None:
	adapter_weights = torch.tensor([1 / self.num_adapter] * self.num_adapter)
	else:
	adapter_weights = torch.tensor(adapter_weights)
	accume_state = None
	for x, w, adapter in zip(xs, adapter_weights, self.adapters):

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py

Lines 884 to 885 in 0f1abc4

	if isinstance(self.adapter, MultiAdapter):
	adapter_state = self.adapter(adapter_input, adapter_conditioning_scale)

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py

Lines 1166 to 1167 in 0f1abc4

	if isinstance(self.adapter, MultiAdapter):
	adapter_state = self.adapter(adapter_input, adapter_conditioning_scale)

Problem:
Both pipelines pass the default adapter_conditioning_scale=1.0 to MultiAdapter.forward. MultiAdapter.forward converts that float to a scalar tensor and then iterates it, raising TypeError: iteration over a 0-d tensor. It also silently truncates when a scale list has the wrong length.

Duplicate check:
This exact default-scale failure was reported in closed issue #6274 and still reproduces on this commit, so this is not a new finding.

Impact:
A documented/default multi-adapter call fails unless users know to pass a list. Wrong-length scale lists can silently skip adapters.

Reproduction:

import torch
from diffusers import MultiAdapter, T2IAdapter

multi = MultiAdapter([
 T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
 T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])
xs = [torch.randn(1, 3, 8, 8), torch.randn(1, 3, 8, 8)]

try:
 multi(xs, 1.0)
except Exception as e:
 print(type(e).__name__, str(e))

print("short list accepted:", multi(xs, [1.0])[0].shape)

Relevant precedent:

diffusers/examples/community/pipeline_stable_diffusion_xl_controlnet_adapter.py

Lines 1089 to 1090 in 0f1abc4

	if isinstance(adapter, MultiAdapter) and isinstance(adapter_conditioning_scale, float):
	adapter_conditioning_scale = [adapter_conditioning_scale] * len(adapter.adapters)

Suggested fix:

if adapter_weights is None:
 adapter_weights = [1 / self.num_adapter] * self.num_adapter
elif isinstance(adapter_weights, (float, int)):
 adapter_weights = [float(adapter_weights)] * self.num_adapter
elif len(adapter_weights) != self.num_adapter:
 raise ValueError(
 f"`adapter_weights` must have length {self.num_adapter}, got {len(adapter_weights)}."
 )

if len(xs) != self.num_adapter:
 raise ValueError(f"`xs` must have length {self.num_adapter}, got {len(xs)}.")

Issue 2: SDXL adapter pipeline does not accept list[T2IAdapter] despite its public signature

Affected code:

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py

Lines 273 to 290 in 0f1abc4

	def __init__(
	self,
	vae: AutoencoderKL,
	text_encoder: CLIPTextModel,
	text_encoder_2: CLIPTextModelWithProjection,
	tokenizer: CLIPTokenizer,
	tokenizer_2: CLIPTokenizer,
	unet: UNet2DConditionModel,
	adapter: T2IAdapter | MultiAdapter | list[T2IAdapter],
	scheduler: KarrasDiffusionSchedulers,
	force_zeros_for_empty_prompt: bool = True,
	feature_extractor: CLIPImageProcessor = None,
	image_encoder: CLIPVisionModelWithProjection = None,
	):
	super().__init__()
	self.register_modules(
	vae=vae,

Problem:
StableDiffusionXLAdapterPipeline.__init__ documents and types adapter as T2IAdapter | MultiAdapter | list[T2IAdapter], but registers the raw list. register_modules then fails because a Python list has no __module__.

Impact:
SDXL is inconsistent with the SD adapter pipeline and breaks a documented constructor form.

Reproduction:

from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter

try:
 StableDiffusionXLAdapterPipeline(
 vae=None, text_encoder=None, text_encoder_2=None,
 tokenizer=None, tokenizer_2=None, unet=None, scheduler=None,
 adapter=[
 T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
 T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
 ],
 )
except Exception as e:
 print(type(e).__name__, str(e))

Relevant precedent:

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py

Lines 260 to 261 in 0f1abc4

	if isinstance(adapter, (list, tuple)):
	adapter = MultiAdapter(adapter)

Suggested fix:

if isinstance(adapter, (list, tuple)):
 adapter = MultiAdapter(adapter)

self.register_modules(
 vae=vae,
 text_encoder=text_encoder,
 text_encoder_2=text_encoder_2,
 tokenizer=tokenizer,
 tokenizer_2=tokenizer_2,
 unet=unet,
 adapter=adapter,
 scheduler=scheduler,
 feature_extractor=feature_extractor,
 image_encoder=image_encoder,
)

Issue 3: SDXL latent output returns before cleanup and ignores return_dict=False

Affected code:

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py

Lines 1278 to 1298 in 0f1abc4

	if not output_type == "latent":
	# make sure the VAE is in float32 mode, as it overflows in float16
	needs_upcasting = self.vae.dtype == torch.float16 and self.vae.config.force_upcast
	if needs_upcasting:
	self.upcast_vae()
	latents = latents.to(next(iter(self.vae.post_quant_conv.parameters())).dtype)
	image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dict=False)[0]
	# cast back to fp16 if needed
	if needs_upcasting:
	self.vae.to(dtype=torch.float16)
	else:
	image = latents
	return StableDiffusionXLPipelineOutput(images=image)
	image = self.image_processor.postprocess(image, output_type=output_type)
	# Offload all models
	self.maybe_free_model_hooks()

Problem:
For output_type="latent", StableDiffusionXLAdapterPipeline.__call__ returns immediately, before maybe_free_model_hooks() and before the return_dict handling.

Impact:
Model offload hooks are not released on latent output, and return_dict=False still returns StableDiffusionXLPipelineOutput.

Reproduction:

import types
import torch
from diffusers import AutoencoderKL, EulerDiscreteScheduler, StableDiffusionXLAdapterPipeline, T2IAdapter, UNet2DConditionModel

unet = UNet2DConditionModel(
 block_out_channels=(32, 64), layers_per_block=1, sample_size=32,
 in_channels=4, out_channels=4,
 down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
 up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
 attention_head_dim=(2, 4), use_linear_projection=True,
 addition_embed_type="text_time", addition_time_embed_dim=8,
 transformer_layers_per_block=(1, 1),
 projection_class_embeddings_input_dim=80, cross_attention_dim=64,
)
vae = AutoencoderKL(
 block_out_channels=[32, 64], in_channels=3, out_channels=3,
 down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
 up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], latent_channels=4,
)
pipe = StableDiffusionXLAdapterPipeline(
 vae=vae, text_encoder=None, text_encoder_2=None, tokenizer=None, tokenizer_2=None,
 unet=unet,
 adapter=T2IAdapter(in_channels=3, channels=[32, 64], num_res_blocks=1, downscale_factor=4, adapter_type="full_adapter_xl"),
 scheduler=EulerDiscreteScheduler(),
)
pipe.set_progress_bar_config(disable=True)
pipe.freed = False
pipe.maybe_free_model_hooks = types.MethodType(lambda self: setattr(self, "freed", True), pipe)

out = pipe(
 prompt_embeds=torch.zeros(1, 2, 64),
 negative_prompt_embeds=torch.zeros(1, 2, 64),
 pooled_prompt_embeds=torch.zeros(1, 32),
 negative_pooled_prompt_embeds=torch.zeros(1, 32),
 image=torch.zeros(1, 3, 64, 64),
 num_inference_steps=1,
 guidance_scale=1.0,
 output_type="latent",
 return_dict=False,
)
print(type(out).__name__, pipe.freed)

Relevant precedent:

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py

Lines 1287 to 1300 in 0f1abc4

	if not output_type == "latent":
	# apply watermark if available
	if self.watermark is not None:
	image = self.watermark.apply_watermark(image)
	image = self.image_processor.postprocess(image, output_type=output_type)
	# Offload all models
	self.maybe_free_model_hooks()
	if not return_dict:
	return (image,)
	return StableDiffusionXLPipelineOutput(images=image)

Suggested fix:

else:
 image = latents

if not output_type == "latent":
 image = self.image_processor.postprocess(image, output_type=output_type)

self.maybe_free_model_hooks()

if not return_dict:
 return (image,)

return StableDiffusionXLPipelineOutput(images=image)

Issue 4: MultiAdapter.save_pretrained and from_pretrained reject PathLike

Affected code:

diffusers/src/diffusers/models/adapter.py

Lines 130 to 145 in 0f1abc4

	variant (`str`, *optional*):
	If specified, weights are saved in the format `pytorch_model.<variant>.bin`.
	"""
	idx = 0
	model_path_to_save = save_directory
	for adapter in self.adapters:
	adapter.save_pretrained(
	model_path_to_save,
	is_main_process=is_main_process,
	save_function=save_function,
	safe_serialization=safe_serialization,
	variant=variant,
	)
	idx += 1
	model_path_to_save = model_path_to_save + f"_{idx}"

diffusers/src/diffusers/models/adapter.py

Lines 196 to 204 in 0f1abc4

	# first adapter has to be saved under `./mydirectory/adapter` to be compliant with `DiffusionPipeline.from_pretrained`
	# second, third, ... adapters have to be saved under `./mydirectory/adapter_1`, `./mydirectory/adapter_2`, ...
	model_path_to_load = pretrained_model_path
	while os.path.isdir(model_path_to_load):
	adapter = T2IAdapter.from_pretrained(model_path_to_load, **kwargs)
	adapters.append(adapter)
	idx += 1
	model_path_to_load = pretrained_model_path + f"_{idx}"

Problem:
The signatures accept str | os.PathLike, but the implementation concatenates paths with + f"_{idx}", which fails for pathlib.Path.

Impact:
Serialization/loading works with strings but fails with standard path objects.

Reproduction:

from pathlib import Path
import tempfile
from diffusers import MultiAdapter, T2IAdapter

multi = MultiAdapter([
 T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
 T2IAdapter(in_channels=3, channels=[4], num_res_blocks=1, downscale_factor=2),
])

with tempfile.TemporaryDirectory() as d:
 try:
 multi.save_pretrained(Path(d) / "adapter")
 except Exception as e:
 print("save:", type(e).__name__, str(e))

with tempfile.TemporaryDirectory() as d:
 path = Path(d) / "adapter"
 multi.save_pretrained(str(path))
 try:
 MultiAdapter.from_pretrained(path)
 except Exception as e:
 print("load:", type(e).__name__, str(e))

Relevant precedent:
T2IAdapter inherits the normal ModelMixin path handling; this custom override should preserve the same public contract.

Suggested fix:

save_directory = os.fspath(save_directory)
...
model_path_to_save = f"{save_directory}_{idx}"

pretrained_model_path = os.fspath(pretrained_model_path)
...
model_path_to_load = f"{pretrained_model_path}_{idx}"

Issue 5: SD adapter has dead LoRA/textual-inversion hooks because it does not inherit the loader mixins

Affected code:

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py

Line 25 in 0f1abc4

from ...loaders import FromSingleFileMixin, StableDiffusionLoraLoaderMixin, TextualInversionLoaderMixin

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py

Line 191 in 0f1abc4

class StableDiffusionAdapterPipeline(DiffusionPipeline, StableDiffusionMixin, FromSingleFileMixin):

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py

Lines 354 to 372 in 0f1abc4

	if lora_scale is not None and isinstance(self, StableDiffusionLoraLoaderMixin):
	self._lora_scale = lora_scale
	# dynamically adjust the LoRA scale
	if not USE_PEFT_BACKEND:
	adjust_lora_scale_text_encoder(self.text_encoder, lora_scale)
	else:
	scale_lora_layers(self.text_encoder, lora_scale)
	if prompt is not None and isinstance(prompt, str):
	batch_size = 1
	elif prompt is not None and isinstance(prompt, list):
	batch_size = len(prompt)
	else:
	batch_size = prompt_embeds.shape[0]
	if prompt_embeds is None:
	# textual inversion: process multi-vector tokens if necessary
	if isinstance(self, TextualInversionLoaderMixin):

Problem:
StableDiffusionAdapterPipeline imports StableDiffusionLoraLoaderMixin and TextualInversionLoaderMixin, and encode_prompt checks for them, but the class does not inherit either mixin.

Impact:
StableDiffusionAdapterPipeline cannot load LoRA or textual inversion, unlike StableDiffusionPipeline and StableDiffusionXLAdapterPipeline.

Reproduction:

from diffusers import StableDiffusionAdapterPipeline, StableDiffusionPipeline, StableDiffusionXLAdapterPipeline

for cls in [StableDiffusionPipeline, StableDiffusionAdapterPipeline, StableDiffusionXLAdapterPipeline]:
 print(cls.__name__, hasattr(cls, "load_lora_weights"), hasattr(cls, "load_textual_inversion"))

Relevant precedent:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Lines 154 to 160 in 0f1abc4

	class StableDiffusionPipeline(
	DiffusionPipeline,
	StableDiffusionMixin,
	TextualInversionLoaderMixin,
	StableDiffusionLoraLoaderMixin,
	IPAdapterMixin,
	FromSingleFileMixin,

diffusers/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py

Lines 213 to 220 in 0f1abc4

	class StableDiffusionXLAdapterPipeline(
	DiffusionPipeline,
	StableDiffusionMixin,
	TextualInversionLoaderMixin,
	StableDiffusionXLLoraLoaderMixin,
	IPAdapterMixin,
	FromSingleFileMixin,
	):

Suggested fix:

class StableDiffusionAdapterPipeline(
 DiffusionPipeline,
 StableDiffusionMixin,
 TextualInversionLoaderMixin,
 StableDiffusionLoraLoaderMixin,
 FromSingleFileMixin,
):
 ...

Issue 6: T2I-Adapter docs import a nonexistent scheduler class

Affected code:

diffusers/examples/t2i_adapter/README_sdxl.md

Lines 97 to 110 in 0f1abc4

	from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteSchedulerTest
	from diffusers.utils import load_image
	import torch
	base_model_path = "stabilityai/stable-diffusion-xl-base-1.0"
	adapter_path = "path to adapter"
	adapter = T2IAdapter.from_pretrained(adapter_path, torch_dtype=torch.float16)
	pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
	base_model_path, adapter=adapter, torch_dtype=torch.float16
	)
	# speed up diffusion process with faster scheduler and memory optimization
	pipe.scheduler = EulerAncestralDiscreteSchedulerTest.from_config(pipe.scheduler.config)

diffusers/docs/source/en/training/t2i_adapters.md

Lines 191 to 200 in 0f1abc4

	from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteSchedulerTest
	from diffusers.utils import load_image
	import torch
	adapter = T2IAdapter.from_pretrained("path/to/adapter", torch_dtype=torch.float16)
	pipeline = StableDiffusionXLAdapterPipeline.from_pretrained(
	"stabilityai/stable-diffusion-xl-base-1.0", adapter=adapter, torch_dtype=torch.float16
	)
	pipeline.scheduler = EulerAncestralDiscreteSchedulerTest.from_config(pipe.scheduler.config)

Problem:
The inference snippets import EulerAncestralDiscreteSchedulerTest, which is not exported. The training docs also assign from pipe.scheduler.config while the variable is named pipeline.

Impact:
Users following the example hit an immediate import/name error.

Reproduction:

try:
 from diffusers import EulerAncestralDiscreteSchedulerTest
except Exception as e:
 print(type(e).__name__, str(e))

Relevant precedent:
Use the public scheduler class exported by diffusers.

Suggested fix:

from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler

...
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(pipeline.scheduler.config)

Issue 7: SDXL adapter lacks a plain slow golden test in its pipeline test file

Affected code:

diffusers/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py

Line 52 in 0f1abc4

class StableDiffusionXLAdapterPipelineFastTests(IPAdapterTesterMixin, PipelineTesterMixin, unittest.TestCase):

diffusers/tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py

Lines 607 to 609 in 0f1abc4

	@slow
	@require_torch_accelerator
	class StableDiffusionAdapterPipelineSlowTests(unittest.TestCase):

Problem:
Fast SDXL adapter tests exist, and there are SDXL adapter slow paths in single-file and LoRA integration tests, but tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py has no plain slow golden inference test for the default SDXL adapter pipeline.

Impact:
Core SDXL adapter behavior can regress without a direct slow pipeline fixture. The output_type="latent" return bug and constructor/list handling are not covered by existing slow SDXL adapter tests.

Reproduction:

from pathlib import Path

sdxl_test = Path("tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py").read_text()
sd_test = Path("tests/pipelines/stable_diffusion_adapter/test_stable_diffusion_adapter.py").read_text()

print("@slow in SDXL adapter pipeline test:", "@slow" in sdxl_test)
print("@slow in SD adapter pipeline test:", "@slow" in sd_test)

Relevant precedent:
The SD adapter pipeline has a dedicated slow class with real adapter checkpoints and expected arrays.

Suggested fix:
Add a @slow SDXL adapter pipeline regression test in tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py, using an hf-internal-testing image and a stable expected array under datasets/diffusers/test-arrays, covering at least normal inference and output_type="latent", return_dict=False.