controlnet model/pipeline review

[hlky]

controlnet model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Issue 1: MultiControlNetUnionModel is missing from top-level exports

Affected code:

diffusers/src/diffusers/__init__.py

Lines 229 to 269 in 0f1abc4

	"ConsistencyDecoderVAE",
	"ContextParallelConfig",
	"ControlNetModel",
	"ControlNetUnionModel",
	"ControlNetXSAdapter",
	"CosmosControlNetModel",
	"CosmosTransformer3DModel",
	"DiTTransformer2DModel",
	"EasyAnimateTransformer3DModel",
	"ErnieImageTransformer2DModel",
	"Flux2Transformer2DModel",
	"FluxControlNetModel",
	"FluxMultiControlNetModel",
	"FluxTransformer2DModel",
	"GlmImageTransformer2DModel",
	"HeliosTransformer3DModel",
	"HiDreamImageTransformer2DModel",
	"HunyuanDiT2DControlNetModel",
	"HunyuanDiT2DModel",
	"HunyuanDiT2DMultiControlNetModel",
	"HunyuanImageTransformer2DModel",
	"HunyuanVideo15Transformer3DModel",
	"HunyuanVideoFramepackTransformer3DModel",
	"HunyuanVideoTransformer3DModel",
	"I2VGenXLUNet",
	"Kandinsky3UNet",
	"Kandinsky5Transformer3DModel",
	"LatteTransformer3DModel",
	"LongCatAudioDiTTransformer",
	"LongCatAudioDiTVae",
	"LongCatImageTransformer2DModel",
	"LTX2VideoTransformer3DModel",
	"LTXVideoTransformer3DModel",
	"Lumina2Transformer2DModel",
	"LuminaNextDiT2DModel",
	"MochiTransformer3DModel",
	"ModelMixin",
	"MotionAdapter",
	"MultiAdapter",
	"MultiControlNetModel",
	"NucleusMoEImageTransformer2DModel",

diffusers/src/diffusers/models/__init__.py

Lines 75 to 79 in 0f1abc4

	_import_structure["controlnets.controlnet_union"] = ["ControlNetUnionModel"]
	_import_structure["controlnets.controlnet_xs"] = ["ControlNetXSAdapter", "UNetControlNetXSModel"]
	_import_structure["controlnets.controlnet_z_image"] = ["ZImageControlNetModel"]
	_import_structure["controlnets.multicontrolnet"] = ["MultiControlNetModel"]
	_import_structure["controlnets.multicontrolnet_union"] = ["MultiControlNetUnionModel"]

Problem:
MultiControlNetUnionModel is exported from diffusers.models, but not from diffusers, while adjacent public ControlNet classes are top-level exports.

Impact:
Users cannot follow the standard from diffusers import ... pattern for this public wrapper.

Reproduction:

from diffusers import ControlNetUnionModel, MultiControlNetModel
print(ControlNetUnionModel, MultiControlNetModel)

from diffusers import MultiControlNetUnionModel

Relevant precedent:
MultiControlNetModel is top-level exported.

Suggested fix:

# src/diffusers/__init__.py
# Add "MultiControlNetUnionModel" next to "MultiControlNetModel"
# in both the lazy _import_structure["models"] list and TYPE_CHECKING imports.

Issue 2: ControlNetUnionModel() default constructor crashes

Affected code:

diffusers/src/diffusers/models/controlnets/controlnet_union.py

Line 182 in 0f1abc4

addition_time_embed_dim: int | None = None,

diffusers/src/diffusers/models/controlnets/controlnet_union.py

Lines 307 to 308 in 0f1abc4

	self.control_type_proj = Timesteps(addition_time_embed_dim, flip_sin_to_cos, freq_shift)
	self.control_add_embedding = TimestepEmbedding(addition_time_embed_dim * num_control_type, time_embed_dim)

Problem:
addition_time_embed_dim defaults to None, but the constructor always uses it to create Timesteps and TimestepEmbedding.

Impact:
A documented public model constructor fails before forward or serialization can be tested.

Reproduction:

from diffusers import ControlNetUnionModel

ControlNetUnionModel(
 in_channels=4,
 conditioning_channels=3,
 down_block_types=("DownBlock2D",),
 block_out_channels=(8,),
 layers_per_block=1,
 norm_num_groups=4,
 cross_attention_dim=16,
 attention_head_dim=1,
 num_trans_channel=8,
 num_trans_head=1,
 num_proj_channel=8,
 conditioning_embedding_out_channels=(4, 8),
)

Relevant precedent:
ControlNetModel only constructs add_time_proj when the matching addition embedding mode requires it.

Suggested fix:

if addition_time_embed_dim is None:
 raise ValueError("`addition_time_embed_dim` must be set for `ControlNetUnionModel`.")

Issue 3: Union pipelines advertise control_mode=None but crash

Affected code:

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py

Line 1007 in 0f1abc4

control_mode: int | list[int] | list[list[int]] | None = None,

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py

Lines 1184 to 1189 in 0f1abc4

	if not isinstance(control_mode, list):
	control_mode = [control_mode]
	if isinstance(controlnet, MultiControlNetUnionModel):
	control_image = [[item] for item in control_image]
	control_mode = [[item] for item in control_mode]

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py

Lines 793 to 805 in 0f1abc4

	# Check `control_mode`
	if isinstance(controlnet, ControlNetUnionModel):
	if max(control_mode) >= controlnet.config.num_control_type:
	raise ValueError(f"control_mode: must be lower than {controlnet.config.num_control_type}.")
	elif isinstance(controlnet, MultiControlNetUnionModel):
	for _control_mode, _controlnet in zip(control_mode, self.controlnet.nets):
	if max(_control_mode) >= _controlnet.config.num_control_type:
	raise ValueError(f"control_mode: must be lower than {_controlnet.config.num_control_type}.")
	# Equal number of `image` and `control_mode` elements
	if isinstance(controlnet, ControlNetUnionModel):
	if len(image) != len(control_mode):
	raise ValueError("Expected len(control_image) == len(control_mode)")

Problem:
The Union pipelines default control_mode to None, wrap it into [None], then compare None >= num_control_type.

Impact:
Calling a Union pipeline without explicitly passing control_mode fails with a TypeError.

Reproduction:

from types import MethodType
from diffusers import ControlNetUnionModel, StableDiffusionXLControlNetUnionPipeline

pipe = object.__new__(StableDiffusionXLControlNetUnionPipeline)
pipe._callback_tensor_inputs = []
pipe.check_image = MethodType(lambda self, image, prompt, prompt_embeds: None, pipe)
pipe.controlnet = ControlNetUnionModel(
 in_channels=4,
 conditioning_channels=3,
 down_block_types=("DownBlock2D",),
 block_out_channels=(8,),
 layers_per_block=1,
 norm_num_groups=4,
 cross_attention_dim=16,
 attention_head_dim=1,
 addition_time_embed_dim=8,
 num_trans_channel=8,
 num_trans_head=1,
 num_proj_channel=8,
 conditioning_embedding_out_channels=(4, 8),
)

control_mode = None
if not isinstance(control_mode, list):
 control_mode = [control_mode]

pipe.check_inputs(
 prompt="a prompt",
 prompt_2=None,
 image=[object()],
 control_guidance_start=[0.0],
 control_guidance_end=[1.0],
 control_mode=control_mode,
 callback_on_step_end_tensor_inputs=[],
)

Relevant precedent:
Merged PR #10747 added multi-union handling but did not make None a valid default.

Suggested fix:

if control_mode is None:
 control_mode = 0
if not isinstance(control_mode, list):
 control_mode = [control_mode]

Issue 4: Multi-ControlNet scale length validation is unreachable or missing

Affected code:

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet.py

Lines 701 to 712 in 0f1abc4

	if isinstance(controlnet_conditioning_scale, list):
	if any(isinstance(i, list) for i in controlnet_conditioning_scale):
	raise ValueError(
	"A single batch of varying conditioning scale settings (e.g. [[1.0, 0.5], [0.2, 0.8]]) is not supported at the moment. "
	"The conditioning scale must be fixed across the batch."
	)
	elif isinstance(controlnet_conditioning_scale, list) and len(controlnet_conditioning_scale) != len(
	self.controlnet.nets
	):
	raise ValueError(
	"For multiple controlnets: When `controlnet_conditioning_scale` is specified as `list`, it must have"
	" the same length as the number of controlnets"

diffusers/src/diffusers/models/controlnets/multicontrolnet.py

Lines 47 to 63 in 0f1abc4

	for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)):
	down_samples, mid_sample = controlnet(
	sample=sample,
	timestep=timestep,
	encoder_hidden_states=encoder_hidden_states,
	controlnet_cond=image,
	conditioning_scale=scale,
	class_labels=class_labels,
	timestep_cond=timestep_cond,
	attention_mask=attention_mask,
	added_cond_kwargs=added_cond_kwargs,
	cross_attention_kwargs=cross_attention_kwargs,
	guess_mode=guess_mode,
	return_dict=return_dict,
	)
	# merge samples

diffusers/src/diffusers/models/controlnets/controlnet_union.py

Line 691 in 0f1abc4

for cond, control_idx, scale in zip(controlnet_cond, control_type_idx, conditioning_scale):

Problem:
The elif isinstance(controlnet_conditioning_scale, list) length check is unreachable after the preceding if isinstance(..., list). In Union variants, some multi-condition paths have no equivalent scale length check.

Impact:
A too-short scale list passes validation, and later zip(...) silently drops later ControlNets or later Union conditions.

Reproduction:

from types import MethodType
import torch
from diffusers import MultiControlNetModel, StableDiffusionControlNetPipeline

class DummyControlNet(torch.nn.Module):
 pass

pipe = object.__new__(StableDiffusionControlNetPipeline)
pipe._callback_tensor_inputs = []
pipe.controlnet = MultiControlNetModel([DummyControlNet(), DummyControlNet()])
pipe.check_image = MethodType(lambda self, image, prompt, prompt_embeds: None, pipe)

pipe.check_inputs(
 prompt="a prompt",
 image=[object(), object()],
 callback_steps=None,
 callback_on_step_end_tensor_inputs=[],
 controlnet_conditioning_scale=[0.5],
 control_guidance_start=[0.0, 0.0],
 control_guidance_end=[1.0, 1.0],
)
print("No error, but one scale for two ControlNets should be rejected.")

Relevant precedent:
Issue #11828 is related to Union scale/list acceptance, but not this silent truncation.

Suggested fix:

if isinstance(controlnet_conditioning_scale, list):
 if any(isinstance(i, list) for i in controlnet_conditioning_scale):
 raise ValueError("Batched varying conditioning scales are not supported.")
 if len(controlnet_conditioning_scale) != len(self.controlnet.nets):
 raise ValueError("Scale list length must match the number of ControlNets.")

Issue 5: All-zero MultiControlNetUnionModel scales return None

Affected code:

diffusers/src/diffusers/models/controlnets/multicontrolnet_union.py

Lines 52 to 83 in 0f1abc4

	zip(controlnet_cond, control_type, control_type_idx, conditioning_scale, self.nets)
	):
	if scale == 0.0:
	continue
	down_samples, mid_sample = controlnet(
	sample=sample,
	timestep=timestep,
	encoder_hidden_states=encoder_hidden_states,
	controlnet_cond=image,
	control_type=ctype,
	control_type_idx=ctype_idx,
	conditioning_scale=scale,
	class_labels=class_labels,
	timestep_cond=timestep_cond,
	attention_mask=attention_mask,
	added_cond_kwargs=added_cond_kwargs,
	cross_attention_kwargs=cross_attention_kwargs,
	from_multi=True,
	guess_mode=guess_mode,
	return_dict=return_dict,
	)
	# merge samples
	if down_block_res_samples is None and mid_block_res_sample is None:
	down_block_res_samples, mid_block_res_sample = down_samples, mid_sample
	else:
	down_block_res_samples = [
	samples_prev + samples_curr
	for samples_prev, samples_curr in zip(down_block_res_samples, down_samples)
	]
	mid_block_res_sample += mid_sample

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py

Lines 1518 to 1522 in 0f1abc4

	if guess_mode and self.do_classifier_free_guidance:
	# Inferred ControlNet only for the conditional batch.
	# To apply the output of ControlNet to both the unconditional and conditional batches,
	# add 0 to the unconditional batch to keep it unchanged.
	down_block_res_samples = [torch.cat([torch.zeros_like(d), d]) for d in down_block_res_samples]

Problem:
MultiControlNetUnionModel.forward() skips every ControlNet whose scale is 0.0. If all scales are zero, it returns (None, None). Union pipelines then iterate over down_block_res_samples in guess-mode CFG.

Impact:
Valid schedules such as control_guidance_start/end outside a step, or explicit controlnet_conditioning_scale=[0.0], can crash in guess mode.

Reproduction:

import torch
from diffusers.models import MultiControlNetUnionModel

class DummyUnion(torch.nn.Module):
 config = type("Config", (), {"num_control_type": 6})()

 def forward(self, *args, **kwargs):
 return [torch.ones(1, 1, 1, 1)], torch.ones(1, 1, 1, 1)

multi = MultiControlNetUnionModel([DummyUnion()])
down, mid = multi(
 sample=torch.zeros(1, 4, 8, 8),
 timestep=0,
 encoder_hidden_states=torch.zeros(1, 1, 4),
 controlnet_cond=[torch.zeros(1, 3, 8, 8)],
 control_type=[torch.zeros(1, 6)],
 control_type_idx=[[0]],
 conditioning_scale=[0.0],
 return_dict=False,
)

[torch.cat([torch.zeros_like(d), d]) for d in down]

Relevant precedent:
MultiControlNetModel does not skip zero scales; it lets the child model return zeroed residual tensors.

Suggested fix:

# Do not skip zero scales. Let the child ControlNet return correctly shaped zero residuals.
# Remove:
if scale == 0.0:
 continue

Issue 6: ControlNetUnionModel rejects documented bgr channel order

Affected code:

diffusers/src/diffusers/models/controlnets/controlnet_union.py

Lines 143 to 145 in 0f1abc4

	controlnet_conditioning_channel_order (`str`, defaults to `"rgb"`):
	The channel order of conditional image. Will convert to `rgb` if it's `bgr`.
	conditioning_embedding_out_channels (`tuple[int]`, *optional*, defaults to `(48, 96, 192, 384)`):

diffusers/src/diffusers/models/controlnets/controlnet_union.py

Lines 608 to 611 in 0f1abc4

	channel_order = self.config.controlnet_conditioning_channel_order
	if channel_order != "rgb":
	raise ValueError(f"unknown `controlnet_conditioning_channel_order`: {channel_order}")

Problem:
The config/docstring exposes controlnet_conditioning_channel_order, but Union forward only accepts "rgb" and raises for "bgr".

Impact:
Union behaves inconsistently with ControlNetModel and rejects a documented compatibility mode.

Reproduction:

import torch
from diffusers import ControlNetUnionModel

model = ControlNetUnionModel(
 in_channels=4,
 conditioning_channels=3,
 down_block_types=("DownBlock2D",),
 block_out_channels=(8,),
 layers_per_block=1,
 norm_num_groups=4,
 cross_attention_dim=16,
 attention_head_dim=1,
 addition_time_embed_dim=8,
 num_trans_channel=8,
 num_trans_head=1,
 num_proj_channel=8,
 conditioning_embedding_out_channels=(4, 8),
 controlnet_conditioning_channel_order="bgr",
)

model(
 sample=torch.randn(1, 4, 8, 8),
 timestep=0,
 encoder_hidden_states=torch.randn(1, 2, 16),
 controlnet_cond=[torch.randn(1, 3, 8, 8)],
 control_type=torch.zeros(1, 6),
 control_type_idx=[0],
 return_dict=False,
)

Relevant precedent:

diffusers/src/diffusers/models/controlnets/controlnet.py

Lines 654 to 664 in 0f1abc4

	If `return_dict` is `True`, a [`~models.controlnets.controlnet.ControlNetOutput`] is returned,
	otherwise a tuple is returned where the first element is the sample tensor.
	"""
	# check channel order
	channel_order = self.config.controlnet_conditioning_channel_order
	if channel_order == "rgb":
	# in rgb order by default
	...
	elif channel_order == "bgr":
	controlnet_cond = torch.flip(controlnet_cond, dims=[1])

Suggested fix:

if channel_order == "rgb":
 pass
elif channel_order == "bgr":
 controlnet_cond = [torch.flip(cond, dims=[1]) for cond in controlnet_cond]
else:
 raise ValueError(f"unknown `controlnet_conditioning_channel_order`: {channel_order}")

Issue 7: Multi-ControlNet wrappers diverge from public API contracts

Affected code:

diffusers/src/diffusers/models/controlnets/multicontrolnet.py

Lines 37 to 73 in 0f1abc4

	controlnet_cond: list[torch.tensor],
	conditioning_scale: list[float],
	class_labels: torch.Tensor | None = None,
	timestep_cond: torch.Tensor | None = None,
	attention_mask: torch.Tensor | None = None,
	added_cond_kwargs: dict[str, torch.Tensor] | None = None,
	cross_attention_kwargs: dict[str, Any] | None = None,
	guess_mode: bool = False,
	return_dict: bool = True,
	) -> ControlNetOutput | tuple:
	for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)):
	down_samples, mid_sample = controlnet(
	sample=sample,
	timestep=timestep,
	encoder_hidden_states=encoder_hidden_states,
	controlnet_cond=image,
	conditioning_scale=scale,
	class_labels=class_labels,
	timestep_cond=timestep_cond,
	attention_mask=attention_mask,
	added_cond_kwargs=added_cond_kwargs,
	cross_attention_kwargs=cross_attention_kwargs,
	guess_mode=guess_mode,
	return_dict=return_dict,
	)
	# merge samples
	if i == 0:
	down_block_res_samples, mid_block_res_sample = down_samples, mid_sample
	else:
	down_block_res_samples = [
	samples_prev + samples_curr
	for samples_prev, samples_curr in zip(down_block_res_samples, down_samples)
	]
	mid_block_res_sample += mid_sample
	return down_block_res_samples, mid_block_res_sample

diffusers/src/diffusers/models/controlnets/multicontrolnet.py

Lines 75 to 176 in 0f1abc4

	def save_pretrained(
	self,
	save_directory: str | os.PathLike,
	is_main_process: bool = True,
	save_function: Callable = None,
	safe_serialization: bool = True,
	variant: str | None = None,
	):
	"""
	Save a model and its configuration file to a directory, so that it can be re-loaded using the
	`[`~models.controlnets.multicontrolnet.MultiControlNetModel.from_pretrained`]` class method.
	Arguments:
	save_directory (`str` or `os.PathLike`):
	Directory to which to save. Will be created if it doesn't exist.
	is_main_process (`bool`, *optional*, defaults to `True`):
	Whether the process calling this is the main process or not. Useful when in distributed training like
	TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on
	the main process to avoid race conditions.
	save_function (`Callable`):
	The function to use to save the state dictionary. Useful on distributed training like TPUs when one
	need to replace `torch.save` by another method. Can be configured with the environment variable
	`DIFFUSERS_SAVE_MODE`.
	safe_serialization (`bool`, *optional*, defaults to `True`):
	Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
	variant (`str`, *optional*):
	If specified, weights are saved in the format pytorch_model.<variant>.bin.
	"""
	for idx, controlnet in enumerate(self.nets):
	suffix = "" if idx == 0 else f"_{idx}"
	controlnet.save_pretrained(
	save_directory + suffix,
	is_main_process=is_main_process,
	save_function=save_function,
	safe_serialization=safe_serialization,
	variant=variant,
	)
	@classmethod
	def from_pretrained(cls, pretrained_model_path: str | os.PathLike | None, **kwargs):
	r"""
	Instantiate a pretrained MultiControlNet model from multiple pre-trained controlnet models.
	The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train
	the model, you should first set it back in training mode with `model.train()`.
	The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come
	pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
	task.
	The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those
	weights are discarded.
	Parameters:
	pretrained_model_path (`os.PathLike`):
	A path to a *directory* containing model weights saved using
	[`~models.controlnets.multicontrolnet.MultiControlNetModel.save_pretrained`], e.g.,
	`./my_model_directory/controlnet`.
	torch_dtype (`torch.dtype`, *optional*):
	Override the default `torch.dtype` and load the model under this dtype.
	output_loading_info(`bool`, *optional*, defaults to `False`):
	Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages.
	device_map (`str` or `dict[str, int | str | torch.device]`, *optional*):
	A map that specifies where each submodule should go. It doesn't need to be refined to each
	parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
	same device.
	To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
	more information about each option see [designing a device
	map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
	max_memory (`Dict`, *optional*):
	A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
	GPU and the available CPU RAM if unset.
	low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`):
	Speed up model loading by not initializing the weights and only loading the pre-trained weights. This
	also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the
	model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,
	setting this argument to `True` will raise an error.
	variant (`str`, *optional*):
	If specified load weights from `variant` filename, *e.g.* pytorch_model.<variant>.bin. `variant` is
	ignored when using `from_flax`.
	use_safetensors (`bool`, *optional*, defaults to `None`):
	If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the
	`safetensors` library is installed. If set to `True`, the model will be forcibly loaded from
	`safetensors` weights. If set to `False`, loading will *not* use `safetensors`.
	"""
	idx = 0
	controlnets = []
	# load controlnet and append to list until no controlnet directory exists anymore
	# first controlnet has to be saved under `./mydirectory/controlnet` to be compliant with `DiffusionPipeline.from_prertained`
	# second, third, ... controlnets have to be saved under `./mydirectory/controlnet_1`, `./mydirectory/controlnet_2`, ...
	model_path_to_load = pretrained_model_path
	while os.path.isdir(model_path_to_load):
	controlnet = ControlNetModel.from_pretrained(model_path_to_load, **kwargs)
	controlnets.append(controlnet)
	idx += 1
	model_path_to_load = pretrained_model_path + f"_{idx}"
	logger.info(f"{len(controlnets)} controlnets loaded from {pretrained_model_path}.")

diffusers/src/diffusers/models/controlnets/multicontrolnet_union.py

Lines 47 to 83 in 0f1abc4

	guess_mode: bool = False,
	return_dict: bool = True,
	) -> ControlNetOutput | tuple:
	down_block_res_samples, mid_block_res_sample = None, None
	for i, (image, ctype, ctype_idx, scale, controlnet) in enumerate(
	zip(controlnet_cond, control_type, control_type_idx, conditioning_scale, self.nets)
	):
	if scale == 0.0:
	continue
	down_samples, mid_sample = controlnet(
	sample=sample,
	timestep=timestep,
	encoder_hidden_states=encoder_hidden_states,
	controlnet_cond=image,
	control_type=ctype,
	control_type_idx=ctype_idx,
	conditioning_scale=scale,
	class_labels=class_labels,
	timestep_cond=timestep_cond,
	attention_mask=attention_mask,
	added_cond_kwargs=added_cond_kwargs,
	cross_attention_kwargs=cross_attention_kwargs,
	from_multi=True,
	guess_mode=guess_mode,
	return_dict=return_dict,
	)
	# merge samples
	if down_block_res_samples is None and mid_block_res_sample is None:
	down_block_res_samples, mid_block_res_sample = down_samples, mid_sample
	else:
	down_block_res_samples = [
	samples_prev + samples_curr
	for samples_prev, samples_curr in zip(down_block_res_samples, down_samples)
	]
	mid_block_res_sample += mid_sample

diffusers/src/diffusers/models/controlnets/multicontrolnet_union.py

Lines 86 to 192 in 0f1abc4

	# Copied from diffusers.models.controlnets.multicontrolnet.MultiControlNetModel.save_pretrained with ControlNet->ControlNetUnion
	def save_pretrained(
	self,
	save_directory: str | os.PathLike,
	is_main_process: bool = True,
	save_function: Callable = None,
	safe_serialization: bool = True,
	variant: str | None = None,
	):
	"""
	Save a model and its configuration file to a directory, so that it can be re-loaded using the
	`[`~models.controlnets.multicontrolnet.MultiControlNetUnionModel.from_pretrained`]` class method.
	Arguments:
	save_directory (`str` or `os.PathLike`):
	Directory to which to save. Will be created if it doesn't exist.
	is_main_process (`bool`, *optional*, defaults to `True`):
	Whether the process calling this is the main process or not. Useful when in distributed training like
	TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on
	the main process to avoid race conditions.
	save_function (`Callable`):
	The function to use to save the state dictionary. Useful on distributed training like TPUs when one
	need to replace `torch.save` by another method. Can be configured with the environment variable
	`DIFFUSERS_SAVE_MODE`.
	safe_serialization (`bool`, *optional*, defaults to `True`):
	Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
	variant (`str`, *optional*):
	If specified, weights are saved in the format pytorch_model.<variant>.bin.
	"""
	for idx, controlnet in enumerate(self.nets):
	suffix = "" if idx == 0 else f"_{idx}"
	controlnet.save_pretrained(
	save_directory + suffix,
	is_main_process=is_main_process,
	save_function=save_function,
	safe_serialization=safe_serialization,
	variant=variant,
	)
	@classmethod
	# Copied from diffusers.models.controlnets.multicontrolnet.MultiControlNetModel.from_pretrained with ControlNet->ControlNetUnion
	def from_pretrained(cls, pretrained_model_path: str | os.PathLike | None, **kwargs):
	r"""
	Instantiate a pretrained MultiControlNetUnion model from multiple pre-trained controlnet models.
	The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train
	the model, you should first set it back in training mode with `model.train()`.
	The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come
	pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
	task.
	The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those
	weights are discarded.
	Parameters:
	pretrained_model_path (`os.PathLike`):
	A path to a *directory* containing model weights saved using
	[`~models.controlnets.multicontrolnet.MultiControlNetUnionModel.save_pretrained`], e.g.,
	`./my_model_directory/controlnet`.
	torch_dtype (`torch.dtype`, *optional*):
	Override the default `torch.dtype` and load the model under this dtype.
	output_loading_info(`bool`, *optional*, defaults to `False`):
	Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages.
	device_map (`str` or `dict[str, int | str | torch.device]`, *optional*):
	A map that specifies where each submodule should go. It doesn't need to be refined to each
	parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
	same device.
	To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
	more information about each option see [designing a device
	map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
	max_memory (`Dict`, *optional*):
	A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
	GPU and the available CPU RAM if unset.
	low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`):
	Speed up model loading by not initializing the weights and only loading the pre-trained weights. This
	also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the
	model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,
	setting this argument to `True` will raise an error.
	variant (`str`, *optional*):
	If specified load weights from `variant` filename, *e.g.* pytorch_model.<variant>.bin. `variant` is
	ignored when using `from_flax`.
	use_safetensors (`bool`, *optional*, defaults to `None`):
	If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the
	`safetensors` library is installed. If set to `True`, the model will be forcibly loaded from
	`safetensors` weights. If set to `False`, loading will *not* use `safetensors`.
	"""
	idx = 0
	controlnets = []
	# load controlnet and append to list until no controlnet directory exists anymore
	# first controlnet has to be saved under `./mydirectory/controlnet` to be compliant with `DiffusionPipeline.from_prertained`
	# second, third, ... controlnets have to be saved under `./mydirectory/controlnet_1`, `./mydirectory/controlnet_2`, ...
	model_path_to_load = pretrained_model_path
	while os.path.isdir(model_path_to_load):
	controlnet = ControlNetUnionModel.from_pretrained(model_path_to_load, **kwargs)
	controlnets.append(controlnet)
	idx += 1
	model_path_to_load = pretrained_model_path + f"_{idx}"
	logger.info(f"{len(controlnets)} controlnets loaded from {pretrained_model_path}.")
	if len(controlnets) == 0:
	raise ValueError(
	f"No ControlNetUnions found under {os.path.dirname(pretrained_model_path)}. Expected at least {pretrained_model_path + '_0'}."

Problem:
The wrappers accept os.PathLike but concatenate paths with strings. They also accept return_dict=True but always return tuples.

Impact:
Path users get TypeError, and direct model callers do not get the advertised ControlNetOutput.

Reproduction:

from pathlib import Path
from tempfile import TemporaryDirectory
import torch
from diffusers import MultiControlNetModel
from diffusers.models import MultiControlNetUnionModel

class DummyControlNet(torch.nn.Module):
 def save_pretrained(self, save_directory, **kwargs):
 print(save_directory)

for cls in (MultiControlNetModel, MultiControlNetUnionModel):
 with TemporaryDirectory() as tmp:
 try:
 cls([DummyControlNet()]).save_pretrained(Path(tmp) / "controlnet")
 except Exception as e:
 print(cls.__name__, type(e).__name__, e)

Relevant precedent:
Related closed issue for multi-control save layout: #7814

Suggested fix:

save_directory = os.fspath(save_directory)
...
model_path_to_load = os.fspath(pretrained_model_path)
...
if not return_dict:
 return down_block_res_samples, mid_block_res_sample
return ControlNetOutput(
 down_block_res_samples=down_block_res_samples,
 mid_block_res_sample=mid_block_res_sample,
)

Issue 8: Test coverage is missing for several target files

Affected code:

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py

Line 175 in 0f1abc4

class StableDiffusionXLControlNetUnionPipeline(

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_blip_diffusion.py

Line 85 in 0f1abc4

class BlipDiffusionControlNetPipeline(DeprecatedPipelineMixin, DiffusionPipeline):

diffusers/src/diffusers/pipelines/controlnet/pipeline_flax_controlnet.py

Line 113 in 0f1abc4

class FlaxStableDiffusionControlNetPipeline(FlaxDiffusionPipeline):

Problem:
No tests under tests/pipelines/controlnet, tests/models/controlnets, or tests/single_file reference ControlNetUnionModel, MultiControlNetUnionModel, the Union pipelines, BlipDiffusionControlNetPipeline, FlaxControlNetModel, or FlaxStableDiffusionControlNetPipeline. Slow tests are also missing for SDXL img2img and SDXL inpaint ControlNet files.

Impact:
The confirmed regressions above are not covered by fast tests, and several public or deprecated target pipelines have no slow coverage.

Reproduction:

from pathlib import Path

roots = [Path("tests/pipelines/controlnet"), Path("tests/models/controlnets"), Path("tests/single_file")]
files = [p for root in roots for p in root.rglob("*.py")]

for label, terms in {
 "Union": ["ControlNetUnionModel", "MultiControlNetUnionModel", "StableDiffusionXLControlNetUnion"],
 "BLIP": ["BlipDiffusionControlNetPipeline"],
 "Flax": ["FlaxControlNetModel", "FlaxStableDiffusionControlNetPipeline"],
}.items():
 hits = [str(p) for p in files if any(term in p.read_text(encoding="utf-8") for term in terms)]
 print(label, hits or "NO TEST REFERENCES")

for p in sorted(Path("tests/pipelines/controlnet").glob("test_controlnet*.py")):
 print(p, "@slow" in p.read_text(encoding="utf-8"))

Relevant precedent:
PR #10747 introduced MultiControlNetUnionModel; the discussion explicitly called out adding tests, but this checkout has no Union test references.

Suggested fix:
Add fast tests for Union constructor, control_mode=None, multi-union zero scales, scale length validation, top-level import, and save/load PathLike. Add slow Union pipeline tests and slow SDXL img2img/inpaint ControlNet tests. For BLIP-Diffusion ControlNet, either add deprecated-pipeline smoke coverage or document why it is intentionally untested.

Duplicate search status: searched GitHub Issues and PRs for controlnet, ControlNetUnionModel, MultiControlNetUnionModel, control_mode None, addition_time_embed_dim, controlnet_conditioning_scale, bgr, PathLike save_pretrained, BLIP, and Flax. Related items found were #10747, #11828, and #7814, but I did not find exact open duplicates for the issues above.