kandinsky2_2 model/pipeline review
Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423
Review performed against the repository review rules.
Duplicate search checked kandinsky2_2, KandinskyV22, affected class/file names, and the specific failure modes. Existing related items: #4183 covers the ControlNet guidance_scale <= 1 subset of Issue 2; #4818 is related to stale PriorEmb2Emb latents plumbing, but I did not find an exact duplicate for the current interpolate() failure.
Issue 1: PriorEmb2Emb.interpolate() cannot handle documented text entries
Affected code:
| forcond, weightinzip(images_and_prompts, weights): |
| ifisinstance(cond, str): |
| image_emb=self( |
| cond, |
| num_inference_steps=num_inference_steps, |
| num_images_per_prompt=num_images_per_prompt, |
| generator=generator, |
| latents=latents, |
| negative_prompt=negative_prior_prompt, |
| guidance_scale=guidance_scale, |
| ).image_embeds.unsqueeze(0) |
| @torch.no_grad() |
| @replace_example_docstring(EXAMPLE_DOC_STRING) |
| def__call__( |
| self, |
| prompt: str|list[str], |
| image: torch.Tensor|list[torch.Tensor] |PIL.Image.Image|list[PIL.Image.Image], |
| strength: float=0.3, |
| negative_prompt: str|list[str] |None=None, |
| num_images_per_prompt: int=1, |
| num_inference_steps: int=25, |
| generator: torch.Generator|list[torch.Generator] |None=None, |
| guidance_scale: float=4.0, |
| output_type: str|None="pt", # pt only |
| return_dict: bool=True, |
| ): |
| image_emb=torch.cat(image_embeddings).sum(dim=0) |
| |
| returnKandinskyPriorPipelineOutput(image_embeds=image_emb, negative_image_embeds=torch.randn_like(image_emb)) |
Problem:
interpolate() advertises list[str | PIL.Image.Image | torch.Tensor], but the string branch calls self(..., latents=latents) while KandinskyV22PriorEmb2EmbPipeline.__call__ has no latents parameter and requires image. The same method also returns torch.randn_like(image_emb) for negative_image_embeds, making the negative conditioning random and unrelated to negative_prompt.
Impact:
The documented mixed text/image interpolation workflow fails immediately for text entries. Image-only interpolation is nondeterministic on the negative branch and can change decoder CFG behavior across calls.
Reproduction:
fromdiffusersimportKandinskyV22PriorEmb2EmbPipelinepipe=KandinskyV22PriorEmb2EmbPipeline(
prior=None, image_encoder=None, text_encoder=None,
tokenizer=None, scheduler=None, image_processor=None,
)
try:
pipe.interpolate(["a cat"], [1.0])
exceptTypeErrorase:
print(type(e).__name__, str(e).split("\n")[0])Relevant precedent:
KandinskyV22PriorPipeline.interpolate() supports text entries by calling a compatible text-only prior path and derives the negative image embedding from a real negative/zero embedding:
| image_embeddings= [] |
| forcond, weightinzip(images_and_prompts, weights): |
| ifisinstance(cond, str): |
| image_emb=self( |
| cond, |
| num_inference_steps=num_inference_steps, |
| num_images_per_prompt=num_images_per_prompt, |
| generator=generator, |
| latents=latents, |
| negative_prompt=negative_prior_prompt, |
| guidance_scale=guidance_scale, |
| ).image_embeds.unsqueeze(0) |
| |
| elifisinstance(cond, (PIL.Image.Image, torch.Tensor)): |
| ifisinstance(cond, PIL.Image.Image): |
| cond= ( |
| self.image_processor(cond, return_tensors="pt") |
| .pixel_values[0] |
| .unsqueeze(0) |
| .to(dtype=self.image_encoder.dtype, device=device) |
| ) |
| |
| image_emb=self.image_encoder(cond)["image_embeds"].repeat(num_images_per_prompt, 1).unsqueeze(0) |
| |
| else: |
| raiseValueError( |
| f"`images_and_prompts` can only contains elements to be of type `str`, `PIL.Image.Image` or `torch.Tensor` but is {type(cond)}" |
| ) |
| |
| image_embeddings.append(image_emb*weight) |
| |
| image_emb=torch.cat(image_embeddings).sum(dim=0) |
| |
| out_zero=self( |
| negative_prompt, |
| num_inference_steps=num_inference_steps, |
| num_images_per_prompt=num_images_per_prompt, |
| generator=generator, |
| latents=latents, |
| negative_prompt=negative_prior_prompt, |
| guidance_scale=guidance_scale, |
| ) |
| zero_image_emb=out_zero.negative_image_embedsifnegative_prompt==""elseout_zero.image_embeds |
| |
| returnKandinskyPriorPipelineOutput(image_embeds=image_emb, negative_image_embeds=zero_image_emb) |
Suggested fix:
Implement the text branch by reusing/copying the text-only prior logic from KandinskyV22PriorPipeline, or narrow the accepted input types and docs to image/tensor only. Replace torch.randn_like(image_emb) with a deterministic negative embedding, likely get_zero_embed(...) for the empty negative case and a real negative-prompt path when supported.
Issue 2: Decoder no-CFG paths do not repeat or cast conditioning tensors
Affected code:
| ifisinstance(image_embeds, list): |
| image_embeds=torch.cat(image_embeds, dim=0) |
| batch_size=image_embeds.shape[0] *num_images_per_prompt |
| ifisinstance(negative_image_embeds, list): |
| negative_image_embeds=torch.cat(negative_image_embeds, dim=0) |
| |
| ifself.do_classifier_free_guidance: |
| image_embeds=image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| negative_image_embeds=negative_image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| |
| image_embeds=torch.cat([negative_image_embeds, image_embeds], dim=0).to( |
| dtype=self.unet.dtype, device=device |
| ) |
| ifisinstance(image_embeds, list): |
| image_embeds=torch.cat(image_embeds, dim=0) |
| batch_size=image_embeds.shape[0] |
| ifisinstance(negative_image_embeds, list): |
| negative_image_embeds=torch.cat(negative_image_embeds, dim=0) |
| |
| ifself.do_classifier_free_guidance: |
| image_embeds=image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| negative_image_embeds=negative_image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| |
| image_embeds=torch.cat([negative_image_embeds, image_embeds], dim=0).to( |
| dtype=self.unet.dtype, device=device |
| ) |
| ifisinstance(image_embeds, list): |
| image_embeds=torch.cat(image_embeds, dim=0) |
| batch_size=image_embeds.shape[0] *num_images_per_prompt |
| ifisinstance(negative_image_embeds, list): |
| negative_image_embeds=torch.cat(negative_image_embeds, dim=0) |
| |
| ifself.do_classifier_free_guidance: |
| image_embeds=image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| negative_image_embeds=negative_image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| |
| image_embeds=torch.cat([negative_image_embeds, image_embeds], dim=0).to( |
| dtype=self.unet.dtype, device=device |
| ) |
| ifisinstance(image_embeds, list): |
| image_embeds=torch.cat(image_embeds, dim=0) |
| ifisinstance(negative_image_embeds, list): |
| negative_image_embeds=torch.cat(negative_image_embeds, dim=0) |
| ifisinstance(hint, list): |
| hint=torch.cat(hint, dim=0) |
| |
| batch_size=image_embeds.shape[0] *num_images_per_prompt |
| |
| ifdo_classifier_free_guidance: |
| image_embeds=image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| negative_image_embeds=negative_image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| hint=hint.repeat_interleave(num_images_per_prompt, dim=0) |
| |
| image_embeds=torch.cat([negative_image_embeds, image_embeds], dim=0).to( |
| dtype=self.unet.dtype, device=device |
| ) |
| hint=torch.cat([hint, hint], dim=0).to(dtype=self.unet.dtype, device=device) |
| ifisinstance(image_embeds, list): |
| image_embeds=torch.cat(image_embeds, dim=0) |
| ifisinstance(negative_image_embeds, list): |
| negative_image_embeds=torch.cat(negative_image_embeds, dim=0) |
| ifisinstance(hint, list): |
| hint=torch.cat(hint, dim=0) |
| |
| batch_size=image_embeds.shape[0] |
| |
| ifdo_classifier_free_guidance: |
| image_embeds=image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| negative_image_embeds=negative_image_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| hint=hint.repeat_interleave(num_images_per_prompt, dim=0) |
| |
| image_embeds=torch.cat([negative_image_embeds, image_embeds], dim=0).to( |
| dtype=self.unet.dtype, device=device |
| ) |
| hint=torch.cat([hint, hint], dim=0).to(dtype=self.unet.dtype, device=device) |
Problem:
image_embeds, and hint for ControlNet, are repeated for num_images_per_prompt and moved to self.unet.dtype/device only inside the CFG branch. With guidance_scale <= 1, latents are sized for batch * num_images_per_prompt, but conditioning remains at the original batch size and dtype/device.
Impact:
No-CFG generation can fail for num_images_per_prompt > 1, half-precision pipelines, CPU-to-accelerator inputs, and ControlNet hints. The ControlNet subset is already reported in #4183.
Reproduction:
fromtypesimportSimpleNamespaceimporttorchfromdiffusersimportKandinskyV22PipelineclassFakeUNet:
dtype=torch.float32config=SimpleNamespace(in_channels=4)
def__call__(self, sample, timestep, encoder_hidden_states=None, added_cond_kwargs=None, return_dict=False):
assertadded_cond_kwargs["image_embeds"].shape[0] ==sample.shape[0], (
added_cond_kwargs["image_embeds"].shape, sample.shape
)
return (torch.zeros(sample.shape[0], sample.shape[1] *2, sample.shape[2], sample.shape[3]),)
classFakeScheduler:
init_noise_sigma=1.0config=SimpleNamespace(variance_type="learned")
defset_timesteps(self, *args, **kwargs): self.timesteps=torch.tensor([1])
defstep(self, noise_pred, t, latents, generator=None): return (latents,)
classFakeMovq:
config=SimpleNamespace(block_out_channels=[1, 1], latent_channels=4)
pipe=KandinskyV22Pipeline(FakeUNet(), FakeScheduler(), FakeMovq())
pipe.set_progress_bar_config(disable=True)
pipe(
image_embeds=torch.randn(1, 32),
negative_image_embeds=torch.randn(1, 32),
num_images_per_prompt=2,
guidance_scale=1.0,
output_type="latent",
)
Relevant precedent:
The prior pipeline expands text conditioning before CFG concatenation:
| prompt_embeds=prompt_embeds.repeat_interleave(num_images_per_prompt, dim=0) |
| text_encoder_hidden_states=text_encoder_hidden_states.repeat_interleave(num_images_per_prompt, dim=0) |
| text_mask=text_mask.repeat_interleave(num_images_per_prompt, dim=0) |
Suggested fix:
image_embeds=image_embeds.repeat_interleave(num_images_per_prompt, dim=0).to(
dtype=self.unet.dtype, device=device
)
ifself.do_classifier_free_guidance:
negative_image_embeds=negative_image_embeds.repeat_interleave(num_images_per_prompt, dim=0).to(
dtype=self.unet.dtype, device=device
)
image_embeds=torch.cat([negative_image_embeds, image_embeds], dim=0)
Apply the same pattern to hint in both ControlNet pipelines before duplicating it for CFG.
Issue 3: Inpainting batches preserve the first image/mask for every sample
Affected code:
| init_latents_proper=image[:1] |
| init_mask=mask_image[:1] |
| |
| ifi<len(timesteps) -1: |
| noise_timestep=timesteps[i+1] |
| init_latents_proper=self.scheduler.add_noise( |
| init_latents_proper, noise, torch.tensor([noise_timestep]) |
| ) |
| |
| latents=init_mask*init_latents_proper+ (1-init_mask) *latents |
| # post-processing |
| latents=mask_image[:1] *image[:1] + (1-mask_image[:1]) *latents |
Problem:
The inpaint loop uses image[:1] and mask_image[:1] when reinserting preserved regions. For batched inputs, every sample uses the first encoded image and first mask instead of its own.
Impact:
Batched inpainting silently produces wrong preserved regions. This can make all batch outputs inherit the first input image under preserved mask areas.
Reproduction:
fromtypesimportSimpleNamespaceimporttorchfromdiffusersimportKandinskyV22InpaintPipelineclassFakeUNet:
dtype=torch.float32config=SimpleNamespace(in_channels=9)
def__call__(self, sample, timestep, encoder_hidden_states=None, added_cond_kwargs=None, return_dict=False):
return (torch.zeros(sample.shape[0], 8, sample.shape[2], sample.shape[3]),)
classFakeScheduler:
init_noise_sigma=1.0config=SimpleNamespace(variance_type="learned")
defset_timesteps(self, *args, **kwargs): self.timesteps=torch.tensor([1])
defstep(self, noise_pred, t, latents, generator=None): return (latents,)
classFakeMovq:
config=SimpleNamespace(block_out_channels=[1, 1], latent_channels=4)
defencode(self, image):
values=image.mean(dim=(1, 2, 3), keepdim=True)
return {"latents": values.expand(image.shape[0], 4, 32, 32)}
pipe=KandinskyV22InpaintPipeline(FakeUNet(), FakeScheduler(), FakeMovq())
pipe.set_progress_bar_config(disable=True)
image=torch.stack([torch.full((3, 64, 64), -1.0), torch.full((3, 64, 64), 1.0)])
mask=torch.zeros(2, 1, 64, 64) # preserve both inputsout=pipe(
image_embeds=torch.randn(2, 32),
negative_image_embeds=torch.randn(2, 32),
image=image,
mask_image=mask,
output_type="latent",
).imagesprint(torch.unique(out[0]).item(), torch.unique(out[1]).item())Relevant precedent:
Stable Diffusion inpainting keeps per-sample mask and masked-image latents batched through the denoising loop rather than slicing to the first item.
Suggested fix:
init_latents=image.repeat_interleave(num_images_per_prompt, dim=0)
init_mask=mask_image.repeat_interleave(num_images_per_prompt, dim=0)
mask_image=init_maskmasked_image=init_latents*init_maskifself.do_classifier_free_guidance:
mask_image=mask_image.repeat(2, 1, 1, 1)
masked_image=masked_image.repeat(2, 1, 1, 1)
...
init_latents_proper=init_latentsifi<len(timesteps) -1:
noise_timestep=timesteps[i+1]
init_latents_proper=self.scheduler.add_noise(init_latents, noise, noise_timestep[None])
latents=init_mask*init_latents_proper+ (1-init_mask) *latents
...
latents=init_mask*init_latents+ (1-init_mask) *latents
Issue 4: User-provided latents are not cast to the requested dtype
Affected code:
| # Copied from diffusers.pipelines.deprecated.unclip.pipeline_unclip.UnCLIPPipeline.prepare_latents |
| defprepare_latents(self, shape, dtype, device, generator, latents, scheduler): |
| iflatentsisNone: |
| latents=randn_tensor(shape, generator=generator, device=device, dtype=dtype) |
| else: |
| iflatents.shape!=shape: |
| raiseValueError(f"Unexpected latents shape, got {latents.shape}, expected {shape}") |
| latents=latents.to(device) |
| |
| latents=latents*scheduler.init_noise_sigma |
| # Copied from diffusers.pipelines.deprecated.unclip.pipeline_unclip.UnCLIPPipeline.prepare_latents |
| defprepare_latents(self, shape, dtype, device, generator, latents, scheduler): |
| iflatentsisNone: |
| latents=randn_tensor(shape, generator=generator, device=device, dtype=dtype) |
| else: |
| iflatents.shape!=shape: |
| raiseValueError(f"Unexpected latents shape, got {latents.shape}, expected {shape}") |
| latents=latents.to(device) |
| |
| latents=latents*scheduler.init_noise_sigma |
| defprepare_latents(self, shape, dtype, device, generator, latents, scheduler): |
| iflatentsisNone: |
| latents=randn_tensor(shape, generator=generator, device=device, dtype=dtype) |
| else: |
| iflatents.shape!=shape: |
| raiseValueError(f"Unexpected latents shape, got {latents.shape}, expected {shape}") |
| latents=latents.to(device) |
| |
| latents=latents*scheduler.init_noise_sigma |
| defprepare_latents(self, shape, dtype, device, generator, latents, scheduler): |
| iflatentsisNone: |
| latents=randn_tensor(shape, generator=generator, device=device, dtype=dtype) |
| else: |
| iflatents.shape!=shape: |
| raiseValueError(f"Unexpected latents shape, got {latents.shape}, expected {shape}") |
| latents=latents.to(device) |
| |
| latents=latents*scheduler.init_noise_sigma |
Problem:
The copied prepare_latents() blocks cast generated latents to dtype, but user-provided latents only call .to(device). In half precision, a float32 latent tensor stays float32 and can be passed into fp16 modules.
Impact:
Supplying reusable latents to fp16 pipelines can fail with dtype mismatch errors or force unintended float32 compute.
Reproduction:
fromtypesimportSimpleNamespaceimporttorchfromdiffusersimportKandinskyV22Pipelinepipe=KandinskyV22Pipeline.__new__(KandinskyV22Pipeline)
scheduler=SimpleNamespace(init_noise_sigma=1.0)
latents=torch.randn(1, 4, 32, 32, dtype=torch.float32)
out=KandinskyV22Pipeline.prepare_latents(
pipe, latents.shape, torch.float16, torch.device("cpu"), None, latents, scheduler
)
print(out.dtype) # torch.float32, expected torch.float16Relevant precedent:
Flux casts provided latents with both device and dtype:
| iflatentsisnotNone: |
| latent_image_ids=self._prepare_latent_image_ids(batch_size, height//2, width//2, device, dtype) |
| returnlatents.to(device=device, dtype=dtype), latent_image_ids |
| |
Suggested fix:
latents=latents.to(device=device, dtype=dtype)
Because this is a copied block, either update the copied source and run make fix-copies, or remove the copy annotation if Kandinsky needs target-specific behavior.
Issue 5: Deprecated callback crashes when callback_steps is omitted
Affected code:
| callback=kwargs.pop("callback", None) |
| callback_steps=kwargs.pop("callback_steps", None) |
| |
| ifcallbackisnotNone: |
| deprecate( |
| "callback", |
| "1.0.0", |
| "Passing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", |
| ) |
| ifcallback_stepsisnotNone: |
| deprecate( |
| "callback_steps", |
| "1.0.0", |
| "Passing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", |
| ) |
| ifcallbackisnotNoneandi%callback_steps==0: |
| step_idx=i//getattr(self.scheduler, "order", 1) |
| callback(step_idx, t, latents) |
| callback=kwargs.pop("callback", None) |
| callback_steps=kwargs.pop("callback_steps", None) |
| |
| ifcallbackisnotNone: |
| deprecate( |
| "callback", |
| "1.0.0", |
| "Passing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", |
| ) |
| ifcallback_stepsisnotNone: |
| deprecate( |
| "callback_steps", |
| "1.0.0", |
| "Passing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", |
| ) |
| callback=kwargs.pop("callback", None) |
| callback_steps=kwargs.pop("callback_steps", None) |
| |
| ifcallbackisnotNone: |
| deprecate( |
| "callback", |
| "1.0.0", |
| "Passing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", |
| ) |
| ifcallback_stepsisnotNone: |
| deprecate( |
| "callback_steps", |
| "1.0.0", |
| "Passing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", |
| ) |
Problem:
The deprecated callback kwarg is still accepted, but callback_steps defaults to None after kwargs.pop("callback_steps", None). The loop then evaluates i % callback_steps, causing TypeError.
Impact:
Backward-compatible callback usage fails unless users also pass a deprecated callback_steps kwarg.
Reproduction:
fromtypesimportSimpleNamespaceimporttorchfromdiffusersimportKandinskyV22PipelineclassFakeUNet:
dtype=torch.float32config=SimpleNamespace(in_channels=4)
def__call__(self, sample, timestep, encoder_hidden_states=None, added_cond_kwargs=None, return_dict=False):
return (torch.zeros(sample.shape[0], 8, sample.shape[2], sample.shape[3]),)
classFakeScheduler:
init_noise_sigma=1.0config=SimpleNamespace(variance_type="learned")
defset_timesteps(self, *args, **kwargs): self.timesteps=torch.tensor([1])
defstep(self, noise_pred, t, latents, generator=None): return (latents,)
classFakeMovq:
config=SimpleNamespace(block_out_channels=[1, 1], latent_channels=4)
pipe=KandinskyV22Pipeline(FakeUNet(), FakeScheduler(), FakeMovq())
pipe.set_progress_bar_config(disable=True)
pipe(
image_embeds=torch.randn(1, 32),
negative_image_embeds=torch.randn(1, 32),
output_type="latent",
callback=lambdastep, t, latents: None,
)
Relevant precedent:
The ControlNet variants still expose callback_steps: int = 1 in the signature:
| generator: torch.Generator|list[torch.Generator] |None=None, |
| latents: torch.Tensor|None=None, |
| output_type: str|None="pil", |
| callback: Callable[[int, int, torch.Tensor], None] |None=None, |
| callback_steps: int=1, |
| return_dict: bool=True, |
Suggested fix:
callback_steps=kwargs.pop("callback_steps", 1)
ifcallback_stepsisNone:
callback_steps=1Also validate that callback_steps is a positive integer before the denoising loop.
Issue 6: Slow coverage is missing for several exported pipelines
Affected code:
| deftest_kandinsky_prior(self): |
| device="cpu" |
| |
| components=self.get_dummy_components() |
| |
| pipe=self.pipeline_class(**components) |
| pipe=pipe.to(device) |
| |
| pipe.set_progress_bar_config(disable=None) |
| |
| output=pipe(**self.get_dummy_inputs(device)) |
| image=output.image_embeds |
| |
| image_from_tuple=pipe( |
| **self.get_dummy_inputs(device), |
| return_dict=False, |
| )[0] |
| |
| image_slice=image[0, -10:] |
| |
| image_from_tuple_slice=image_from_tuple[0, -10:] |
| |
| assertimage.shape== (1, 32) |
| |
| expected_slice=np.array( |
| [-0.5948, 0.1875, -0.1523, -1.1995, -1.4061, -0.6367, -1.4607, -0.6406, 0.8793, -0.3891] |
| ) |
| |
| assertnp.abs(image_slice.flatten() -expected_slice).max() <1e-2 |
| assertnp.abs(image_from_tuple_slice.flatten() -expected_slice).max() <1e-2 |
| |
| @skip_mps |
| deftest_inference_batch_single_identical(self): |
| self._test_inference_batch_single_identical(expected_max_diff=1e-3) |
| |
| @skip_mps |
| deftest_attention_slicing_forward_pass(self): |
| test_max_difference=torch_device=="cpu" |
| test_mean_pixel_difference=False |
| |
| self._test_attention_slicing_forward_pass( |
| test_max_difference=test_max_difference, |
| test_mean_pixel_difference=test_mean_pixel_difference, |
| ) |
| |
| # override default test because no output_type "latent", use "pt" instead |
| deftest_callback_inputs(self): |
| deftest_kandinsky_prior_emb2emb(self): |
| device="cpu" |
| |
| components=self.get_dummy_components() |
| |
| pipe=self.pipeline_class(**components) |
| pipe=pipe.to(device) |
| |
| pipe.set_progress_bar_config(disable=None) |
| |
| output=pipe(**self.get_dummy_inputs(device)) |
| image=output.image_embeds |
| |
| image_from_tuple=pipe( |
| **self.get_dummy_inputs(device), |
| return_dict=False, |
| )[0] |
| |
| image_slice=image[0, -10:] |
| |
| image_from_tuple_slice=image_from_tuple[0, -10:] |
| |
| assertimage.shape== (1, 32) |
| |
| expected_slice=np.array( |
| [-0.8947, 0.7225, -0.2400, -1.4224, -1.9268, -1.1454, -1.8220, -0.7972, 1.0465, -0.5207] |
| ) |
| |
| assertnp.abs(image_slice.flatten() -expected_slice).max() <1e-2 |
| assertnp.abs(image_from_tuple_slice.flatten() -expected_slice).max() <1e-2 |
| |
| @skip_mps |
| deftest_inference_batch_single_identical(self): |
| self._test_inference_batch_single_identical(expected_max_diff=1e-2) |
| |
| @skip_mps |
| deftest_attention_slicing_forward_pass(self): |
| test_max_difference=torch_device=="cpu" |
| deftest_float16_inference(self): |
| super().test_float16_inference(expected_max_diff=1e-1) |
| |
| deftest_inference_batch_single_identical(self): |
| super().test_inference_batch_single_identical(expected_max_diff=5e-4) |
| |
| |
| @nightly |
| @require_torch_accelerator |
| classKandinskyV22ControlnetPipelineIntegrationTests(unittest.TestCase): |
| defsetUp(self): |
| # clean up the VRAM before each test |
| super().setUp() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftearDown(self): |
| # clean up the VRAM after each test |
| super().tearDown() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftest_kandinsky_controlnet(self): |
| deftest_inference_batch_single_identical(self): |
| super().test_inference_batch_single_identical(expected_max_diff=1.75e-3) |
| |
| deftest_float16_inference(self): |
| super().test_float16_inference(expected_max_diff=2e-1) |
| |
| |
| @nightly |
| @require_torch_accelerator |
| classKandinskyV22ControlnetImg2ImgPipelineIntegrationTests(unittest.TestCase): |
| defsetUp(self): |
| # clean up the VRAM before each test |
| super().setUp() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftearDown(self): |
| # clean up the VRAM after each test |
| super().tearDown() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftest_kandinsky_controlnet_img2img(self): |
| classKandinskyV22PipelineCombinedFastTests(PipelineTesterMixin, unittest.TestCase): |
| pipeline_class=KandinskyV22CombinedPipeline |
| params= ["prompt"] |
| batch_params= ["prompt", "negative_prompt"] |
| required_optional_params= [ |
| "generator", |
| "height", |
| "width", |
| "latents", |
| "guidance_scale", |
| "negative_prompt", |
| "num_inference_steps", |
| "return_dict", |
| "guidance_scale", |
| "num_images_per_prompt", |
| "output_type", |
| "return_dict", |
| ] |
| test_xformers_attention=True |
| callback_cfg_params= ["image_embds"] |
| |
| supports_dduf=False |
Problem:
The family has fast tests, and text/img2img/inpaint decoder pipelines have @slow integration tests. But KandinskyV22PriorPipeline, KandinskyV22PriorEmb2EmbPipeline, all three combined pipelines, and both ControlNet pipelines do not have @slow tests. The ControlNet integration tests are @nightly, which means they are not collected by slow CI.
Impact:
Published checkpoints and connected-pipeline loading paths can regress without slow-suite coverage. This also leaves the broken PriorEmb2Emb.interpolate() path untested.
Reproduction:
frompathlibimportPathforpathinsorted(Path("tests/pipelines/kandinsky2_2").glob("test_*.py")):
text=path.read_text()
print(f"{path.name}: slow={'@slow'intext}, nightly={'@nightly'intext}")Relevant precedent:
Existing slow coverage for decoder text/img2img/inpaint:
| @slow |
| @require_torch_accelerator |
| classKandinskyV22PipelineIntegrationTests(unittest.TestCase): |
| defsetUp(self): |
| # clean up the VRAM before each test |
| super().setUp() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftearDown(self): |
| # clean up the VRAM after each test |
| super().tearDown() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftest_kandinsky_text2img(self): |
| @slow |
| @require_torch_accelerator |
| classKandinskyV22Img2ImgPipelineIntegrationTests(unittest.TestCase): |
| defsetUp(self): |
| # clean up the VRAM before each test |
| super().setUp() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftearDown(self): |
| # clean up the VRAM after each test |
| super().tearDown() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftest_kandinsky_img2img(self): |
| @slow |
| @require_torch_accelerator |
| classKandinskyV22InpaintPipelineIntegrationTests(unittest.TestCase): |
| defsetUp(self): |
| # clean up the VRAM before each test |
| super().setUp() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftearDown(self): |
| # clean up the VRAM after each test |
| super().tearDown() |
| gc.collect() |
| backend_empty_cache(torch_device) |
| |
| deftest_kandinsky_inpaint(self): |
Suggested fix:
Add @slow integration tests for prior, prior emb2emb interpolation, combined text/img2img/inpaint loading, and ControlNet text/img2img. Keep @nightly if desired, but add @slow so slow CI covers the family.
Notes
Public exports, lazy loading, top-level imports, dummy objects, AutoPipeline registrations, docs, examples, and fast tests were checked. Top-level imports for all exported KandinskyV22* classes succeed locally.
I attempted .venv pytest collection, but this environment's torch build fails during test import with ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package, before Kandinsky tests are collected.
kandinsky2_2model/pipeline reviewCommit tested:
0f1abc4ae8b0eb2a3b40e82a310507281144c423Review performed against the repository review rules.
Duplicate search checked
kandinsky2_2,KandinskyV22, affected class/file names, and the specific failure modes. Existing related items: #4183 covers the ControlNetguidance_scale <= 1subset of Issue 2; #4818 is related to stalePriorEmb2Emblatents plumbing, but I did not find an exact duplicate for the currentinterpolate()failure.Issue 1:
PriorEmb2Emb.interpolate()cannot handle documented text entriesAffected code:
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior_emb2emb.py
Lines 220 to 230 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior_emb2emb.py
Lines 399 to 413 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior_emb2emb.py
Lines 244 to 246 in 0f1abc4
Problem:
interpolate()advertiseslist[str | PIL.Image.Image | torch.Tensor], but the string branch callsself(..., latents=latents)whileKandinskyV22PriorEmb2EmbPipeline.__call__has nolatentsparameter and requiresimage. The same method also returnstorch.randn_like(image_emb)fornegative_image_embeds, making the negative conditioning random and unrelated tonegative_prompt.Impact:
The documented mixed text/image interpolation workflow fails immediately for text entries. Image-only interpolation is nondeterministic on the negative branch and can change decoder CFG behavior across calls.
Reproduction:
Relevant precedent:
KandinskyV22PriorPipeline.interpolate()supports text entries by calling a compatible text-only prior path and derives the negative image embedding from a real negative/zero embedding:diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py
Lines 197 to 241 in 0f1abc4
Suggested fix:
Implement the text branch by reusing/copying the text-only prior logic from
KandinskyV22PriorPipeline, or narrow the accepted input types and docs to image/tensor only. Replacetorch.randn_like(image_emb)with a deterministic negative embedding, likelyget_zero_embed(...)for the empty negative case and a real negative-prompt path when supported.Issue 2: Decoder no-CFG paths do not repeat or cast conditioning tensors
Affected code:
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py
Lines 227 to 239 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_img2img.py
Lines 286 to 298 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_inpainting.py
Lines 420 to 432 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_controlnet.py
Lines 236 to 253 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_controlnet_img2img.py
Lines 278 to 295 in 0f1abc4
Problem:
image_embeds, andhintfor ControlNet, are repeated fornum_images_per_promptand moved toself.unet.dtype/deviceonly inside the CFG branch. Withguidance_scale <= 1, latents are sized forbatch * num_images_per_prompt, but conditioning remains at the original batch size and dtype/device.Impact:
No-CFG generation can fail for
num_images_per_prompt > 1, half-precision pipelines, CPU-to-accelerator inputs, and ControlNet hints. The ControlNet subset is already reported in #4183.Reproduction:
Relevant precedent:
The prior pipeline expands text conditioning before CFG concatenation:
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py
Lines 301 to 303 in 0f1abc4
Suggested fix:
Apply the same pattern to
hintin both ControlNet pipelines before duplicating it for CFG.Issue 3: Inpainting batches preserve the first image/mask for every sample
Affected code:
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_inpainting.py
Lines 510 to 519 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_inpainting.py
Lines 540 to 541 in 0f1abc4
Problem:
The inpaint loop uses
image[:1]andmask_image[:1]when reinserting preserved regions. For batched inputs, every sample uses the first encoded image and first mask instead of its own.Impact:
Batched inpainting silently produces wrong preserved regions. This can make all batch outputs inherit the first input image under preserved mask areas.
Reproduction:
Relevant precedent:
Stable Diffusion inpainting keeps per-sample mask and masked-image latents batched through the denoising loop rather than slicing to the first item.
Suggested fix:
Issue 4: User-provided latents are not cast to the requested dtype
Affected code:
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py
Lines 106 to 115 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py
Lines 243 to 252 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_inpainting.py
Lines 279 to 287 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_controlnet.py
Lines 149 to 157 in 0f1abc4
Problem:
The copied
prepare_latents()blocks cast generated latents todtype, but user-provided latents only call.to(device). In half precision, a float32 latent tensor stays float32 and can be passed into fp16 modules.Impact:
Supplying reusable latents to fp16 pipelines can fail with dtype mismatch errors or force unintended float32 compute.
Reproduction:
Relevant precedent:
Flux casts provided latents with both device and dtype:
diffusers/src/diffusers/pipelines/flux/pipeline_flux.py
Lines 615 to 618 in 0f1abc4
Suggested fix:
Because this is a copied block, either update the copied source and run
make fix-copies, or remove the copy annotation if Kandinsky needs target-specific behavior.Issue 5: Deprecated
callbackcrashes whencallback_stepsis omittedAffected code:
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py
Lines 200 to 214 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2.py
Lines 303 to 305 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_img2img.py
Lines 259 to 273 in 0f1abc4
diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_inpainting.py
Lines 393 to 407 in 0f1abc4
Problem:
The deprecated
callbackkwarg is still accepted, butcallback_stepsdefaults toNoneafterkwargs.pop("callback_steps", None). The loop then evaluatesi % callback_steps, causingTypeError.Impact:
Backward-compatible callback usage fails unless users also pass a deprecated
callback_stepskwarg.Reproduction:
Relevant precedent:
The ControlNet variants still expose
callback_steps: int = 1in the signature:diffusers/src/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_controlnet.py
Lines 171 to 176 in 0f1abc4
Suggested fix:
Also validate that
callback_stepsis a positive integer before the denoising loop.Issue 6: Slow coverage is missing for several exported pipelines
Affected code:
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky_prior.py
Lines 199 to 245 in 0f1abc4
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky_prior_emb2emb.py
Lines 201 to 238 in 0f1abc4
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky_controlnet.py
Lines 224 to 246 in 0f1abc4
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky_controlnet_img2img.py
Lines 230 to 252 in 0f1abc4
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky_combined.py
Lines 37 to 58 in 0f1abc4
Problem:
The family has fast tests, and text/img2img/inpaint decoder pipelines have
@slowintegration tests. ButKandinskyV22PriorPipeline,KandinskyV22PriorEmb2EmbPipeline, all three combined pipelines, and both ControlNet pipelines do not have@slowtests. The ControlNet integration tests are@nightly, which means they are not collected by slow CI.Impact:
Published checkpoints and connected-pipeline loading paths can regress without slow-suite coverage. This also leaves the broken
PriorEmb2Emb.interpolate()path untested.Reproduction:
Relevant precedent:
Existing slow coverage for decoder text/img2img/inpaint:
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky.py
Lines 225 to 240 in 0f1abc4
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky_img2img.py
Lines 242 to 257 in 0f1abc4
diffusers/tests/pipelines/kandinsky2_2/test_kandinsky_inpaint.py
Lines 303 to 318 in 0f1abc4
Suggested fix:
Add
@slowintegration tests for prior, prior emb2emb interpolation, combined text/img2img/inpaint loading, and ControlNet text/img2img. Keep@nightlyif desired, but add@slowso slow CI covers the family.Notes
Public exports, lazy loading, top-level imports, dummy objects, AutoPipeline registrations, docs, examples, and fast tests were checked. Top-level imports for all exported
KandinskyV22*classes succeed locally.I attempted
.venvpytest collection, but this environment's torch build fails during test import withModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package, before Kandinsky tests are collected.