Support of ip-adapter to the StableDiffusionControlNetInpaintPipeline

[juancopi81]

What does this PR do?

This PR adds support of the ip-adapter to the StableDiffusionControlNetInpaintPipeline. The ip-adapter was added in #5713 and help was asked in #5884 I think it is very cool to have this support (see example below).

This PR refers to #5884

This is how it works:

fromdiffusersimport (
StableDiffusionControlNetInpaintPipeline,
ControlNetModel,
LCMScheduler,
)
importtorchfromdiffusers.utilsimportload_imagefromPILimportImageimportcv2importnumpyasnpdefget_canny_filter(image):
ifnotisinstance(image, np.ndarray):
image=np.array(image) image=cv2.Canny(image, 100, 200)
image=image[:, :, None]
image=np.concatenate([image, image, image], axis=2)
canny_image=Image.fromarray(image)
returncanny_imagecontrolnet=ControlNetModel.from_pretrained(
"lllyasviel/control_v11p_sd15_canny",
torch_dtype=torch.float16,
)
pipe=StableDiffusionControlNetInpaintPipeline.from_pretrained(
"jayparmr/icbinp_v8_inpaint_v2",
controlnet=controlnet,
torch_dtype=torch.float16,
requires_safety_checker=False,
safety_checker=None,
)
pipe.to("cuda")
#Load imagesimage=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/inpaint_image.png")
mask=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/mask.png")
ip_image=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/girl.png")
cn_image=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/inpaint_image.png")
cn_image=get_canny_filter(cn_image)
# Resize imagesimage=image.resize((512, 768))
mask=mask.resize((512, 768))
cn_image=cn_image.resize((512, 768))
# Load ip-adapterpipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
# Generate imagegenerator=torch.Generator(device="cpu").manual_seed(33)
images=pipe(
prompt='best quality, high quality', image=image,
mask_image=mask,
control_image=cn_image,
ip_adapter_image=ip_image,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_images_per_prompt=1, num_inference_steps=50,
generator=generator,
strength=0.5,
controlnet_conditioning_scale=0.5,
).imagesimages[0].save("juancopi_test_4_out.png")

Initial image | Mask | ip-adapter image | canny image | output

Final output:

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case. [IP-Adapter] adding IP adapter support to all ControlNet and T2I pipelines #5884
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests? Some test are failing, but I wanted to check if the threshold is ok:
  FAILED tests/pipelines/controlnet/test_controlnet_inpaint.py::ControlNetInpaintPipelineFastTests::test_save_load_local - AssertionError: 0.0016172826 not less than 0.0005
  FAILED tests/pipelines/controlnet/test_controlnet_inpaint.py::ControlNetSimpleInpaintPipelineFastTests::test_save_load_local - AssertionError: 0.00053209066 not less than 0.0005
  FAILED tests/pipelines/controlnet/test_controlnet_inpaint.py::MultiControlNetInpaintPipelineFastTests::test_save_load_local - AssertionError: 0.00051498413 not less than 0.0005

Who can review?

@yiyixuxu Hi @yiyixuxu please let me know any comments 😃 I hope it is ok, I am happy to change anything you need.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[yiyixuxu]

thank you!

[patrickvonplaten]

Let's also add one test related to IP Adapters here :-)

[patrickvonplaten]

@yiyixuxu wdyt?

[wwirving]

Hi @juancopi81@yiyixuxu - Can we expect this PR to support StableDiffusionXLControlNetInpaintPipeline?

[juancopi81]

Hi @wwirving, not really, this PR is only for the StableDiffusionControlNetInpaintPipeline pipeline.

[yiyixuxu]

thanks! looks great :)
can we add a test like @patrickvonplaten requested?

[juancopi81]

Sure @patrickvonplaten and @yiyixuxu I'll work on that. Just to be sure, you are expecting something like this:

classMultiControlNetInpaintPipelineFastTests(
PipelineTesterMixin, PipelineKarrasSchedulerTesterMixin, unittest.TestCase
):
pipeline_class=StableDiffusionControlNetInpaintPipelineparams=TEXT_GUIDED_IMAGE_INPAINTING_PARAMSbatch_params=TEXT_GUIDED_IMAGE_INPAINTING_BATCH_PARAMSdefget_dummy_components(self):
torch.manual_seed(0)
unet= ...
### NEW CODE ####torch.manual_seed(0)
image_encoder_config=CLIPVisionConfig(
image_size=64,
num_channels=3,
hidden_size=32,
projection_dim=32,
num_hidden_layers=2,
num_attention_heads=4,
intermediate_size=37,
dropout=0.1,
attention_dropout=0.1,
initializer_range=0.02,
scope=None,
)
image_encoder=CLIPVisionModelWithProjection(config=image_encoder_config)
torch.manual_seed(0)
feature_extractor_config= {
"image_size": 64,
"num_channels": 3,
"do_resize": True,
"size": {"shortest_edge": 20},
"do_center_crop": True,
"crop_size": {"height": 64, "width": 64},
"do_normalize": True,
"image_mean": [0.48145466, 0.4578275, 0.40821073],
"image_std": [0.26862954, 0.26130258, 0.27577711],
"do_convert_rgb": True
}
feature_extractor=CLIPImageProcessor(**feature_extractor_config)
components= {
"unet": unet,
"controlnet": controlnet,
"scheduler": scheduler,
"vae": vae,
"text_encoder": text_encoder,
"tokenizer": tokenizer,
"safety_checker": None,
"feature_extractor": feature_extractor,
"image_encoder": image_encoder,
}
returncomponents

And something else? Maybe a specific test of the image_adapter? like:

deftest_with_ip_adapter_image_encoder(self):
components=self.get_dummy_components()
pipe=self.pipeline_class(**components)
pipe.to(torch_device)
inputs=self.get_dummy_inputs(torch_device)
pipe.set_ip_adapter_scale(1.0)
output_1=pipe(**inputs)[0]
inputs=self.get_dummy_inputs(torch_device)
pipe.set_ip_adapter_scale(0.0)
output_2=pipe(**inputs)[0]
# make sure that all outputs are differentassertnp.sum(np.abs(output_1-output_2)) >1e-3self.assertEqual(output_2.shape, (1, 64, 64, 3))

Right now this test:

assert np.sum(np.abs(output_1 - output_2)) > 1e-3

fails 😢, but I could take a look into that if that is what you expect, so I wanted to ask first.

EDIT: I changed the code so the feature_extractor gets also initialized in the get_dummy_components function.

[patrickvonplaten]

Actually, I think he should be good here without adapting all the controlnet tests. We also didn't add it here: #5713