[Kolors] Add IP Adapter

[asomoza]

What does this PR do?

Add the new released IP Adapter to the Kolors Pipelines

Note: I'll add the img2img after the initial review and after #8856 is merged.

How to test

T2I

importtorchfromtransformersimportCLIPVisionModelWithProjectionfromdiffusersimportDPMSolverMultistepScheduler, KolorsPipelinefromdiffusers.utilsimportload_imageimage_encoder=CLIPVisionModelWithProjection.from_pretrained(
"Kwai-Kolors/Kolors-IP-Adapter-Plus",
subfolder="image_encoder",
low_cpu_mem_usage=True,
torch_dtype=torch.float16,
revision="refs/pr/4",
)
pipe=KolorsPipeline.from_pretrained(
"Kwai-Kolors/Kolors-diffusers", image_encoder=image_encoder, torch_dtype=torch.float16, variant="fp16"
).to("cuda")
pipe.scheduler=DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)
pipe.load_ip_adapter(
"Kwai-Kolors/Kolors-IP-Adapter-Plus",
subfolder="",
weight_name="ip_adapter_plus_general.safetensors",
revision="refs/pr/4",
image_encoder_folder=None,
)
pipe.enable_model_cpu_offload()
ipa_image=load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/kolors/cat_square.png")
image=pipe(
prompt="best quality, high quality",
negative_prompt="",
guidance_scale=6.5,
num_inference_steps=25,
ip_adapter_image=ipa_image,
).images[0]
image.save("kolors_ipa_result.png")

source	result 1	result 2

IMG2IMG

importmathimporttorchfromtransformersimportCLIPVisionModelWithProjectionfromdiffusersimportDPMSolverMultistepScheduler, KolorsImg2ImgPipelinefromdiffusers.utilsimportload_imageimage_encoder=CLIPVisionModelWithProjection.from_pretrained(
"Kwai-Kolors/Kolors-IP-Adapter-Plus",
subfolder="image_encoder",
low_cpu_mem_usage=True,
torch_dtype=torch.float16,
revision="refs/pr/4",
)
pipe=KolorsImg2ImgPipeline.from_pretrained(
"Kwai-Kolors/Kolors-diffusers", image_encoder=image_encoder, torch_dtype=torch.float16, variant="fp16"
).to("cuda")
pipe.scheduler=DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)
pipe.load_ip_adapter(
"Kwai-Kolors/Kolors-IP-Adapter-Plus",
subfolder="",
weight_name="ip_adapter_plus_general.safetensors",
revision="refs/pr/4",
image_encoder_folder=None,
)
pipe.set_ip_adapter_scale(0.4)
pipe.enable_model_cpu_offload()
source_image=load_image(
"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/kolors/capyrabbit.png?download=true"
)
ipa_image=load_image(
"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/kolors/ip_image.png?download=true"
)
prompt="a capybara wearing sunglasses. In the background of the image there are trees, poles, grass and other objects. At the bottom of the object there is the road., 8k, highly detailed."strength=0.5steps=25num_inference_steps=math.ceil(steps/strength)
image=pipe(
prompt=prompt,
image=source_image,
negative_prompt="",
guidance_scale=6.5,
num_inference_steps=num_inference_steps,
strength=strength,
ip_adapter_image=ipa_image,
).images[0]
image.save("kolors_img2img_ipa_result.png")

original	ip image	result	style

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

also, don't forget to set it back in unload_ip_adapter

diffusers/src/diffusers/loaders/ip_adapter.py

Line 300 in a785992

defunload_ip_adapter(self):

[yiyixuxu]

thank you!

[asomoza]

@stevhliu can you please review the documentation.

[stevhliu]

Very nice! Just a few comments to improve clarity 😄

[e1ijah1]

Hello, and thank you for this excellent PR! I'm currently working with a model that's encountering memory issues when loading the IP-Adapter on a single GPU. I noticed that the load_ip_adapter method doesn't seem to support specifying different devices directly.
I'm wondering if you could provide some guidance on how to effectively utilize multiple GPUs when loading the model and IP-Adapter?

[asomoza]

AFAIK we don't have a method of separating the IP Adapter from the model device, what you can do is to get the image embeddings before inference.

Also, you can split the pipeline modules in different devices too, this guide should be applicable for this use case too.

You can even get the text embeddings separately as shown in that guide, so actually the most important part here would be to free the VRAM of the text encoder than separating the IP Adapter into another device.