Support Flux IP Adapter

[hlky]

What does this PR do?

Adds support for XLabs Flux IP Adapter.

Example

importtorchfromdiffusersimportFluxPipelinefromdiffusers.utilsimportload_imagepipe: FluxPipeline=FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
torch_dtype=torch.bfloat16,
).to("cuda")
image=load_image("assets_statue.jpg").resize((1024, 1024))
pipe.load_ip_adapter("XLabs-AI/flux-ip-adapter", weight_name="ip_adapter.safetensors", image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14")
pipe.set_ip_adapter_scale(1.0)
image=pipe(
width=1024,
height=1024,
prompt='wearing sunglasses',
negative_prompt="",
true_cfg=4.0,
generator=torch.Generator().manual_seed(4444),
ip_adapter_image=image,
).images[0]
image.save('flux_ipadapter_4444.jpg')

Input	Output

flux-ip-adapter-v2

Details

Note: true_cfg=1.0 is important, and strength is sensitive, fixed strength may not work, see here for more strength schedules, good results will require experimentation with strength schedules and the start/stop values. Results also vary with input image, I had no success with the statue image used for v1 test.

Multiple input images is not yet supported (dev note: apply torch.mean to the batch of image_embeds and to ip_attention)

importtorchfromdiffusersimportFluxPipelinefromdiffusers.utilsimportload_imagepipe: FluxPipeline=FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
torch_dtype=torch.bfloat16,
).to("cuda")
image=load_image("monalisa.jpg").resize((1024, 1024))
pipe.load_ip_adapter("XLabs-AI/flux-ip-adapter-v2", weight_name="ip_adapter.safetensors", image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14")
defLinearStrengthModel(start, finish, size):
return [
(start+ (finish-start) * (i/ (size-1))) foriinrange(size)
]
ip_strengths=LinearStrengthModel(0.4, 1.0, 19)
pipe.set_ip_adapter_scale(ip_strengths)
image=pipe(
width=1024,
height=1024,
prompt='wearing red sunglasses, golden chain and a green cap',
negative_prompt="",
true_cfg_scale=1.0,
generator=torch.Generator().manual_seed(0),
ip_adapter_image=image,
).images[0]
image.save('result.jpg')

Input	Output

Notes

• XLabs Flux IP Adapter produces bad results when used without CFG
  • Verifiable in original codebase, set --timestep_to_start_cfg greater than the number of steps to disable CFG
• XLabs Flux IP Adapter also produces bad results when used with CFG in a batch (negative and positive concat)
  • Original codebase also runs separate positive and negative inference
• This PR copies (most) of the changes from our pipeline_flux_with_cfg community example, except we run positive and negative separately.
• Conversion script is optional, original weights will be converted on-the-fly from load_ip_adapter.
• load_ip_adapter supports image_encoder_pretrained_model_name_or_path e.g. "openai/clip-vit-large-patch14" rather than just image_encoder_folder, also supports image_encoder_dtype with default torch.float16.
• This required some changes to FluxTransformerBlock because of where ip_attention is applied to the hidden_states, see here in the original codebase.
• flux-ip-adapter-v2 will be fixed and tested shortly.

Fixes#9825
Fixes#9403

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul@yiyixuxu@DN6

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

Small nits. Looks good otherwise.

[DN6]

@hlky Could we add a fast test using something similar to what's been done here

diffusers/tests/pipelines/test_pipelines_common.py

Lines 269 to 301 in 9020086

	def_modify_inputs_for_ip_adapter_test(self, inputs: Dict[str, Any]):
	parameters=inspect.signature(self.pipeline_class.__call__).parameters
	if"image"inparameters.keys() and"strength"inparameters.keys():
	inputs["num_inference_steps"] =4
	inputs["output_type"] ="np"
	inputs["return_dict"] =False
	returninputs
	deftest_ip_adapter(self, expected_max_diff: float=1e-4, expected_pipe_slice=None):
	r"""Tests for IP-Adapter.
	The following scenarios are tested:
	- Single IP-Adapter with scale=0 should produce same output as no IP-Adapter.
	- Multi IP-Adapter with scale=0 should produce same output as no IP-Adapter.
	- Single IP-Adapter with scale!=0 should produce different output compared to no IP-Adapter.
	- Multi IP-Adapter with scale!=0 should produce different output compared to no IP-Adapter.
	"""
	# Raising the tolerance for this test when it's run on a CPU because we
	# compare against static slices and that can be shaky (with a VVVV low probability).
	expected_max_diff=9e-4iftorch_device=="cpu"elseexpected_max_diff
	components=self.get_dummy_components()
	pipe=self.pipeline_class(**components).to(torch_device)
	pipe.set_progress_bar_config(disable=None)
	cross_attention_dim=pipe.unet.config.get("cross_attention_dim", 32)
	# forward pass without ip adapter
	inputs=self._modify_inputs_for_ip_adapter_test(self.get_dummy_inputs(torch_device))
	ifexpected_pipe_sliceisNone:
	output_without_adapter=pipe(**inputs)[0]
	else:
	output_without_adapter=expected_pipe_slice

[hlky]

Done.

[yiyixuxu]

left one comment, looks good otherwise!

[chasehuh]

@hlky
Thank you so much for your amazing work!
From what I've gathered, it seems that IP-Adapter isn't currently supported for the Img2Img or Inpaint pipelines. Is that correct?
If so, could you please let me know if there's a way to integrate IP-Adapter into FLUX Inpaint?

Thank you in advance for your help!

[DN6]

@chaewon-huh IP Adapter support for those pipelines is available

diffusers/src/diffusers/pipelines/flux/pipeline_flux_img2img.py

Line 169 in 1001425

classFluxImg2ImgPipeline(DiffusionPipeline, FluxLoraLoaderMixin, FromSingleFileMixin, FluxIPAdapterMixin):

diffusers/src/diffusers/pipelines/flux/pipeline_flux_inpaint.py

Line 166 in 1001425

classFluxInpaintPipeline(DiffusionPipeline, FluxLoraLoaderMixin, FluxIPAdapterMixin):