SD3 IP-Adapter runtime checkpoint conversion

[guiyrt]

What does this PR do?

Fixes#9966.

Now IP-Adapter from InstantX/SD3.5-Large-IP-Adapter can be used directly, with checkpoint runtime conversion.

I also changed the structure to have the familiar _convert_ip_adapter_attn_to_diffusers and _convert_ip_adapter_image_proj_to_diffusers functions.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@hlky@yiyixuxu

[guiyrt]

Example output:

Inference code

importosimporttorchfrompathlibimportPathfromdiffusersimportStableDiffusion3Pipelinefromdiffusers.utilsimportload_imagefromtransformersimportSiglipVisionModel, SiglipImageProcessormodel_id="stabilityai/stable-diffusion-3.5-large"image_encoder_id="google/siglip-so400m-patch14-384"ip_adapter_id="InstantX/SD3.5-Large-IP-Adapter"feature_extractor=SiglipImageProcessor.from_pretrained(
image_encoder_id, torch_dtype=torch.bfloat16
)
image_encoder=SiglipVisionModel.from_pretrained(
image_encoder_id, torch_dtype=torch.bfloat16
)
pipe=StableDiffusion3Pipeline.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
feature_extractor=feature_extractor,
image_encoder=image_encoder,
)
# Load IP Adapterpipe.load_ip_adapter(ip_adapter_id, "ip-adapter.bin")
pipe.set_ip_adapter_scale(0.5)
pipe._exclude_from_cpu_offload.append("image_encoder")
pipe.enable_sequential_cpu_offload()
# Inputip_adapter_img=load_image("astronaut.jpg")
# please note that SD3.5 Large is sensitive to highres generation like 1536x1536image=pipe(
width=1024,
height=1024,
prompt="an astronaut on top of a waffle",
negative_prompt="lowres, low quality, worst quality",
num_images_per_prompt=1,
generator=torch.manual_seed(42),
ip_adapter_image=ip_adapter_img,
guidance_scale=6,
num_inference_steps=24,
).images[0]
image.save("result.jpg")

Prompt: "an astronaut on top of a waffle"

Image Prompt:

Output:

[hlky]

Thanks @guiyrt. We can also change the checkpoint in docs/examples back to the original. You can also go ahead and add Fixes #9966 to close the issue when this is merged.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[guiyrt]

Done and done, thanks for the review @hlky !

[guiyrt]

Updated PR accounting for #10728

[yiyixuxu]

thanks a lot everyone!!