Support IPAdapters for FLUX pipelines

[chenxiao111222]

Model/Pipeline/Scheduler description

IPAdapter for FLUX is available now, do you have any plans to add IPAdapter to FLUX pipelines?

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

model implementation:

• https://github.com/XLabs-AI/x-flux/blob/main/src/flux/xflux_pipeline.py#L55

model weights:

• https://huggingface.co/XLabs-AI/flux-ip-adapter-v2
• https://huggingface.co/XLabs-AI/flux-ip-adapter

[sayakpaul]

Cc: @asomoza

#9787

[hj13-mtlab]

@sayakpaul flux ip adapter error, 'FluxTransformer2DModel' object has no attribute 'encoder_hid_proj'

[vladmandic]

@sayakpaul @yiyixuxu @asomoza as requested yesterday in a discussion, tagging this as a high-priority ask.
(if you'd prefer to use a different mechanism for tagging, let me know)

[sayakpaul]

Thanks for doing that, appreciated!

Last time I checked with @asomoza regarding this, the conclusion was the results aren't as promising but things may have changed. So, I will let @asomoza to comment further here.

[asomoza]

the v2 seems to be a lot better, the v1 has more downloads though, even though I don't see that much generations with IP Adapter, we should probably add it now since it seems better and Flux is still very popular.

I'll do some test with it but I'm guessing the reason why isn't used that much its because of the VRAM requirement, same as controlnets.

I'll take a look at the PR and see how much is needed to add it, but in the meantime let's open it up to the community if someone wants to take it.

[peki12345]

the v2 seems to be a lot better, the v1 has more downloads though, even though I don't see that much generations with IP Adapter, we should probably add it now since it seems better and Flux is still very popular.

I'll do some test with it but I'm guessing the reason why isn't used that much its because of the VRAM requirement, same as controlnets.

I'll take a look at the PR and see how much is needed to add it, but in the meantime let's open it up to the community if someone wants to take it.

Hi, may I ask if implementing flux IP adapter, can it support IP adapters with a given mask like SDXL? I think this is a very practical feature. I haven't seen similar functionality in other frameworks like ComfyUI, it seems to be unique to diffusers.

[yiyixuxu]

cc @fabiorigano here

[asomoza]

Hi, may I ask if implementing flux IP adapter, can it support IP adapters with a given mask like SDXL? I think this is a very practical feature. I haven't seen similar functionality in other frameworks like ComfyUI, it seems to be unique to diffusers.

Of course, when we add it, it should have the same functionality as the other ones. The ComfyUI ones also have this though and actually we used the ComfyUI implementation as a reference.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[vladmandic]

ping.