Skip to content

[IP-Adapter] Adding IP-adapter masking feature #6802

Description

@yiyixuxu

We just added multi-adapter and multi-image support for IP-Adapter #6573

Anyone interested in adding an IP adapter masking feature? It will allow you to divide the image and assign different ip-adapter to the specified portion of the image.

more information on the feature

notes on the design

  • We can add community pipelines (text-to-image and ControlNet) that use multi-ip-adapter with mask
  • or, we can integrate this feature into core diffusers if we can add this feature without adding additional code to our pipelines. Some thoughts on how to achieve this:
    1. we can add utility functions to pre-process IP-adapter mask to our image processor class
      classVaeImageProcessor(ConfigMixin):
    2. the pre-processed masked can be passed down to IPAdapterAttnProcessor/IPAdapterAttnProcessor2.0 via cross_attention_kwargs
    3. modify the IPAdapterAttnProcessor/IPAdapterAttnProcessor2.0 so it can handle mask

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions