Skip to content

Multi-image masking for single IP Adapter - #7499

Merged
yiyixuxu merged 12 commits into
huggingface:mainfrom
fabiorigano:multiimageipmask
Apr 9, 2024
Merged

Multi-image masking for single IP Adapter#7499
yiyixuxu merged 12 commits into
huggingface:mainfrom
fabiorigano:multiimageipmask

Conversation

@fabiorigano

@fabioriganofabiorigano commented Mar 27, 2024

Copy link
Copy Markdown
Contributor

What does this PR do?

Image masking is currently supported by passing a mask and an image for each IP Adapter in the pipeline.

This PR extends this usage by allowing multiple masks and images to be passed to a single IP Adapter. Scales must be passed accordingly.

Discussed in #7419

@sayakpaul@yiyixuxu

@fabioriganofabiorigano changed the title [WIP] Multi-image support for single IP Adapter[WIP] Multi-image masking for single IP AdapterMar 27, 2024
sayakpaul
sayakpaul reviewed Mar 28, 2024
View reviewed changes
Comment threadsrc/diffusers/models/attention_processor.py Outdated
sayakpaul
sayakpaul reviewed Mar 28, 2024
View reviewed changes

@sayakpaulsayakpaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes looking nice ❤️ Great work as always. Let's add a fast test and see how it goes?

@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Mar 28, 2024

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@fabiorigano

fabiorigano commented Mar 31, 2024

Copy link
Copy Markdown
ContributorAuthor

thank you @sayakpaul

ready

@fabioriganofabiorigano changed the title [WIP] Multi-image masking for single IP AdapterMulti-image masking for single IP AdapterMar 31, 2024
sayakpaul
sayakpaul reviewed Apr 1, 2024
View reviewed changes
Comment threadsrc/diffusers/models/attention_processor.py Outdated
sayakpaul
sayakpaul reviewed Apr 1, 2024
View reviewed changes
Comment threadtests/pipelines/ip_adapters/test_ip_adapter_stable_diffusion.py
max_diff = numpy_cosine_similarity_distance(image_slice, expected_slice)
assert max_diff < 5e-4

def test_ip_adapter_multiple_masks_one_adapter(self):

@sayakpaulsayakpaulApr 1, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slow test. I was wondering if it's possible to add a fast test like

diffusers/tests/pipelines/test_pipelines_common.py

Line 238 in c2e8786

deftest_ip_adapter_single(self, expected_max_diff: float=1e-4, expected_pipe_slice=None):

?

@fabioriganofabioriganoApr 1, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I missed that

I'll add it now

@DN6
DN6 requested a review from yiyixuxuApril 1, 2024 07:21
@fabiorigano@sayakpaul
Update src/diffusers/models/attention_processor.py
b6fdc96
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
yiyixuxu
yiyixuxu reviewed Apr 3, 2024
View reviewed changes

@yiyixuxuyiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for working on this!
looks good. I left some feedbacks. let's make very clear what's the expected shape for ip_adapter_mask and scale

Comment threadsrc/diffusers/models/attention_processor.py
Comment threadsrc/diffusers/models/attention_processor.py Outdated
Comment threadsrc/diffusers/models/attention_processor.py Outdated
Comment threadsrc/diffusers/models/attention_processor.py Outdated
@fabiorigano

fabiorigano commented Apr 6, 2024

Copy link
Copy Markdown
ContributorAuthor

@yiyixuxu@sayakpaul now it's ready
thanks for your reviews

yiyixuxu
yiyixuxu approved these changes Apr 8, 2024
View reviewed changes

@yiyixuxuyiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!
I will run slow tests to make sure it does not break anything before merge

@yiyixuxuyiyixuxu self-assigned this Apr 8, 2024
@yiyixuxuyiyixuxu mentioned this pull request Apr 8, 2024
Closed
@fabiorigano

fabiorigano commented Apr 8, 2024

Copy link
Copy Markdown
ContributorAuthor

thank you YiYi!

I fixed the fast test for the panorama pipeline that was failing and the slow test

I ran all the slow tests related to image masking locally and they didn't break :)

@yiyixuxu
yiyixuxu merged commit a0cf607 into huggingface:mainApr 9, 2024
@fabiorigano
fabiorigano deleted the multiimageipmask branch April 9, 2024 19:21
sayakpaul added a commit that referenced this pull request Dec 23, 2024
@fabiorigano@sayakpaul@yiyixuxu
Multi-image masking for single IP Adapter (#7499)
cc4918f
* Support multiimage masking
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sayakpaulsayakpaulsayakpaul left review comments
@yiyixuxuyiyixuxuyiyixuxu approved these changes

Assignees

@yiyixuxuyiyixuxu

Labels

IPAdapter

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@fabiorigano@HuggingFaceDocBuilderDev@yiyixuxu@sayakpaul