Move IP Adapter Face ID to core by fabiorigano · Pull Request #7186 · huggingface/diffusers

fabiorigano · 2024-03-02T15:17:50Z

What does this PR do?

Switch to PEFT
Move to core
Add tests
Add docs

Create face embeddings

importtorchimportcv2importnumpyasnpfromdiffusers.utilsimportload_imagefromdiffusersimportAutoPipelineForText2Image, AutoencoderKL, DDIMSchedulerfrominsightface.appimportFaceAnalysisimage1=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/ai_face2.png")
image2=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/women_input.png")
ref_images_embeds= []
ref_unc_images_embeds= []
app=FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))
forimin [image1, image2]:
image=cv2.cvtColor(np.asarray(im), cv2.COLOR_BGR2RGB)
faces=app.get(image)
image=torch.from_numpy(faces[0].normed_embedding)
image_embeds=image.unsqueeze(0)
uncond_image_embeds=torch.zeros_like(image_embeds)
ref_images_embeds.append(image_embeds)
ref_unc_images_embeds.append(uncond_image_embeds)
ref_images_embeds=torch.stack(ref_images_embeds, dim=0)
ref_unc_images_embeds=torch.stack(ref_unc_images_embeds, dim=0)
single_image_embeds=torch.cat([ref_unc_images_embeds, ref_images_embeds], dim=0).to(dtype=torch.float16)

IP Adapter Face ID (SD 1.5)

base_model_path="SG161222/Realistic_Vision_V4.0_noVAE"noise_scheduler=DDIMScheduler(
num_train_timesteps=1000,
beta_start=0.00085,
beta_end=0.012,
beta_schedule="scaled_linear",
clip_sample=False,
set_alpha_to_one=False,
steps_offset=1,
)
pipeline=AutoPipelineForText2Image.from_pretrained(
base_model_path,
torch_dtype=torch.float16,
scheduler=noise_scheduler
)
pipeline.to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter-FaceID",
subfolder=None, weight_name="ip-adapter-faceid_sd15.bin", image_encoder_folder=None)
pipeline.set_ip_adapter_scale(0.7)
pipeline.enable_model_cpu_offload()
generator=torch.Generator(device="cpu").manual_seed(42)
num_images=2images=pipeline(
prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
ip_adapter_image=[single_image_embeds],
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_inference_steps=20, num_images_per_prompt=num_images, width=512, height=704, generator=generator, ).images

IP Adapter Face ID XL (SDXL)

base_model_path="SG161222/RealVisXL_V3.0"pipeline=AutoPipelineForText2Image.from_pretrained(
base_model_path,
torch_dtype=torch.float16,
scheduler=noise_scheduler
)
pipeline.to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter-FaceID",
subfolder=None, weight_name="ip-adapter-faceid_sdxl.bin", image_encoder_folder=None)
pipeline.set_ip_adapter_scale(0.7)
pipeline.enable_model_cpu_offload()
generator=torch.Generator(device="cpu").manual_seed(42)
num_images=2images=pipeline(
prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
ip_adapter_image=[single_image_embeds], guidance_scale=7.5,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_inference_steps=30, num_images_per_prompt=2,
generator=generator
).images

sayakpaul

Refreshing change!

jfischoff · 2024-03-09T01:42:48Z

I'm getting the error

table_diffusion_xl.py", line 497, in encode_image
AttributeError: 'NoneType' object has no attribute 'parameters'

when I try to use this.

fabiorigano · 2024-03-09T07:35:33Z

I'm getting the error

table_diffusion_xl.py", line 497, in encode_image
AttributeError: 'NoneType' object has no attribute 'parameters'

when I try to use this.

you cannot use Face ID with SDXL, the current changes only affect the Stable Diffusion pipeline

fabiorigano · 2024-03-09T17:16:53Z

@jfischoff you can use it now, I also updated the example code

yiyixuxu · 2024-03-09T18:08:03Z

@fabiorigano is this ready for a review?

fabiorigano · 2024-03-09T18:42:00Z

@yiyixuxu I have to add some checks on the inputs, but I would appreciate your feedback. thanks :)

fabiorigano · 2024-03-09T19:53:56Z

Since both Face ID adapter and Face ID XL don't use an image encoder, I tested the multi-adapter feature by separately extracting and then concatenating the image embeddings of Face ID XL and another IP Adapter, Plus Face SDXL.

I think that prepare_ip_adapter_image_embeds would become too specific for this use case if we have to support an ip_adapter_images containing both images and insightface embeddings.

Here it is the code of the test:

# Create a SDXL pipeline# ...# Load sample imagesimage1=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/ai_face2.png")
image2=load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/women_input.png")
# Extract Face features using insightfaceref_images= []
app=FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))
forimin [image1, image2]:
image=cv2.cvtColor(np.asarray(im), cv2.COLOR_BGR2RGB)
faces=app.get(image)
image=torch.from_numpy(faces[0].normed_embedding)
ref_images.append(image.unsqueeze(0))
ref_images=torch.cat(ref_images, dim=0)
# Load Face ID XL adapter into the pipelinepipeline.load_ip_adapter("h94/IP-Adapter-FaceID", subfolder=None, weight_name="ip-adapter-faceid_sdxl.bin", image_encoder_folder=None
)
# Generate Face ID image embeddings and save them locallyimage_embeds=pipeline.prepare_ip_adapter_image_embeds(
ip_adapter_image=ref_images,
ip_adapter_image_embeds=None,
device="cuda",
num_images_per_prompt=1,
do_classifier_free_guidance=True,
)
torch.save(image_embeds, "faceid_xl.ipadpt")
# Unload ip adapter and lora# ...# Load Plus SDXL adapter into the pipelinepipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter-plus-face_sdxl_vit-h.safetensors")
# Generate Plus SDXL image embeddings and save them locallyip_images=[[image1, image2]]
image_embeds=pipeline.prepare_ip_adapter_image_embeds(
ip_adapter_image=ip_images,
ip_adapter_image_embeds=None,
device="cuda",
num_images_per_prompt=1,
do_classifier_free_guidance=True,
)
torch.save(image_embeds, "plus_face_xl.ipadpt")
# Unload the IP adapter# ...# Load both IP Adapterspipeline.load_ip_adapter(["h94/IP-Adapter", "h94/IP-Adapter-FaceID"], subfolder=["sdxl_models", None], weight_name=["ip-adapter-plus-face_sdxl_vit-h.safetensors", "ip-adapter-faceid_sdxl.bin"]
)
pipeline.set_ip_adapter_scale([0.7]*2)
# Load image embeddings and run inferencegenerator=torch.Generator(device="cpu").manual_seed(42)
t1=torch.load("plus_face_xl.ipadpt")
t2=torch.load("faceid_xl.ipadpt")
t= [t1[0], t2[0]]
images=pipeline(
prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
ip_adapter_image_embeds=t, guidance_scale=7.5,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_inference_steps=30, num_images_per_prompt=num_images, width=1024, height=1024, generator=generator
).images

yiyixuxu · 2024-03-10T02:24:53Z

@fabiorigano

I think that prepare_ip_adapter_image_embeds would become too specific for this use case if we have to support an ip_adapter_images containing both images and insightface embeddings.

good news is that we do not want to support ip_adapter_image for face-id! :) we should make it clear in the docs

also, to make it easier to test, can you upload the ref_images embedding somewhere, maybe a hf dataset, so that we can just use that as input directly?

# Extract Face features using insightface
ref_images = []
app = FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))
for im in [image1, image2]:
image = cv2.cvtColor(np.asarray(im), cv2.COLOR_BGR2RGB)
faces = app.get(image)
image = torch.from_numpy(faces[0].normed_embedding)
ref_images.append(image.unsqueeze(0))
ref_images = torch.cat(ref_images, dim=0)

yiyixuxu · 2024-03-10T02:25:53Z

@fabiorigano

I tested the multi-adapter feature by separately extracting and then concatenating the image embeddings of Face ID XL and another IP Adapter, Plus Face SDXL

can you combine face-id with other ip-adaper models? I thought it required its own attention processor

fabiorigano · 2024-03-10T08:43:02Z

@fabiorigano
I tested the multi-adapter feature by separately extracting and then concatenating the image embeddings of Face ID XL and another IP Adapter, Plus Face SDXL
can you combine face-id with other ip-adaper models? I thought it required its own attention processor

@yiyixuxu I used PEFT to load the LoRA weights, so we don't need additional attention processors :)

fabiorigano · 2024-03-10T09:52:57Z

I uploaded some tensors here https://huggingface.co/datasets/fabiorigano/testing-images/tree/main

Some of my tests and the results (input image embeddings are computed from "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/ai_face2.png"):

Face ID SD 1.5 only

pipeline.load_ip_adapter("h94/IP-Adapter-FaceID", subfolder=None, weight_name="ip-adapter-faceid_sd15.bin", image_encoder_folder=None)
pipeline.set_ip_adapter_scale(0.6)
image_embeds=load_pt("https://huggingface.co/datasets/fabiorigano/testing-images/resolve/main/ai_face2.ipadpt")
images=pipeline(
prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
ip_adapter_image_embeds=image_embeds,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_inference_steps=20, num_images_per_prompt=1, width=512, height=704, generator=torch.Generator(device="cpu").manual_seed(0)
).images

Output image

Plus Face SD 1.5 only

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter-plus-face_sd15.bin")
pipeline.set_ip_adapter_scale(0.6)
image_embeds=load_pt("https://huggingface.co/datasets/fabiorigano/testing-images/resolve/main/clip_ai_face2.ipadpt")
images=pipeline(
prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
ip_adapter_image_embeds=image_embeds,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_inference_steps=20, num_images_per_prompt=1, width=512, height=704, generator=torch.Generator(device="cpu").manual_seed(0)
).images

Output image

Plus Face SD 1.5 + Face ID SD 1.5

pipeline.load_ip_adapter(["h94/IP-Adapter", "h94/IP-Adapter-FaceID"], subfolder=["models", None], weight_name=["ip-adapter-plus-face_sd15.safetensors", "ip-adapter-faceid_sd15.bin"])
pipeline.set_ip_adapter_scale([0.5, 0.5])
t1=load_pt("https://huggingface.co/datasets/fabiorigano/testing-images/resolve/main/clip_ai_face2.ipadpt")
t2=load_pt("https://huggingface.co/datasets/fabiorigano/testing-images/resolve/main/ai_face2.ipadpt")
image_embeds= [t1[0], t2[0]]
images=pipeline(
prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
ip_adapter_image_embeds=image_embeds,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_inference_steps=20, num_images_per_prompt=1, width=512, height=704, generator=torch.Generator(device="cpu").manual_seed(0)
).images

Output image

fabiorigano · 2024-03-11T08:05:25Z

@yiyixuxu it is ready for review

HuggingFaceDocBuilderDev · 2024-03-12T16:17:31Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu

thnaks!
I left some comments and questions

yiyixuxu · 2024-03-12T17:39:09Z

 logger.warning(
- "image_encoder is not loaded since `image_encoder_folder=None` passed. You will not be able to use `ip_adapter_image` when calling the pipeline with IP-Adapter."
- "Use `ip_adapter_image_embeds` to pass pre-generated image embedding instead."
+ "image_encoder is not loaded since `image_encoder_folder=None` passed. `ip_adapter_image` is allowed only if you are loading an IP-Adapter Face ID model."


a little bit confused here - I thought it was the opposite, i.e. we do not allow using ip_adapter_image with the Face ID model.

conceptually Face ID embeddings are image embeddings, but the tensor as it is doesn't have the unconditioned part, so in encode_image it is updated as is expected.
do you think it is better to leave this to the user?

yes - let's make it clear on the doc how to create the ip_adapter_image_embedding for face-id

yiyixuxu · 2024-03-12T19:53:01Z

+ ]
+ }
+ )
+ key_id += 1


are there more than one face-id checkpoints right now? does it make sense for us to support more than one?

Face ID and Face ID XL are both supported by this PR
Face ID Plus models have different image projection layers

sayakpaul · 2024-04-12T08:00:00Z

+ """Forward pass.
+
+ Args:
+ ----
+ id_embeds (torch.Tensor): Input Tensor (ID embeds).
+
+ Returns:
+ -------
+ torch.Tensor: Output Tensor.
+ """


I think this needs to follow our doc-string format?

ok, I will update it (also IPAdapterPlusImageProjection for code consistency)?

sayakpaul · 2024-04-12T08:02:34Z

+ nn.LayerNorm(embed_dims),
+ nn.LayerNorm(embed_dims),
+ Attention(
+ query_dim=embed_dims,
+ dim_head=dim_head,
+ heads=heads,
+ out_bias=False,
+ ),
+ nn.Sequential(
+ nn.LayerNorm(embed_dims),
+ FeedForward(embed_dims, embed_dims, activation_fn="gelu", mult=ffn_ratio, bias=False),
+ ),


I don't have strong opinions here but perhaps we could create a small block consisting of these layers and use that block here instead. Then
for ln0, ln1, attn, ff in self.layers: residual = latents encoder_hidden_states = ln0(x) latents = ln1(latents) encoder_hidden_states = torch.cat([encoder_hidden_states, latents], dim=-2) latents = attn(latents, encoder_hidden_states) + residual latents = ff(latents) + latents
could become:
forblockinself.blocks: ...
If the checkpoint needs to be rejigged to match this structure, we could have a load state dict hook to deal with the modifications. But I would wait for @yiyixuxu to comment further before making any changes.

nice but I don't think it is a big deal
if it requires a lot of effort from @fabiorigano I don't think it's worth it

Yeah totally fine by me. It was just a suggestion.

hi, I added IPAdapterPlusImageProjectionBlock, let me know if it works for you

sayakpaul · 2024-04-12T08:08:33Z

 # load ip-adapter into unet
 unet = getattr(self, self.unet_name) if not hasattr(self, "unet") else self.unet
- unet._load_ip_adapter_weights(state_dicts, low_cpu_mem_usage=low_cpu_mem_usage)
+ extra_loras = unet._load_ip_adapter_weights(state_dicts, low_cpu_mem_usage=low_cpu_mem_usage)


Interesting. To reduce the maintenance burden and to promote better readability, perhaps we could separate out the LoRA-related code from _load_ip_adapter_weights()?

sayakpaul

Thank you! Left a couple of comments.

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

…into faceidcore

sayakpaul · 2024-04-16T04:51:52Z

+ extra_loras = unet._load_ip_adapter_loras(state_dicts)
+ if extra_loras != {}:
+ # apply the IP Adapter Face ID LoRA weights
+ peft_config = getattr(unet, "peft_config", {})
+ for k, lora in extra_loras.items():
+ if f"faceid_{k}" not in peft_config:
+ self.load_lora_weights(lora, adapter_name=f"faceid_{k}")
+ self.set_adapters([f"faceid_{k}"], adapter_weights=[1.0])


sayakpaul · 2024-04-16T04:52:12Z

+ heads=heads,
+ id_embeddings_dim=id_embeddings_dim,
+ )
+ print(state_dict.keys())


Needs to go away.

sayakpaul · 2024-04-16T04:52:27Z

+ print(updated_state_dict.keys())
+ print(image_projection.state_dict().keys())


Needs to go away.

sayakpaul · 2024-04-16T04:54:16Z

 max_diff = numpy_cosine_similarity_distance(image_slice, expected_slice)
 assert max_diff < 5e-4

+ def test_text_to_image_face_id(self):


Can we add a fast test as well?

sayakpaul

The PR is looking quite nice to me. Thanks a lot for working on it. Also, do we need to add a check like so

diffusers/src/diffusers/loaders/lora.py

Line 108 in cf6e040

ifnotUSE_PEFT_BACKEND:

when there's a call to use the IP Adapter Face ID weights?

I will defer to @yiyixuxu to merge this. I would just run the concerned slow tests on our CI infrastructure as well to ensure nothing's breaking. @yiyixuxu could you do that before merging?

fabiorigano · 2024-04-16T04:59:35Z

Also, do we need to add a check like so
diffusers/src/diffusers/loaders/lora.py
Line 108 in cf6e040
ifnotUSE_PEFT_BACKEND:
when there's a call to use the IP Adapter Face ID weights?

I will add it

yiyixuxu · 2024-04-19T00:14:09Z

great work as always! thanks a lot :) @fabiorigano

* Switch to peft and multi proj layers * Move Face ID loading and inference to core --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

Switch to peft and multi proj layers
7d7fd83

asomoza mentioned this pull request Mar 2, 2024
[ip-adapter] fix problem using embeds with the plus version of ip adapters #7189
Merged

sayakpaul reviewed Mar 3, 2024
View reviewed changes

fabiorigano added 2 commits March 3, 2024 10:43

Merge branch 'main' into faceidcore
b92c16f

Move Face ID loading and inference to core
4e56997

Add support for Face ID XL
98a1aa4

fabiorigano added 3 commits March 10, 2024 15:41

Add checks
03d84fa

Add test
fe35a42

Fix style
bc016f4

fabiorigano changed the title ~~[WIP] Move IP Adapter Face ID to core~~Move IP Adapter Face ID to coreMar 10, 2024

Merge branch 'main' into faceidcore
acf0a4e

fabiorigano requested a review from sayakpaul March 11, 2024 08:03

fabiorigano added 2 commits March 12, 2024 18:22

Remove old pipeline
92bfe50

Fix copies
b8eed8d

yiyixuxu reviewed Mar 12, 2024
View reviewed changes

fabiorigano added 3 commits March 12, 2024 21:37

Fix loading for full face
ec539d0

Revert community pipeline delete
9521f89

Revert copies
aad463c

sayakpaul reviewed Apr 12, 2024
View reviewed changes

Comment threadsrc/diffusers/models/embeddings.py Outdated

sayakpaul reviewed Apr 12, 2024
View reviewed changes

fabioriganoand others added 8 commits April 12, 2024 20:43

Update docs/source/en/using-diffusers/loading_adapters.md
32d6943
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

Fix test and docs
dc0a5fb

Merge branch 'faceidcore' ofhttps://github.com/fabiorigano/diffusers…
5a7ec1d
…into faceidcore

Fix style
4073be8

Move lora loading to separate function
2b9d5a5

Add IPAdapterPlusImageProjectionBlock
c14f077

Fix style
5cab6da

Merge branch 'main' into faceidcore
8077c30

sayakpaul reviewed Apr 16, 2024
View reviewed changes

sayakpaul approved these changes Apr 16, 2024
View reviewed changes

fabiorigano added 4 commits April 16, 2024 20:22

Fix quality + add PEFT check
0e03446

Fix names
e6495a2

Add fast test
d7772e9

Fix style
f77739f

yiyixuxu merged commit b5c8b55 into huggingface:mainApr 19, 2024

fabiorigano deleted the faceidcore branch April 19, 2024 01:27

Honey-666 mentioned this pull request Apr 24, 2024
IP-Adapter FaceID PLus How to use questions #7766
Closed

This was referenced May 20, 2024
Make IPAdapter compatible to torch.compile#7988
Closed
IP-Adapter-FaceID and FaceID plus not support #6415
Closed

sayakpaul added a commit that referenced this pull request Dec 23, 2024
Move IP Adapter Face ID to core (#7186)
c5a2b97
* Switch to peft and multi proj layers * Move Face ID loading and inference to core --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

		print(updated_state_dict.keys())
		print(image_projection.state_dict().keys())

Uh oh!

Conversation

fabiorigano commented Mar 2, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Create face embeddings

IP Adapter Face ID (SD 1.5)

IP Adapter Face ID XL (SDXL)

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

jfischoff commented Mar 9, 2024

Uh oh!

fabiorigano commented Mar 9, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fabiorigano commented Mar 9, 2024

Uh oh!

yiyixuxu commented Mar 9, 2024

Uh oh!

fabiorigano commented Mar 9, 2024

Uh oh!

fabiorigano commented Mar 9, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxu commented Mar 10, 2024

Uh oh!

yiyixuxu commented Mar 10, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fabiorigano commented Mar 10, 2024

Uh oh!

fabiorigano commented Mar 10, 2024

Face ID SD 1.5 only

Plus Face SD 1.5 only

Plus Face SD 1.5 + Face ID SD 1.5

Uh oh!

fabiorigano commented Mar 11, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Mar 12, 2024

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

fabiorigano commented Mar 2, 2024•
edited
Loading

fabiorigano commented Mar 9, 2024•
edited
Loading

fabiorigano commented Mar 9, 2024•
edited
Loading

yiyixuxu commented Mar 10, 2024•
edited
Loading

sayakpaul left a comment •
edited
Loading