IP-Adapter masking

[stevhliu]

Continuing the issue from here about assigning a separate input image to each IP-Adapter without passing a mask. @sayakpaul suspects it's because the images need to have the exact same resolution. cc @yiyixuxu

Code to reproduce:

from diffusers.utils import load_image
from diffusers import AutoPipelineForText2Image
from PIL import Image
import torch

face_image1 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl1.png")
face_image2 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl2.png")
ip_images = [[face_image1], [face_image2]]

pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name=["ip-adapter-plus-face_sdxl_vit-h.safetensors"] * 2)
pipeline.set_ip_adapter_scale([0.7] * 2)
generator = torch.Generator(device="cpu").manual_seed(0)
num_images = 1

image = pipeline(
    prompt="2 girls",
    ip_adapter_image=ip_images,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
    num_inference_steps=20,
    num_images_per_prompt=num_images,
    generator=generator,
).images[0]
image

Error:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (514x1664 and 1280x1280)

[fabiorigano]

hi @stevhliu, when using SDXL checkpoints, you have to explicitly load the image encoder stored in the "models" folder:

from transformers import CLIPVisionModelWithProjection
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter", 
    subfolder="models/image_encoder",
    torch_dtype=torch.float16,
)
pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, image_encoder=image_encoder).to("cuda")

or with image_encoder_folder

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name=["ip-adapter-plus-face_sdxl_vit-h.safetensors"] * 2, image_encoder_folder="models/image_encoder")

[stevhliu]

Thank you! This is the way 👍