[Stable Diffusion Inpainting] Allow standard text-to-img checkpoints to be useable for SD inpainting

[patrickvonplaten]

This PR allows to use StableDiffusionControlNetInpaintPipeline and StableDiffusionInpaintPipeline for both inpainting models and normal text-to-image models. This has two purposes:

1.)

We can completely remove **StableDiffusionInpaintLegacyPipeline**. People will still want to use one and the same model weights for text2img, img2img, and inpainting so we need to allow the following scenario:

fromdiffusersimport (
StableDiffusionPipeline,
StableDiffusionImg2ImgPipeline,
StableDiffusionInpaintPipeline,
)
text2img=StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
img2img=StableDiffusionImg2ImgPipeline(**text2img.components)
inpaint=StableDiffusionInpaintPipeline(**text2img.components)

It's also stated in our official docs, here:
https://huggingface.co/docs/diffusers/main/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.components.example

With this PR the standard inpainting pipeline is augmented to allow for inpainting with sd-v1-5 (the text2image checkpoint, not the inpainting) which I think removes some barries. It was also very much a requested feature:

• 'StableDiffusionInpaintPipeline' has no attribute 'from_ckpt' #3497 (comment)
• ControlNet v1.1 #3095 (comment)

2.)

We need this PR to fully support the ControlNet inpainting model: https://huggingface.co/lllyasviel/control_v11p_sd15_inpaint

This checkpoint works very well with the new inpaint pipeline:

# !pip install transformers acceleratefromdiffusersimportStableDiffusionControlNetInpaintPipeline, ControlNetModel, DDIMSchedulerfromdiffusers.utilsimportload_imageimportnumpyasnpimporttorchinit_image=load_image(
"https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy.png"
)
init_image=init_image.resize((512, 512))
generator=torch.Generator(device="cpu").manual_seed(1)
mask_image=load_image(
"https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy_mask.png"
)
mask_image=mask_image.resize((512, 512))
defmake_inpaint_condition(image, image_mask):
image=np.array(image.convert("RGB")).astype(np.float32) /255.0image_mask=np.array(image_mask.convert("L")).astype(np.float32) /255.0assertimage.shape[0:1] ==image_mask.shape[0:1], "image and image_mask must have the same image size"image[image_mask>0.5] =-1.0# set as masked pixelimage=np.expand_dims(image, 0).transpose(0, 3, 1, 2)
image=torch.from_numpy(image)
returnimagecontrol_image=make_inpaint_condition(init_image, mask_image)
controlnet=ControlNetModel.from_pretrained(
"lllyasviel/control_v11p_sd15_inpaint", torch_dtype=torch.float16
)
pipe=StableDiffusionControlNetInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
# speed up diffusion process with faster scheduler and memory optimizationpipe.scheduler=DDIMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
# generate imageimage=pipe(
"a handsome man with ray-ban sunglasses",
num_inference_steps=20,
generator=generator,
guidance_scale=9.0,
eta=1.0,
image=init_image,
mask_image=mask_image,
control_image=control_image,
).images[0]

becomes:

Also, I also played around with other SAM to generate masks and then use our SAM ControlNet model: https://colab.research.google.com/drive/1D6mBtne_m-3E9R-cl_ZCRY87Fq1OjkyI?usp=sharing

cc @sayakpaul this could be useful for a diffusers tool, but had very limited success here tbh.
=> very limited success, the model doesn't seem to work super well. It seems like we either need a ControlNet checkpoint that is purely made for inpainting or use the 9-channel inpainting models.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[wangdong2023]

nice, was looking for this functionality yesterday.

[patrickvonplaten]

Let's fully deprecate this class. Now "normal" inpaint has all features that are needed.

[patrickvonplaten]

Will update README of: https://huggingface.co/lllyasviel/control_v11p_sd15_inpaint once this PR is merged

[williamberman]

lgtm!

[rb-synth]

Looking at the example, the control image is just the masked original. How can I use e.g., normalbae with inpainting?

[ksai2324]

I tried using the pipeline with lllyasviel/sd-controlnet-canny , but it didn't work. It just delivered a random noise in the masked part.
I'm still trying to understand why.
@rb-synth: did you manage to run it with normalbae?