Unmasked areas are degraded during inpainting.

[oniatsu]

Describe the bug

Unmasked areas are degraded during inpainting.

Easy to see the difference under the bench. (This is a GIF animation)

Is this a bug?
Or how can I generate an image while keeping the original image except for the mask?

Reproduction

I tried inpaint according to the following documentation
Text-guided image-inpainting

The same results were obtained for both the method by code and the method by HuggingFace Space.

Logs

No response

System Info

HuggingFace Space

or

Google Colab (!pip install --upgrade diffusers[torch] transformers)

- `diffusers` version: 0.17.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Huggingface_hub version: 0.15.1
- Transformers version: 4.30.2
- Accelerate version: 0.20.3
- xFormers version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Who can help?

@patrickvonplaten @sayakpaul @williamberman

[Zhuhh0311]

I had the same problem, I used ControlNet +SD (StableDiffusionControlNetInpaintPipeline) to inpaint image, the unmasked areas are degraded.
I tried inpaint according to the code, but the same results were obtained

[patrickvonplaten]

Hey @oniatsu,

You are right that unmasked areas can change using the official inpainting pipeline, but this is because of the way it has been trained.

The problem is that if we force the unmasked area to stay 100% the same there will be inconsistencies in the transition to the masked area. E.g. compare the following two outptus:

1. With the current pipeline:
   

2. Now we force the unmasked area to not change:
   

You can see that the second picture has weird transitions to the generated dog.

Nevertheless I opened a PR for how we could solve it, but I'm really not sure we need it at the moment. Happy to finish the PR though if others strongly feel like we need it (cc @sayakpaul @yiyixuxu @pcuenca @takuma104 @dg845 maybe)

[patrickvonplaten]

Draft PR here: #3893

[oniatsu]

@patrickvonplaten Thanks!

I could see the difference in your sample images.
I think it is important to explain why the functionality of unmasked areas not being changed is necessary.

For example, if someone uses inpainting in their work, the ability to keep unmasked areas unchanged is important, because unmasked areas are not required to be modified.
This is because there will be work requirements that unmasked areas will not be changed.
In addition, if the inpainting process needs to be repeated multiple times, the changes will be more deviated from the original image and will not be able to withstand the demands.

I'd like to help with the PR for that, as I believe it's a necessary feature, but it seemed difficult to understand right away.

So now I am enforcing it myself by adding a process to replace unmasked areas with the prior original image.
I realize this is a bad pattern, of course.

from PIL import Image
import cv2
import numpy as np

# A generated image of unmasked areas by diffusers is changed a little bit
# So we need to replace the unmasked areas with the original image
def fix_converted_image(
    original_image: Image.Image, generated_image: Image.Image, mask_image: Image.Image
) -> Image.Image:
    # PIL.Image to numpy
    original_image_array = np.array(original_image)
    generated_image_array = np.array(generated_image)

    # invert mask
    mask_image_array = np.array(mask_image)
    mask_image_inverted = cv2.bitwise_not(mask_image_array)

    # replace pixels
    converted_image_array = np.where(
        mask_image_inverted == 255, original_image_array, generated_image_array
    )

    # numpy to PIL.Image
    converted_image = Image.fromarray(converted_image_array)
    return converted_image

[ayvbrg]

Hi, at dashtoon we use a custom inpaint pipeline which gives results that are similar to the results given by automatic1111 UI implementation of diffusers. Here also the un-masked are is not degraded and the images generated are better than the current inpaint implementation (if you only want to change the masked pixels). @patrickvonplaten would you be interested in a PR for a community pipeline for this

[patrickvonplaten]

Yes this sounds like a great idea for a community pipeline :-) (is this the same as #3921 ?)

[ayvbrg]

It is not exactly similar, automatic1111 does a few postprocessing and preprocessing methods to both the mask and images. It does have a latent mode in inpaint which i suspect could be similar to #3921, we have that implemented in diffusers but just never tried it out

[dg845]

Hi @oniatsu, so I can understand the issue better, is your requirement that the unmasked area remain exactly the same, or is it only required not to be degraded too much?

If the former is true, I think that having a force_unmasked_unchanged argument like in #3893 would be valuable, since models like Blended Latent Diffusion (#3921) do not guarantee that the unmasked area remains unchanged (see Section 4 of the paper). (Not sure if this is true of @benihime91's model.)

[AbdallaMohammed]

Hi @dg845, having a new argument in the existing pipeline is great, and easy-peasy solution also great for many cases, but why not implementing the model proposed in the paper in a stand-alone pipeline since even it doesn't guarantee that the unmasked area remains unchanged but it provides good approaches (see Section 4.2, 4.3).

@benihime91 maybe we can collaborate on (somehow) merging the proposed model and your dashtoon postprocessing and preprocessing techniques?

[dg845]

@AbdallaMohammed I think adding the Blended Latent Diffusion model would be a great addition to the library, and don't think that it should be blocked because it doesn't guarantee that the unmasked area remains unchanged.

My opinion is that if there is a use case that requires the unmasked area to remain absolutely unchanged, the new model wouldn't satisfy that use case on its own, and adding something like force_unmasked_unchanged as an argument to inpainting pipelines (in addition to new pipelines) would be useful in order to satisfy that use case.

[oniatsu]

@dg845 Thanks!

Is it required that the unmasked areas be exactly the same, or just not too degraded?

I'd prefer the former.
In my opinion, too, it would be better and more convenient to use if it could be specified as an option with an argument like force_unmasked_unchanged.

[patrickvonplaten]

Anybody interested in opening a PR here?

[dg845]

@patrickvonplaten, I can open a PR based on #3893 / finish #3893 to add a force_unmasked_unchanged argument if you think that would be a good idea :).

[haoqiangyu]

Hi @dg845, having a new argument in the existing pipeline is great, and easy-peasy solution also great for many cases, but why not implementing the model proposed in the paper in a stand-alone pipeline since even it doesn't guarantee that the unmasked area remains unchanged but it provides good approaches (see Section 4.2, 4.3).

@benihime91 maybe we can collaborate on (somehow) merging the proposed model and your dashtoon postprocessing and preprocessing techniques?

I don't think this script is any different than the StableDiffusionInpaintPipeline except for this line below, am I missing something

https://github.com/omriav/blended-latent-diffusion/blob/848eb0628a5c812dae3d0bbafc90c593c90de3f7/scripts/text_editing_stable_diffusion.py#L141