Improve Variations feature by freezing parts of the image

[morganavr]

Variations is a great feature if you tried it. It could be even better if it could accept image mask to freeze parts of the image you don't want to be changed - this is what this feature proposal is about.

---

Possibly this feature could be implemented by providing mask image using code from this repo: https://github.com/basujindal/stable-diffusion/blob/main/optimizedSD/inpaint_gradio.py

Alternative solution is to generate a specific "noise array" with frozen pixels but I am not sure it's possible.
Thoughts on noise array are below.

---

Discussion about this new feature started in this PR:
Seed Fuzzing : #184 (comment)

Is it possible to construct such "noise array" from "seed fuzzing" PR that will change only a specific part of the image? It sounds like inpainting but it's a different type of inpainting (inside trained model without providing external source image).

While using that PR I noticed that very often I would love some part of the image to stay the same, and I could use brush and paint over parts of the image... :) So yeah, basically inpainting. But with this "seed fuzzing" feature. I imagine that separate PNG file with mask need to be created (mask is alpha values from 0 to 1, where 0 - don't change that part of the image, 1 - apply noise level specified by -v argument). Then script would convert that mask data into the information which pixels SD must not change on the final image.

If it is possible to map noise array to image pixels then this feature could be implemented.

img2img module already reads input image somehow and allows SD to generate variants.
Also there is mask feature (where you provide black and white image to specify areas to remove from the image).
They work somehow and inpainting (in some forms) is implemented in other forks of SD.
Is not these "facts" mean that there IS a way to create a match between pixels in the input image and SD internals?

[lstein]

Soon -- hopefully in days to a week -- this fork will let you provide both an initial image and a black and white mask that specifies which parts of the image the AI is allowed to paint over. You'll be able to take a photo of yourself with your best friend, mask him, and use a prompt to replace him with an image of King George. It's all part of Stable Diffusion, but isn't documented well and I just haven't had time to figure out how it works.

Any developer who is reading this and would like to take a crack at the inpainting implementation is very much welcome to.

[morganavr]

@lstein
I guess I poorly described this "txt2img inpainting" feature because you thought that I just created another Issue to ask for Inpainting. Let me highlight key differences between "txt2img inpainting (let's call it textpainting for short) and regular Inpainting mentioned in another Issue #68

Here's how textpainting works given this prompt:
"photo of a smiling woman" -S 777777777 -v 0.01 -mask mask_image.png

mask_image.png is a black image with white color in places where you want noise tensor generated by -v 0.01 to be identical to the initially generated noise for the -S 777777777. If noise is identical it means that generated image won't be changed in those areas of white color.

Therefore, textpainting would allow to produce new variants of the image where parts of the image is guaranteed to remain the same. With textpainting you don't need to provide initial image, only mask image. It's a big difference compared to inpainting because all images produced by this prompt "photo of a smiling woman" -S 777777777 -v 0.01 are already exist inside model.ckpt file and you just access them with a magic key (prompt + seed + args). I expect that textpainting will generate much better context aware image modification than in regular inpainting module where input image is forcefully injected into SD latent space.

---

As a quick experiment I tried to freeze part of the generated image (first 256 px) by decoding/encoding tensors back and forth to images and assigning original noise value but whole image was changed.

[morganavr]

Okay, this is how I think a mask should be applied to freeze parts of the image while generating new variants.
I got this runtime error with it though:

The size of tensor a (8) must match the size of tensor b (64) at non-singleton dimension 3

simplet2i.py#txt2img: