Flux Control(Depth/Canny) + Inpaint by affromero · Pull Request #10192 · huggingface/diffusers

affromero · 2024-12-11T16:19:24Z

What does this PR do?

The Flux pipeline for image inpainting using Flux-dev-Depth/Canny

importtorchfromdiffusersimportFluxControlInpaintPipelinefromdiffusers.models.transformersimportFluxTransformer2DModelfromtransformersimportT5EncoderModelfromdiffusers.utilsimportload_image, make_image_gridfromimage_gen_auximportDepthPreprocessor# https://github.com/huggingface/image_gen_auxfromPILimportImageimportnumpyasnppipe=FluxControlInpaintPipeline.from_pretrained(
"black-forest-labs/FLUX.1-Depth-dev",
torch_dtype=torch.bfloat16,
)
# use following lines if you have GPU constraints# ---------------------------------------------------------------transformer=FluxTransformer2DModel.from_pretrained(
"sayakpaul/FLUX.1-Depth-dev-nf4", subfolder="transformer", torch_dtype=torch.bfloat16
)
text_encoder_2=T5EncoderModel.from_pretrained(
"sayakpaul/FLUX.1-Depth-dev-nf4", subfolder="text_encoder_2", torch_dtype=torch.bfloat16
)
pipe.transformer=transformerpipe.text_encoder_2=text_encoder_2pipe.enable_model_cpu_offload()
# ---------------------------------------------------------------pipe.to("cuda")
prompt="a blue robot singing opera with human-like expressions"image=load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")
head_mask=np.zeros_like(image)
head_mask[65:580,300:642] =255mask_image=Image.fromarray(head_mask)
processor=DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image=processor(image)[0].convert("RGB")
output=pipe(
prompt=prompt,
image=image,
control_image=control_image,
mask_image=mask_image,
num_inference_steps=30,
strength=0.9,
guidance_scale=50.0,
generator=torch.Generator().manual_seed(42),
).images[0]
make_image_grid([image, control_image, mask_image, output.resize(image.size)], rows=1, cols=4).save("output.png")

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@sayakpaul @yiyixuxu

Following this interaction about contributing this pipeline to main.

HuggingFaceDocBuilderDev · 2024-12-12T06:51:23Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul

Thanks for starting this! I have left some comments, LMK if they make sense.

sayakpaul · 2024-12-12T06:49:55Z

+ else:
+ masked_image_latents = retrieve_latents(self.vae.encode(masked_image), generator=generator)
+
+ masked_image_latents = (masked_image_latents - self.vae.config.shift_factor) * self.vae.config.scaling_factor


We could allow passing precomputed masked_image_latents but this is not a blocker for this PR.

sayakpaul · 2024-12-12T06:52:20Z

+ if masked_image_latents is not None:
+ masked_image_latents = masked_image_latents.to(latents.device)
+ else:
+ image = self.image_processor.preprocess(image, height=height, width=width)
+ mask_image = self.mask_processor.preprocess(mask_image, height=height, width=width)
+
+ masked_image = image * (1 - mask_image)
+ masked_image = masked_image.to(device=device, dtype=prompt_embeds.dtype)
+
+ height, width = image.shape[-2:]
+ mask, masked_image_latents = self.prepare_mask_latents(
+ mask_image,
+ masked_image,
+ batch_size,
+ num_channels_latents,
+ num_images_per_prompt,
+ height,
+ width,
+ prompt_embeds.dtype,
+ device,
+ generator,
+ )
+ masked_image_latents = torch.cat((masked_image_latents, mask), dim=-1)


All of this could go to prepare_mask_latents.

affromero · 2024-12-12T20:34:42Z

I believe I've addressed all your comments, @sayakpaul @hlky. Please let me know if there's anything I might have missed.

hlky

Left a suggestion and a couple comments but good to go on my end. Thanks!

hlky · 2024-12-12T20:41:30Z

+ # def test_flux_different_prompts(self):
+ # pipe = self.pipeline_class(**self.get_dummy_components()).to(torch_device)
+
+ # inputs = self.get_dummy_inputs(torch_device)
+ # output_same_prompt = pipe(**inputs).images[0]
+
+ # inputs = self.get_dummy_inputs(torch_device)
+ # inputs["prompt_2"] = "a different prompt"
+ # output_different_prompts = pipe(**inputs).images[0]
+
+ # max_diff = np.abs(output_same_prompt - output_different_prompts).max()
+
+ # # Outputs should be different here
+ # # For some reasons, they don't show large differences
+ # assert max_diff > 1e-6


#10025 (comment)
Possibly related, here's the effect it had on Redux #10056 (comment)

Co-authored-by: hlky <hlky@hlky.ac>

asomoza · 2024-12-13T00:01:29Z

it works ok, I still have to see if it works for some more real use cases, but it produces the expected results.

a yellow martian

Funny that it generated this "unknown logo" with this example ^^

hlky · 2024-12-13T09:30:22Z

@affromero here's a couple example PRs for documentation https://github.com/huggingface/diffusers/pull/10021/files https://github.com/huggingface/diffusers/pull/10131/files files under docs/. Ping @stevhliu if you need help with the docs or when they're ready for review.

affromero · 2024-12-18T03:23:49Z

I think this should be good, but not 100% sure @stevhliu, let me know what I can tweak or do next.

sayakpaul · 2024-12-18T03:28:32Z

Will let @hlky take care of the final merging.

hlky · 2024-12-18T09:14:09Z

Failing test is unrelated, good to go, thanks @affromero!

vladmandic · 2024-12-18T13:48:41Z

please update src/diffusers/pipelines/auto_pipeline.py as well.

@yiyixuxu @sayakpaul @asomoza can we make this somehow part of the mandatory template? 90% of the time when any img2img or inpaint pipeline is added, autopipeline mapping is NOT updated.

* flux_control_inpaint - failing test_flux_different_prompts * removing test_flux_different_prompts? * fix style * fix from PR comments * fix style * reducing guidance_scale in demo * Update src/diffusers/pipelines/flux/pipeline_flux_control_inpaint.py Co-authored-by: hlky <hlky@hlky.ac> * make * prepare_latents is not copied from * update docs * typos --------- Co-authored-by: affromero <ubuntu@ip-172-31-17-146.ec2.internal> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: hlky <hlky@hlky.ac>

affromero added 2 commits December 11, 2024 15:59

flux_control_inpaint - failing test_flux_different_prompts
a48baf0

removing test_flux_different_prompts?
0643318

a-r-r-o-w requested review from asomoza, hlky and sayakpaul December 11, 2024 22:55

Merge branch 'main' into flux_guidancecontrol_inpaint
05d5788

sayakpaul reviewed Dec 12, 2024
View reviewed changes

hlky reviewed Dec 12, 2024
View reviewed changes

Comment threadsrc/diffusers/pipelines/flux/pipeline_flux_control_inpaint.py Outdated
Comment threadsrc/diffusers/pipelines/flux/pipeline_flux_control_inpaint.py Outdated
Comment threadsrc/diffusers/pipelines/flux/pipeline_flux_control_inpaint.py Outdated

affromeroand others added 3 commits December 12, 2024 21:28

fix style
56f5b79

fix from PR comments
c3bc32e

Merge branch 'main' into flux_guidancecontrol_inpaint
30e6db6

fix style
83662a6

hlky approved these changes Dec 12, 2024
View reviewed changes

affromeroand others added 2 commits December 12, 2024 21:46

reducing guidance_scale in demo
7eaec16

Update src/diffusers/pipelines/flux/pipeline_flux_control_inpaint.py
387d48d
Co-authored-by: hlky <hlky@hlky.ac>

yiyixuxu added the close-to-merge label Dec 12, 2024

hlky added 2 commits December 12, 2024 21:07

make
105f1f7

prepare_latents is not copied from
a88265c

asomoza approved these changes Dec 13, 2024
View reviewed changes

update docs
6ddf106

Merge branch 'main' into flux_guidancecontrol_inpaint
4931ac4

typos
17f9cc2

hlky merged commit 83709d5 into huggingface:mainDec 18, 2024

hlky mentioned this pull request Dec 18, 2024
Add Flux Control to AutoPipeline #10292
Merged

Uh oh!

Conversation

affromero commented Dec 11, 2024

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Dec 12, 2024

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaulDec 12, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaulDec 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

affromero commented Dec 12, 2024

Uh oh!

hlky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hlkyDec 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asomoza commented Dec 13, 2024

Uh oh!

hlky commented Dec 13, 2024

Uh oh!

affromero commented Dec 18, 2024

Uh oh!

sayakpaul commented Dec 18, 2024

Uh oh!

hlky commented Dec 18, 2024

Uh oh!

vladmandic commented Dec 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants