add PAG support sd15 controlnet by tuanh123789 · Pull Request #8820 · huggingface/diffusers

tuanh123789 · 2024-07-09T17:58:11Z

What does this PR do?

Adds PAG (Perturbed-Attention Guidance) support for SD 1.5 with controlnet models (StableDiffusionControlNetPAGPipeline)
Continuation of #8710

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Sample using SD1.5 controlnet	Sample SD1.5 controlnet PAG

from diffusers import AutoPipelineForText2Image, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch
import cv2
from PIL import Image
# download an image
image = load_image(
"https://hf.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png"
)
image = np.array(image)
# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)
# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = AutoPipelineForText2Image.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16, enable_pag=True
)
# speed up diffusion process with faster scheduler and memory optimization
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()
pipe.enable_model_cpu_offload()
# generate image
generator = torch.manual_seed(0)
image = pipe(
"aerial view, a futuristic research complex in a bright foggy jungle, hard lighting",
guidance_scale=7.5,
generator=generator,
image=canny_image,
pag_scale=10,
).images[0]

tuanh123789 · 2024-07-09T18:00:44Z

Hi @a-r-r-o-w I open new PR because I cannot undo deleted pipelines. I have edited it according to your comment, please review again, thank you

a-r-r-o-w

Looking great. Just small changes remaining and this should be good for merge.

HuggingFaceDocBuilderDev · 2024-07-10T09:20:20Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

a-r-r-o-w · 2024-07-10T19:42:19Z

Thank you for addressing the comments! Could we fix the failing tests?

Error logs

...
FAILED tests/pipelines/pag/test_pag_controlnet_sd.py::StableDiffusionControlNetPAGPipelineFastTests::test_pag_cfg - AssertionError: output is different from expected, [0.45505235 0.2785938 0.16334778 0.79689944 0.53095645 0.40135607
0.7052706 0.69065094 0.41548574]
assert 0.3865616192955017 < 0.001
FAILED tests/pipelines/pag/test_pag_controlnet_sd.py::StableDiffusionControlNetPAGPipelineFastTests::test_pag_uncond - AssertionError: output is different from expected, [0.45127502 0.2797252 0.15970308 0.7993157 0.5414344 0.40160775
0.7114598 0.69803864 0.4217583 ]
assert 0.39445382411422725 < 0.001
FAILED tests/pipelines/pag/test_pag_controlnet_sd.py::StableDiffusionControlNetPAGPipelineFastTests::test_save_load_optional_components - KeyError: 'tokenizer_2'

Let me know if you need any help. We can merge as soon as these are fixed since everything else looks good to me!

tuanh123789 · 2024-07-10T20:11:33Z

Sure can you help me on tests 🤗

a-r-r-o-w · 2024-07-10T20:23:08Z

Sure can you help me on tests 🤗

For the first two test failures, you will have to update the expected values to what is the actual value of image_slice is. You can print the values of image_slice and replace expected_slice with that 😜

For the third test, you need to remove SDXLOptionalComponentsTesterMixin in the fast tests class.

yiyixuxu

thanks!

yiyixuxu · 2024-07-11T20:48:04Z

+ # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents
+ def decode_latents(self, latents):
+ deprecation_message = "The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) instead"
+ deprecate("decode_latents", "1.0.0", deprecation_message, standard_warn=False)
+
+ latents = 1 / self.vae.config.scaling_factor * latents
+ image = self.vae.decode(latents, return_dict=False)[0]
+ image = (image / 2 + 0.5).clamp(0, 1)
+ # we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16
+ image = image.cpu().permute(0, 2, 3, 1).float().numpy()
+ return image


Suggested change
# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents
defdecode_latents(self, latents):
deprecation_message="The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) instead"
deprecate("decode_latents", "1.0.0", deprecation_message, standard_warn=False)
latents=1/self.vae.config.scaling_factor*latents
image=self.vae.decode(latents, return_dict=False)[0]
image= (image/2+0.5).clamp(0, 1)
# we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16
image=image.cpu().permute(0, 2, 3, 1).float().numpy()
returnimage
I don't think it is used

yiyixuxu · 2024-07-11T20:49:24Z

+ extra_step_kwargs["generator"] = generator
+ return extra_step_kwargs
+
+ def check_inputs(


should we add a #Copied from here

I don't think we have a version of check_inputs with callback_steps removed, and this is the first of its kind

yiyixuxu · 2024-07-11T20:52:27Z

feel free to merge once you're happy with it! @a-r-r-o-w

tuanh123789 · 2024-07-12T02:42:26Z

@a-r-r-o-w Can you help me with last fail of test 🤗

a-r-r-o-w · 2024-07-12T08:31:54Z

@tuanh123789 The failing test is unrelated and nothing needs to be done on your end. I'm happy to merge this once you address YiYi's comment: #8820 (comment)

tuanh123789 · 2024-07-12T08:34:54Z

@tuanh123789 The failing test is unrelated and nothing needs to be done on your end. I'm happy to merge this once you address YiYi's comment: #8820 (comment)

:v Sure, will be done right away

a-r-r-o-w

Thanks, looking good! One other thing that I just realised - could you update the pipelines doc page with this SD15 Controlnet? https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/pag.md. Please see #7944 for an example

tuanh123789 · 2024-07-12T09:19:14Z

Thanks, looking good! One other thing that I just realised - could you update the pipelines doc page with this SD15 Controlnet? https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/pag.md. Please see #7944 for an example

Done

crapthings · 2024-07-14T03:10:16Z

pip install git+https://github.com/huggingface/diffusers

ImportError: cannot import name 'StableDiffusionControlNetPAGPipeline' from 'diffusers'

pip list

diffusers 0.30.0.dev0

how can i test?

crapthings · 2024-07-14T03:20:10Z

how do i know if its working

fix seed and test?

pip uninstall diffusers
pip install git+https://github.com/huggingface/diffusers

Bhavay-2001 · 2024-07-14T16:20:42Z

Hi @tuanh123789, could you please share the colab notebook you used to test the images which you have shown in the above output. Thanks

asomoza · 2024-07-14T19:46:17Z

fix seed and test?

yes, preferably with a complex generation that you can see pag helps to fix.

could you please share the colab notebook you used to test the images which you have shown in the above output

It's literally copy & paste the example from this PR, there's nothing else you have to do.

without pag	with pag

You could just paste everything in one cell block, but here's notebook with the steps to test it.

* add pag support sd15 controlnet * fix quality import * remove unecessary import * remove if state * fix tests * remove useless function * add sd1.5 controlnet pag docs --------- Co-authored-by: anhnct8 <anhnct8@fpt.com>

add pag support sd15 controlnet
059749b

fix quality import
e5de3c9

a-r-r-o-w reviewed Jul 10, 2024
View reviewed changes

Comment threadsrc/diffusers/pipelines/__init__.py Outdated
Comment threadsrc/diffusers/pipelines/pag/pipeline_pag_controlnet_sd.py Outdated

tuanh123789 added 2 commits July 10, 2024 23:39

remove unecessary import
de755da

remove if state
e1cdf2c

tuanh123789 requested a review from a-r-r-o-w July 10, 2024 16:41

fix tests
af21256

yiyixuxu approved these changes Jul 11, 2024
View reviewed changes

remove useless function
a010b4b

a-r-r-o-w approved these changes Jul 12, 2024
View reviewed changes

add sd1.5 controlnet pag docs
f8d3a66

a-r-r-o-w merged commit d704b3b into huggingface:mainJul 12, 2024

yiyixuxu added the PAG label Sep 4, 2024

-# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents
-defdecode_latents(self, latents):
-deprecation_message="The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) instead"
-deprecate("decode_latents", "1.0.0", deprecation_message, standard_warn=False)
-latents=1/self.vae.config.scaling_factor*latents
-image=self.vae.decode(latents, return_dict=False)[0]
-image= (image/2+0.5).clamp(0, 1)
-# we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16
-image=image.cpu().permute(0, 2, 3, 1).float().numpy()
-returnimage

Uh oh!

Conversation

tuanh123789 commented Jul 9, 2024

What does this PR do?

Before submitting

Who can review?

Uh oh!

tuanh123789 commented Jul 9, 2024

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jul 10, 2024

Uh oh!

a-r-r-o-w commented Jul 10, 2024

Uh oh!

tuanh123789 commented Jul 10, 2024

Uh oh!

a-r-r-o-w commented Jul 10, 2024

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

yiyixuxuJul 11, 2024

Choose a reason for hiding this comment

Uh oh!

yiyixuxuJul 11, 2024

Choose a reason for hiding this comment

Uh oh!

a-r-r-o-wJul 11, 2024

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Jul 11, 2024

Uh oh!

tuanh123789 commented Jul 12, 2024

Uh oh!

a-r-r-o-w commented Jul 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tuanh123789 commented Jul 12, 2024

Uh oh!

a-r-r-o-w left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuanh123789 commented Jul 12, 2024

Uh oh!

crapthings commented Jul 14, 2024

Uh oh!

crapthings commented Jul 14, 2024

Uh oh!

Bhavay-2001 commented Jul 14, 2024

Uh oh!

asomoza commented Jul 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

a-r-r-o-w commented Jul 12, 2024•
edited
Loading

a-r-r-o-w left a comment •
edited
Loading