Add StableDiffusionXLControlNetPAGImg2ImgPipeline

[satani99]

What does this PR do?

fix#8700

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@yiyixuxu

[satani99]

Generation code

importtorchimportnumpyasnpfromPILimportImagefromtransformersimportDPTFeatureExtractor, DPTForDepthEstimationfromdiffusersimportControlNetModel, AutoencoderKL, AutoPipelineForImage2Imagefromdiffusers.utilsimportload_imagedepth_estimator=DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to("cuda")
feature_extractor=DPTFeatureExtractor.from_pretrained("Intel/dpt-hybrid-midas")
controlnet=ControlNetModel.from_pretrained(
"diffusers/controlnet-depth-sdxl-1.0-small",
variant="fp16",
use_safetensors="True",
torch_dtype=torch.float16,
)
vae=AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe=AutoPipelineForImage2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
controlnet=controlnet,
vae=vae,
variant="fp16",
use_safetensors=True,
torch_dtype=torch.float16,
enable_pag=True,
)
pipe.enable_model_cpu_offload()
defget_depth_map(image):
image=feature_extractor(images=image, return_tensors="pt").pixel_values.to("cuda")
withtorch.no_grad(), torch.autocast("cuda"):
depth_map=depth_estimator(image).predicted_depthdepth_map=torch.nn.functional.interpolate(
depth_map.unsqueeze(1),
size=(1024, 1024),
mode="bicubic",
align_corners=False,
)
depth_min=torch.amin(depth_map, dim=[1, 2, 3], keepdim=True)
depth_max=torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)
depth_map= (depth_map-depth_min) / (depth_max-depth_min)
image=torch.cat([depth_map] *3, dim=1)
image=image.permute(0, 2, 3, 1).cpu().numpy()[0]
image=Image.fromarray((image*255.0).clip(0, 255).astype(np.uint8))
returnimageprompt="A robot, 4k photo"image=load_image(
"https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main""/kandinsky/cat.png"
).resize((1024, 1024))
controlnet_conditioning_scale=0.5depth_image=get_depth_map(image)
images=pipe(
prompt,
image=image,
control_image=depth_image,
strength=0.99,
num_inference_steps=50,
controlnet_conditioning_scale=controlnet_conditioning_scale,
).imagesimages[0].save(f"robot_cat.png")

It works with enable_pag=False but gives error when enable_pag=True.

Error: AttributeError: 'Image' object has no attribute 'shape'. Did you mean: 'save'?

[satani99]

Any help would be nice. Thanks

[yiyixuxu]

can you share the full stack trace?

[satani99]

File "/home/nikhil/Desktop/pag.py", line 72, in <module> images = pipe( File "/home/nikhil/miniconda3/envs/pag/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/home/nikhil/Desktop/diffusers/src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd_xl_img2img.py", line 1422, in __call__ height, width = control_image.shape[-2:] AttributeError: 'Image' object has no attribute 'shape'. Did you mean: 'save'?
pag.py is the above script.

[satani99]

hi @yiyixuxu can you review this?

[yiyixuxu]

sorry this PR got lost too
could you resolve the conflicts?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

@asomoza
I tested it, and it works now
do you want to check if it works as expected for you? (no worries if you don't have time)

[asomoza]

@yiyixuxu Tested it and seems ok, it's harder to see the difference here because the base image helps a lot even without PAG, but it still works similar to the other ones.

w/o pag	with pag

[yiyixuxu]

@satani99 thank you!