StableDiffusionLatentUpscalePipeline - positive/negative prompt embeds support

[rootonchair]

What does this PR do?

Fixes#8895

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu

[rootonchair]

Test script

fromdiffusersimportStableDiffusionLatentUpscalePipeline, StableDiffusionPipelineimporttorchpipeline=StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipeline.to("cuda")
upscaler=StableDiffusionLatentUpscalePipeline.from_pretrained("stabilityai/sd-x2-latent-upscaler", torch_dtype=torch.float16)
upscaler.to("cuda")
prompt="a photo of an astronaut high resolution, unreal engine, ultra realistic"generator=torch.manual_seed(33)
# we stay in latent space! Let's make sure that Stable Diffusion returns the image# in latent spacelow_res_latents=pipeline(prompt, generator=generator, output_type="latent").imagesupscaled_image=upscaler(
prompt=prompt,
image=low_res_latents,
num_inference_steps=20,
guidance_scale=0,
generator=generator,
).images[0]
# Let's save the upscaled image under "upscaled_astronaut.png"upscaled_image.save("astronaut_1024.png")
prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds=upscaler.encode_prompt(
prompt=prompt,
device=upscaler._execution_device,
do_classifier_free_guidance=False,
)
upscaled_image=upscaler(
image=low_res_latents,
num_inference_steps=20,
guidance_scale=0,
generator=generator,
prompt_embeds=prompt_embeds,
pooled_prompt_embeds=pooled_prompt_embeds,
).images[0]
upscaled_image.save("embeds_astronaut_1024.png")
# as a comparison: Let's also save the low-res imagewithtorch.no_grad():
image=pipeline.decode_latents(low_res_latents)
image=pipeline.numpy_to_pil(image)[0]
image.save("astronaut_512.png")

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

oh thanks! I think it looks really nice!

[sayakpaul]

LGTM! Could you also share some results stemming from this change?

Let's also add a fast test for this?

[rootonchair]

Could you also share some results stemming from this change?

Sure
Here is the original result

And result using embeds

which does not have much different

Let's also add a fast test for this?

Should I open another PR for this @sayakpaul ?

[sayakpaul]

Should I open another PR for this @sayakpaul ?

We can do this in this PR

[rootonchair]

Hi @sayakpaul, could you help me review this PR?

[yiyixuxu]

oh actually I think the tests for latent upscaler is here https://github.com/huggingface/diffusers/blob/main/tests/pipelines/stable_diffusion_2/test_stable_diffusion_latent_upscale.py

maybe we add new tests there? very sorry for all the additional work to create a new test from scratch

[rootonchair]

@yiyixuxu I have just update the existing test. Could you help me review? Btw, adding new test is kinda interesting too. So no worries

[yiyixuxu]

why do we need to update the expected_slice here? the results of existing tests should not change, no?

[rootonchair]

@yiyixuxu I think it's due to this line. The old code does not take in generator

[yiyixuxu]

I think we got the order of the negative_prompt_embeds and prompt_embeds reversed, that's why the previous test wasn't passing. Let's make the change, and change the test back and make sure it passes :)

[rootonchair]

@yiyixuxu thanks for the catch! Sorry I did not check it thoroughly

[yiyixuxu]

thanks!