[Flux Redux] add prompt & multiple image input

[linoytsaban]

Add the following to the flux redux prior pipeline:

• prompt inputs
• image interpolation

inference example:

image = Image.open("Self-portrait-oil-canvas-Thorn-Necklace-Hummingbird-Frida.jpg").convert("RGB")
image2 = Image.open("Mona_Lisa.jpg").convert("RGB")
pipe_prior_output = pipe_prior_redux([image,image2], prompt=["self portrait by frida khalo", "mona lisa"], prompt_embeds_scale=[.9, .75],
pooled_prompt_embeds_scale=[.6,1.25]
)
pipe(
guidance_scale=2.5,
height=1024,
width=1024,
num_inference_steps=50,
max_sequence_length=512,
generator=torch.Generator("cpu").manual_seed(0),
**pipe_prior_output,
).images[0]

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

let us know when it is ready for a review!
cc @asomoza

[linoytsaban]

@yiyixuxu@asomoza I think it's ready for review :)

[hlky]

Works well, thanks! 🤗

Code

importtorchfromdiffusersimportFluxPriorReduxPipeline, FluxPipelinefromdiffusers.utilsimportload_imagedevice="cuda"dtype=torch.bfloat16repo_redux="black-forest-labs/FLUX.1-Redux-dev"repo_base="black-forest-labs/FLUX.1-dev"pipe_prior_redux=FluxPriorReduxPipeline.from_pretrained(
repo_redux, torch_dtype=dtype
).to(device)
pipe=FluxPipeline.from_pretrained(
repo_base, text_encoder=None, text_encoder_2=None, torch_dtype=torch.bfloat16
).to(device)
image=load_image(
"https://www.arthistoryproject.com/site/assets/files/19982/frida-kahlo-self-portrait-with-thorn-necklace-and-hummingbird-1940-trivium-art-history.jpg"
)
image2=load_image(
"https://upload.wikimedia.org/wikipedia/commons/thumb/6/6a/Mona_Lisa.jpg/1354px-Mona_Lisa.jpg"
)
pipe_prior_output=pipe_prior_redux(
[image, image2],
prompt=["self portrait by frida khalo", "mona lisa"],
prompt_embeds_scale=[0.9, 0.75],
pooled_prompt_embeds_scale=[0.6, 1.25],
)
images=pipe(
guidance_scale=2.5,
height=1024,
width=1024,
num_inference_steps=50,
max_sequence_length=512,
generator=torch.Generator("cpu").manual_seed(0),
**pipe_prior_output,
).images[0]
images.save("flux-redux.png")

Output

[yiyixuxu]

thanks! this is really cool - would love to have it on doc somewhere too

[yiyixuxu]

let's throw out a warning here:
if prompt inputs is passed but do not have text_encoder/tokenizer, in this case the text inputs will be ignored

it is a bit of different from our regular pipelines, normally, if you pass a prompt and do not have a text_encoder, you will get an error says like from encode_prompt; here we will just use zero prompt embeds instead, so let's be make an explicit warning about that

[linoytsaban]

agree, added one now

[yiyixuxu]

let's make it clear that it is an experimental feature, and if you pass prompt, you will need to load text_encoders explicitly

[linoytsaban]

done

[sayakpaul]

THIS NEEDS TO BE IN THE DOCS.

@stevhliu any ideas about the location?

[stevhliu]

Super cool! 🤩

We can add it to the "Specific Pipeline Examples" section and then build out the Flux doc there as discussed here.

[yiyixuxu]

thanks! I think we can add doc in a separate PR
but need make style

[linoytsaban]

@yiyixuxu I'm not sure what's the issue, when I run make fixup it doesnt make any changes / flag any issues 🤔

[hlky]

@linoytsaban It's the doc-builder check, try doc-builder style src/diffusers docs/source --max_len 119 on its own or make style, it's not covered with make fixup

[linoytsaban]

thanks @hlky!

[sayakpaul]

@linoytsaban let's add some docs (#10056 (comment)) and communicate?

[Thekey756]

效果不太好，并没有达到官方示例的效果

[hlky]

@Thekey756 do you have an example comparison to the original?

[linoytsaban]

@Thekey756 I think to achieve the original effect you're referring to we need to also take in consideration - #10025

[linoytsaban]

Example using prompts and attention masking for improved prompt adherence:

pipe_prior_output = pipe_prior_redux([image], prompt=["anime illustration"], prompt_embeds_scale=[1.],
pooled_prompt_embeds_scale=[1.]
)

cond_size = 729
hidden_size = 4096
max_sequence_length = 512
full_attention_size = max_sequence_length + hidden_size + cond_size
attention_mask = torch.zeros(
(full_attention_size, full_attention_size), device="cuda", dtype=torch.bfloat16
)
reference_scale: float = 0.04 # example
bias = torch.log(
torch.tensor(reference_scale, dtype=torch.bfloat16, device="cuda").clamp(min=1e-5, max=1)
)
attention_mask[:, max_sequence_length : max_sequence_length + cond_size] = bias
joint_attention_kwargs=dict(attention_mask=attention_mask)
pipe(
guidance_scale=2.5,
height=1024,
width=1024,
num_inference_steps=50,
max_sequence_length=512,
generator=torch.Generator("cpu").manual_seed(0),
joint_attention_kwargs=joint_attention_kwargs,
**pipe_prior_output,
).images[0]

[lhjlhj11]

Add the following to the flux redux prior pipeline:

* prompt inputs
* image interpolation

inference example:

image = Image.open("Self-portrait-oil-canvas-Thorn-Necklace-Hummingbird-Frida.jpg").convert("RGB")
image2 = Image.open("Mona_Lisa.jpg").convert("RGB")
pipe_prior_output = pipe_prior_redux([image,image2], prompt=["self portrait by frida khalo", "mona lisa"], prompt_embeds_scale=[.9, .75],
pooled_prompt_embeds_scale=[.6,1.25]
)
pipe(
guidance_scale=2.5,
height=1024,
width=1024,
num_inference_steps=50,
max_sequence_length=512,
generator=torch.Generator("cpu").manual_seed(0),
**pipe_prior_output,
).images[0]

Can you give an example for your Fast FLUX.1 Redux in the space of huggingface? Thanks! And what is the masking scale?

[MikeHanKK]

how should I set prompt_embeds_scale, pooled_prompt_embeds_scale, and guidance_scale in general? @linoytsaban