error of num_images_per_prompt in the flux pipeline

[rardz]

in the _get_clip_prompt_embeds in flux pipeline, it is:
`
prompt_embeds = self.text_encoder(text_input_ids.to(device), output_hidden_states=False)

 # Use pooled output of CLIPTextModel
 prompt_embeds = prompt_embeds.pooler_output
 prompt_embeds = prompt_embeds.to(dtype=self.text_encoder.dtype, device=device)

 # duplicate text embeddings for each generation per prompt, using mps friendly method
 prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
 prompt_embeds = prompt_embeds.view(batch_size * num_images_per_prompt, -1)

`
prompt_embeds should repeated interleavely on batch, but as a 2d tensor, prompt_embeds here will finally be repeated but not interleavely, which is not corretlly.

[DN6]

@rardz prompt embeds is a 3D tensor? The operation here is equivalent to performing repeat interleave along the batch dimension. If you're encountering issue could you share the code example to demonstrate?

[rardz]

in the line prompt_embeds = self.text_encoder(text_input_ids.to(device), output_hidden_states=False), prompt_embeds is a sequence output exactly. Howerver, the clip text encoder is only treated as a pooler embedding extractor, so prompt_embeds = prompt_embeds.pooler_output get a 2d tensor here.

so for a 2d tensor, I think the code snippet below should clarify the problem:

x = torch.arange(2 * 4).view(2, -1)
# tensor([[0, 1, 2, 3],
# [4, 5, 6, 7]])

y1 = x.repeat(1, 3, 1).view(2 * 3, -1)
# tensor([[[0, 1, 2, 3],
# [4, 5, 6, 7],
# [0, 1, 2, 3],
# [4, 5, 6, 7],
# [0, 1, 2, 3],
# [4, 5, 6, 7]]])

y2 = x.repeat_interleave(3, dim=0)
# tensor([[0, 1, 2, 3],
# [0, 1, 2, 3],
# [0, 1, 2, 3],
# [4, 5, 6, 7],
# [4, 5, 6, 7],
# [4, 5, 6, 7]])

it should be y2 instead of y1, isn't it?

[DN6]

@rardz Ah I see what you mean. My mistake. I've opened a PR to fix. Thank you for catching and the explanation.

[rardz]

and the lines of the of the dreambooth training examples should also be fixed.

diffusers/examples/dreambooth/train_dreambooth_flux.py

Line 934 in b5f591f

prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)

and

diffusers/examples/dreambooth/train_dreambooth_lora_flux.py

Line 956 in b5f591f

prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[asomoza]

The PR that fixed this issue was merged so closing this issue for now, if you feel we're still missing something please open a new issue or feel free to ask to reopen this one.