Full support for Flux attention masking

[squewel]

Is your feature request related to a problem? Please describe.
SimpleTuner/Kohya allow T5 attention masked training, however this is not currently supported natively in diffusers

Describe the solution you'd like.
Already implemented and used in Simpletuner and Kohya: https://github.com/bghira/SimpleTuner/blob/main/helpers/models/flux/transformer.py

Describe alternatives you've considered.
Recent implementation doesn't really solve the use case of using existing fine tunes with attention masking with diffusers
#10122

Additional context.
@yiyixuxu @bghira @AmericanPresidentJimmyCarter

@bghira's suggestion: "i'd suggested they add encoder_attention_mask and image_attention_mask and if image_attention_mask is None that they could then 1-fill those positions and just cat them together"

[bghira]

i guess this could live concurrently to the joint_kwargs implementation and just check_inputs error on whether the encoder or image attn mask are passed in alongside it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.