kohya-ss FLUX LoRA keys not compatible with diffusers

[apolinario]

Is your feature request related to a problem? Please describe.
kohya-ss has added FLUX.1 LoRA training, however the keys are not working with diffusers (silent erroring, it doesn't error but the style isn't applied)

Describe the solution you'd like.
Support kohya-ss FLUX.1 trained keys natively

Additional context.
Kohya has a script to convert their format to diffusers: https://github.com/kohya-ss/sd-scripts/blob/sd3/networks/convert_flux_lora.py - the idea is to support this format natively

Example LoRA:
https://huggingface.co/Norod78/brain-slug-flux

Expected results:

Minimal code to reproduce

import torch
from diffusers import DiffusionPipeline

device = "cuda" if torch.cuda.is_available() else "cpu"

pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe = pipe.to(device)

pipe.load_lora_weights("Norod78/brain-slug-flux")
generator = torch.Generator(device="cpu").manual_seed(0)
prompt = "The girl with a brain slug earring"
image = pipe(prompt, generator=generator).images[0]
image

Current actual results with diffusers:

Which is the exact same result as not loading the LoRA at all

import torch
from diffusers import DiffusionPipeline

device = "cuda" if torch.cuda.is_available() else "cpu"

pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe = pipe.to(device)
generator = torch.Generator(device="cpu").manual_seed(0)
prompt = "The girl with a brain slug earring"
image = pipe(prompt, generator=generator).images[0]
image

produces the same image: