Change KolorsPipeline LoRA Loader to StableDiffusion

[BasileLewan]

The KolorsPipeline inherits from StableDiffusionXLLoraLoaderMixin which expects the pipeline to have two text encoders, whereas Kolors only uses one. This causes an error when trying to load LoRA weights on the pipeline.

Falling back to StableDiffusionLoraLoaderMixin fixes the issue.

(1st contribution here, sorry if I did something wrong)

[asomoza]

oh nice, I forgot about that since at the time there weren't any kolors loras to test it. Can you please share the lora and an example of it?

[BasileLewan]

I'm not sure to get what you mean, this is the case for any LoRA.
If you want a minimal reproduction, here it is:

importtorchfromdiffusersimportKolorsPipelinefromdiffusers.utilsimportconvert_state_dict_to_diffusersfrompeftimportLoraConfigfrompeft.utilsimportget_peft_model_state_dictpipe=KolorsPipeline.from_pretrained("Kwai-Kolors/Kolors-diffusers", variant='fp16')
model=pipe.unetcfg=LoraConfig(target_modules=["to_k", "to_q", "to_v"])
model.add_adapter(cfg)
lora_state_dict=convert_state_dict_to_diffusers(get_peft_model_state_dict(model))
pipe.save_lora_weights("lora_path", lora_state_dict)
pipe.load_lora_weights("lora_path/pytorch_lora_weights.safetensors")
# AttributeError: 'KolorsPipeline' object has no attribute 'text_encoder_2'

[asomoza]

that example is nice but what I meant was a real lora trained that people use, the reason why this wasn't noticed by anyone until now except for you is that there wasn't any more people that used a Kolors Loras, even as unet loras, there's just a few and actually I never seen anyone using them.

Still doesn't matter, I was just curious, we still need to merge this in case that ever happens.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[asomoza]

I don't think there's a need to create a specific lora loader mixin for kolors since there are just a handful of them but this will enable people to use them if they want.

[asomoza]

thanks a lot! Failing tests are not related to this PR.

[BasileLewan]

No problem, thanks for the quick review!