LoRA for clip and vae?

[Zyriix]

LoRA now only add parameters to the attention in unet. In my experiment, the performance is not good. I hope we can support lora layer for unet and clip. best wishes

[patrickvonplaten]

Working on it: #2918

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[chenxinhua]

Working on it: #2918

Text encoder LoRA layers.

from diffusers.utils import TEXT_ENCODER_TARGET_MODULES

text_lora_attn_procs = {}
for name, module in text_encoder.named_modules():
if any([x in name for x in TEXT_ENCODER_TARGET_MODULES]):
text_lora_attn_procs[name] = LoRAAttnProcessor(
hidden_size=module.out_features, cross_attention_dim=None
)

text_encoder_lora_layers = AttnProcsLayers(text_lora_attn_procs)

This piece of code simply identifies the attention and MLP layers of the text encoder and generates LoRAs, but it does not insert it into the forward propagation of the text encoder. Therefore, it does not play any role.

[hyoseok1223]

Any way to apply lora to vae decoder? I implent it but parameter is not updated or output is same

[sgw-ite]

Any way to apply lora to vae decoder? I implent it but parameter is not updated or output is same

Do you have any update? Working on applying lora to vae too