[LoRA] Better conversion of LoRA weights from `.bin` to `.safetensors`

[sayakpaul]

Context: #2829 (comment).

Currently, we have a script that allows us to convert LoRA weights from PyTorch to Safetensors but it outputs the whole pipeline instead of just the LoRA weights.

Ideally, we should just output the LoRA weights in safetensors.

Cc: @patrickvonplaten @Narsil

[pure-rgb]

What are the benefit over pytorch weight to safetensors? Why this attempt (motivation)?

[patrickvonplaten]

Agree @sayakpaul! Thanks for writing this down as an issue. @pure-rgb I don't understand the question here, could you maybe clarify?

[pure-rgb]

@patrickvonplaten Sorry for the confusion, my bad. I needed to see this. Nevermind.

[sayakpaul]

I think we can make use of the existing APIs to support this better.

There is probably a good reason why this script outputs a full pipeline rather than a single LoRA parameters' file:

• It merges the LoRA parameters with the original pre-trained UNet parameters.

I think it can easily be repurposed to just output the UNet with the LoRA parameters merged inside. I suggest we keep the script as is.

But if someone wants to manually convert PyTorch LoRA parameters (such as this one, obtained from this script) to safetensors with the following:

from diffusers import StableDiffusionPipeline 
from huggingface_hub import hf_hub_download
from safetensors.torch import save_file
import torch 

# Load the base pipeline.
base_model_id = "CompVis/stable-diffusion-v1-4"
pipeline = StableDiffusionPipeline.from_pretrained(base_model_id)

# Get the LoRA parameters in PyTorch. 
torch_path = hf_hub_download(
    "sayakpaul/sd-model-finetuned-lora-t4", "pytorch_lora_weights.bin"
)
pt_state_dict = torch.load(torch_path, map_location=torch.device("cpu"))

# Convert to safetensors and serialize.
state_dict_safetensors = "pytorch_lora_weights.safetensors"
save_file(state_dict, state_dict_safetensors, metadata={"format": "pt"})

# Load the safetensors LoRA params into `pipeline.unet`. 
pipeline.unet.load_attn_procs(".")

pipeline.unet.load_attn_procs(".") will load the pytorch_lora_weights.safetensors param iff the latest stable version of safetensors is installed and diffusers installed from source.

Doesn't it solve what the community's been requesting for?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patriciogonzalezvivo]

Hi! I tried this but the resulting .safetensors tile don't seems to work on Automatic1111. Any idea why? The meta data is nothing like the .safetensors generated by https://github.com/kohya-ss/ training script. Any pointer on what to do?