Error converting LoRA to safetensors/ckpt

[jndietz]

Generally speaking, I am not clear on what to do with the output of these LoRA python scripts. I don't think the output can be natively used by the webuis. Other LoRAs I've seen online are usually safetensors. Here is what I did...

I used the provided python script in examples to generate a LoRA:

#!/bin/sh

accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path=$1  \
  --instance_data_dir=$2 \
  --output_dir=$3 \
  --instance_prompt="a photo of laskajavids" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --checkpointing_steps=200 \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=1000 \
  --validation_prompt="A photo of laskajavids by the pool" \
  --validation_epochs=50 \
  --seed="0" \
  --mixed_precision="fp16" \
  --use_8bit_adam

I successfully (or not?) created a LoRA, and it output the following to /output:

checkpoint-1000
checkpoint-200
checkpoint-400
checkpoint-600
checkpoint-800
pytorch_lora_weights.bin

Then, I tried to run scripts\convert_diffusers_to_original_stable_diffusion.py, like so:

 python /diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py --model_path /output --checkpoint_path /test.ckpt --use_safetensors

I received the following error:

Traceback (most recent call last):
  File "/diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py", line 290, in <module>
    unet_state_dict = torch.load(unet_path, map_location="cpu")
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 771, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 270, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 251, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/output/unet/diffusion_pytorch_model.bin'

Did I miss something when creating the LoRA?

[jndietz]

I just tried running the newer script examples\text_to_image\train_text_to_image_lora.py, with the following

export MODEL_NAME=$1
export TRAIN_DIR=$2

accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$TRAIN_DIR --caption_column="additional_feature" \
  --resolution=512 --random_flip \
  --train_batch_size=1 \
  --validation_epochs=120 \
  --max_train_steps=1000 \
  --checkpointing_steps=100 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --output_dir=$3 \
  --validation_prompt="photo of laskajavids" \
  --enable_xformers_memory_efficient_attention

And the output appeared to be the same as examples/train_dreambooth_lora.py. So, I got the same error:

python /diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py --model_path /output --checkpoint_path /test.ckpt --use_safetensors
Traceback (most recent call last):
  File "/diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py", line 290, in <module>
    unet_state_dict = torch.load(unet_path, map_location="cpu")
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 771, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 270, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 251, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/output/unet/diffusion_pytorch_model.bin'

[jndietz]

Looking at the script itself, I'm seeing why this isn't working. My output directory isn't a model. What do I do with the output in there currently?

checkpoint-1000
checkpoint-200
checkpoint-400
checkpoint-600
checkpoint-800
pytorch_lora_weights.bin

[patrickvonplaten]

@williamberman could you take a look here?

[NicholasKao1029]

Following

[patrickvonplaten]

By other WebUIs was mostly Automatic1111 meant?

We should really think about whether we can somehow integrate diffusers into AUTO1111

[jndietz]

@patrickvonplaten Yes, I am speaking to automatic1111's webui specifically. Though, I want to be able to do this via a python script without having to use a webui.

I'll have to go back and make sure I followed the 🤗 docs. I do feel like I followed them closely, which is why I am confused as to how to use the outputted LoRAs and convert them into a safetensors or ckpt.

[patrickvonplaten]

You can use LoRA trained embeddings easily with diffusers, see: https://huggingface.co/docs/diffusers/v0.12.0/en/training/lora#lora-support-in-diffusers but I think it's not so easy to convert them to A1111.

[haofanwang]

Hi, @patrickvonplaten, does diffusers support convert or directly load from Lora+safetensor? I have tried with the converting scripts, but failed.

The weight is from civitai. Any help would be appreciated.

[patrickvonplaten]

I'll try to allocate time for this. Otherwise maybe @williamberman or @sayakpaul could check it out as well. I'm not 100% sure whether we can convert civitai lora weights

[sayakpaul]

W.r.t #2363 I think there are a couple of different conversion pathways we're talking about for completness:

• Diffusers LoRA weights to safetensors
• civitai weights to diffusers
• civitai LoRA weights to diffusers

@patrickvonplaten am I missing out on something?

[haofanwang]

I think the second one has already been ready with ./scripts/convert_original_stable_diffusion_to_diffusers.py, it can convert civitai weights (in safetensors but without lora) into diffusers format.

The third one should be civitai LoRA weights (in safetensors format) to diffusers. I'm actually working on it by diving into stable-diffusion-webui which supports loading from lora+safetensors format, I will provide my script for your reference once I finish.

@sayakpaul

[sayakpaul]

That is amazing! It would be amazing to contribute a PR for that as well!