Loading the model from a single *.safetensors file in the root of a repo

[zetyquickly]

Hello, I'm scrutinizing the docs and PRs and cannot find an answer.

Is there a way to do from_pretrained on the folder which contains a single *.safetensors file with a json config?

Or it has to be a set of folders containing each part in safetensors format?

[GBalogo]

I'm only just now getting into the project, so I'm sorry if this isn't very helpful, but have a look at convert_original_stable_diffusion_to_diffusers.py. It's a script for loading *.ckpt and *.safetensors models and creating a diffuser set of folders.
In particular, you're interested in how the load_pipeline_from_original_stable_diffusion_ckpt method functions.

[haofanwang]

For now, you cannot directly load safetensors file via from_pretrained. But here is already a solution for your need, check it our at Lora-for-Diffusers, star it if it is helpful to you. @zetyquickly

[zetyquickly]

@haofanwang read through your project, but frankly didn't get how it can help. Do you mean to use convert_original_stable_diffusion_to_diffusers.py?

[sayakpaul]

Ccing @pcuenca

[haofanwang]

@zetyquickly Please read the Full model section where we show you how to convert safetensors to diffusers format, then you can easily load it. (LoRA is usually saved in safetensors)

[zetyquickly]

@haofanwang ok, that's what I referred to convert_original_stable_diffusion_to_diffusers script.

[zetyquickly]

So, as far as I understand, there's no way to load from_pretrained using *.safetensors file, for now

[haofanwang]

Yes, that is why we have to convert, which maybe not very elegant.

[dk-crazydiv]

Hi, I'm facing the same issue, and while productionizing, the conversion scripts consume a lot of time and memory (disk) while attempting to build a inference pipeline. eg: Using some features/repos (eg: Controlnet implementations, until it comes on diffusers lib) which use single safetensors while a majority of our workflow is on top of Diffusers (including dreambooth/lora finetuning).

Maybe we can open this issue again, so that load from single safetensor support could be explored in diffusers library, leading to more interoperability with other implementations. My understanding of this ckpt/safetensor issue is limited, but isn't the only drawback of single safetensor(not ckpt) is that it loses the 'plug-n-play' of modules like unet/safety-checker, while most practical uses won't require this plugnplay. And this would be a format which would be supported by Automatic web UI as well.

[julien-c]

just making sure @Narsil had seen this one

[Narsil]

By default we should load in safetensors if:

• All safetensors file exist on the remote
• safetensors is installed in your environment.

And this should work for LoRa weights since #2448

For single file, or forcing using safetensors. I can only ask core maintainers.
@pcuenca @patrickvonplaten

My take is that we could maybe have a different than from_pretrained method even a split function, that could load directly from StableDiffusion (I'm guessing this is the main contender).
Is that what you are suggesting ?

[zetyquickly]

@Narsil yes

[dk-crazydiv]

Yes. Single safetensors support seems to be what can bridge the huge current gap between HF diffusers lib and automatic1111 compatibility.
I think two wrappers could be provided:

1. from_single_safetensor for loading the model.
2. Also when we save models to disk(any finetune/dreambooth), option to directly save to_single_safetensor.
   Unfortunately my knowledge of these formats is so limited that I don't even know what these formats are called. hf-diffusers with/without safetensor vs ckpt(stability), vs ckpt + single safetensor automatic.
   In this example, basically what everyone else also seem to be doing is keep 3 copies of the same model in their repo for interoperability. HF diffusers folder structure(5gb), ckpt(2.13gb), model.safetensors(2.13gb). This is the case with almost all the public models where multiple formats get uploaded (but inconsistently).

Adding support for current hf diffuers inside automatic webui with a plugin could also work but I suspect it's going to be way more difficult. In my view, hf-diffusers is going to be the way forward in production, and automatic1111 is the best UI to test these models and pipelines out. Getting this feature would save developers a lot of time and provide a rock-solid, compatible, interoperable workflow.

If anyone has any relevant discussion threads on single safetensor vs current way of saving in diffusers, please share it for my understanding. Also, if someone uses any shortcuts or a better optimized workflow, please do share.

[Narsil]

If anyone has any relevant discussion threads on single safetensor vs current way of saving in diffusers, please share it for my understanding. Also, if someone uses any shortcuts or a better optimized workflow, please do share.

I cannot really have a comprehensive answer for you, but basically, tensor files contains dicts of { "tensor_name": actual_tensor"}.
Both multiple files and single file should actually contain the same data. However, the names could be different.

diffusers relies on transformers for the CLIP model (translating text into latent space) hence the several files.
stable-diffusion-webui has its own file which I know very little about, my guess is that it look quite similar, except potentially the diffusers unet, which could have different names.

Making compatibility across simply means finding the mapping between the names of the various tensors, and putting them where relevant.

That should really be it.

You could take most of the code from here actually :

diffusers/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py

Line 976 in f4bbcb2

def download_from_original_stable_diffusion_ckpt(

The problem most likely will reside in making code that is both not too bloated, and works seamlessly across the board. I think converters already exist out there