StableDiffusionXLPipeline.from_single_file Error

[cendelian]

Describe the bug

version: 0.27.0, 0.27.1
when use StableDiffusionXLPipeline.from_single_file,error occur:

 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 383, in encode_prompt
 prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True)
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
 return forward_call(*args, **kwargs)
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward
 return self.text_model(
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
 return forward_call(*args, **kwargs)
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 730, in forward
 hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
 return forward_call(*args, **kwargs)
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 229, in forward
 position_embeddings = self.position_embedding(position_ids)
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
 return forward_call(*args, **kwargs)
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
 return F.embedding(
 File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.HalfTensor instead (while checking arguments for embedding)

but version 0.26.3 is OK

Reproduction

pipe = StableDiffusionXLPipeline.from_single_file('xxxxxxx.safetensors',
 original_config_file='aaaa/sd_xl_base.yaml', 
 torch_dtype=torch.float16, 
 local_files_only=True,
 use_safetensors=True,
 add_watermarker=False,
# safety_checker=None,
# requires_safety_checker=False
 )
print("22222222222222")

pipe.to('cuda')
prompt = "black denim,long jeans,1girl,datou,"
negative_prompt = '(worst quality:2.1),(low quality:2.1),(normal quality:2.1)'


# sdxl
image1 = pipe(
 prompt=prompt,
 negative_prompt=negative_prompt,
 num_inference_steps=50,
 eta=1.0,
).images[0]

image1.save("test.png")

Logs

No response

System Info

diffuser 0.27.0 0.27.1

Who can help?

No response

[yiyixuxu]

sorry I can't reproduce this, this runs for me

the config file is https://raw.githubusercontent.com/Stability-AI/generative-models/main/configs/inference/sd_xl_base.yaml

from diffusers import StableDiffusionXLPipeline
import torch

repo_id = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors"
pipe = StableDiffusionXLPipeline.from_single_file(repo_id,
 original_config_file='sd_xl_base.yaml', 
 torch_dtype=torch.float16, 
 local_files_only=True,
 use_safetensors=True,
 add_watermarker=False,
 )

pipe.enable_model_cpu_offload()
prompt = "black denim,long jeans,1girl,datou,"
negative_prompt = '(worst quality:2.1),(low quality:2.1),(normal quality:2.1)'


# sdxl
image1 = pipe(
 prompt=prompt,
 negative_prompt=negative_prompt,
 num_inference_steps=50,
 eta=1.0,
).images[0]

image1.save("yiyi_test_out.png")

[cendelian]

sorry I can't reproduce this, this runs for me

the config file is https://raw.githubusercontent.com/Stability-AI/generative-models/main/configs/inference/sd_xl_base.yaml

from diffusers import StableDiffusionXLPipeline
import torch

repo_id = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors"
pipe = StableDiffusionXLPipeline.from_single_file(repo_id,
 original_config_file='sd_xl_base.yaml', 
 torch_dtype=torch.float16, 
 local_files_only=True,
 use_safetensors=True,
 add_watermarker=False,
 )

pipe.enable_model_cpu_offload()
prompt = "black denim,long jeans,1girl,datou,"
negative_prompt = '(worst quality:2.1),(low quality:2.1),(normal quality:2.1)'


# sdxl
image1 = pipe(
 prompt=prompt,
 negative_prompt=negative_prompt,
 num_inference_steps=50,
 eta=1.0,
).images[0]

image1.save("yiyi_test_out.png")

what's your version? 0.26.3 is OK, but 0.27.0/0.27.1/0.27.2 occur error
sd_xl_base_1.0.safetensors
sd_xl_base.yaml

[sayakpaul]

Unable to reproduce: https://colab.research.google.com/gist/sayakpaul/62e5e57f34161122b8a1439906b84c51/scratchpad.ipynb.

[cbstyle]

Have you solved the problem?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

closing now since we are not able to reproduce on our end
if this is still an issue, please let us know and we will reopen it!

[eija06]

I am able to reproduce it with the following minimal code:

from diffusers import StableDiffusionXLPipeline
import torch

pipeline = StableDiffusionXLPipeline.from_single_file(
"/workspace/stable-diffusion-webui/models/Stable-diffusion/realismEngineSDXL_v30VAE.safetensors",
torch_dtype=torch.float16
).to("cuda")

image = pipeline(prompt="photo").images[0]

The venv is
absl-py==1.4.0
accelerate==0.21.0
addict==2.4.0
aenum==3.1.15
aiofiles==23.2.1
aiohttp==3.8.5
aiosignal==1.3.1
altair==5.1.1
antlr4-python3-runtime==4.9.3
anyio==3.7.1
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asttokens==2.2.1
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.1.0
av==10.0.0
Babel==2.12.1
backcall==0.2.0
basicsr==1.4.2
beautifulsoup4==4.12.2
bleach==6.0.0
blendmodes==2022
boltons==23.0.0
cachetools==5.3.1
certifi==2023.7.22
cffi==1.15.1
chardet==5.2.0
charset-normalizer==3.2.0
clean-fid==0.1.35
click==8.1.7
clip==1.0
cmake==3.25.0
comm==0.1.4
contourpy==1.1.0
cssselect2==0.7.0
cycler==0.11.0
debugpy==1.6.7.post1
decorator==5.1.1
defusedxml==0.7.1
deprecation==2.1.0
diffusers==0.27.2
dynamicprompts==0.31.0
einops==0.4.1
exceptiongroup==1.1.3
executing==1.2.0
facexlib==0.3.0
fake-useragent==1.5.1
fastapi==0.94.0
fastjsonschema==2.18.0
ffmpy==0.3.1
filelock==3.9.0
filterpy==1.4.5
flatbuffers==23.5.26
fonttools==4.42.1
fqdn==1.5.1
frozenlist==1.4.0
fsspec==2023.6.0
ftfy==6.1.1
future==0.18.3
fvcore==0.1.5.post20221221
gdown==4.7.1
gfpgan==1.3.8
gitdb==4.0.10
GitPython==3.1.32
google-auth==2.22.0
google-auth-oauthlib==1.0.0
gradio==3.41.2
gradio_client==0.5.0
grpcio==1.57.0
h11==0.12.0
httpcore==0.15.0
httpx==0.24.1
huggingface-hub==0.23.0
idna==3.4
imageio==2.31.2
imageio-ffmpeg==0.4.8
importlib-metadata==6.8.0
importlib-resources==6.0.1
inflection==0.5.1
iopath==0.1.10
ipykernel==6.25.1
ipython==8.14.0
ipywidgets==8.1.0
isoduration==20.11.0
jedi==0.19.0
Jinja2==3.1.2
json5==0.9.14
jsonmerge==1.8.0
jsonpointer==2.4
jsonschema==4.19.0
jsonschema-specifications==2023.7.1
jupyter-events==0.7.0
jupyter-lsp==2.2.0
jupyter_client==8.3.1
jupyter_core==5.3.1
jupyter_server==2.7.3
jupyter_server_terminals==0.4.4
jupyterlab==4.0.5
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.8
jupyterlab_server==2.24.0
kiwisolver==1.4.5
kornia==0.6.7
lark==1.1.2
lazy_loader==0.3
lightning-utilities==0.9.0
lit==15.0.7
llvmlite==0.40.1
lmdb==1.4.1
lpips==0.1.4
lxml==4.9.3
lycoris-lora==2.2.0.post3
Markdown==3.4.4
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.7.2
matplotlib-inline==0.1.6
mdurl==0.1.2
mediapipe==0.10.3
mistune==3.0.1
mpmath==1.2.1
multidict==6.0.4
nbclient==0.8.0
nbconvert==7.8.0
nbformat==5.9.2
nest-asyncio==1.5.7
networkx==3.0
notebook_shim==0.2.3
numba==0.57.1
numexpr==2.8.5
numpy==1.23.5
oauthlib==3.2.2
omegaconf==2.2.3
open-clip-torch==2.20.0
opencv-contrib-python==4.8.0.76
opencv-python==4.8.0.76
orjson==3.9.5
overrides==7.4.0
packaging==23.1
pandas==2.1.0
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
piexif==1.1.3
Pillow==9.5.0
PIMS==0.6.1
platformdirs==3.10.0
portalocker==2.7.0
prometheus-client==0.17.1
prompt-toolkit==3.0.39
protobuf==3.20.0
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pydantic==1.10.12
pydub==0.25.1
Pygments==2.16.1
pyparsing==3.0.9
PySocks==1.7.1
python-dateutil==2.8.2
python-json-logger==2.0.7
python-multipart==0.0.6
pytorch-lightning==1.9.4
pytz==2023.3
PyWavelets==1.4.1
PyYAML==6.0.1
pyzmq==25.1.1
realesrgan==0.3.0
referencing==0.30.2
regex==2023.8.8
reportlab==4.0.4
requests==2.31.0
requests-oauthlib==1.3.1
resize-right==0.0.2
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.5.2
rpds-py==0.10.0
rsa==4.9
safetensors==0.3.1
scikit-image==0.21.0
scipy==1.11.2
seaborn==0.13.2
semantic-version==2.10.0
Send2Trash==1.8.2
sentencepiece==0.1.99
six==1.16.0
slicerator==1.1.0
smmap==5.0.0
sniffio==1.3.0
sounddevice==0.4.6
soupsieve==2.4.1
stack-data==0.6.2
starlette==0.26.1
svglib==1.5.1
sympy==1.11.1
tabulate==0.9.0
tb-nightly==2.15.0a20230831
tensorboard-data-server==0.7.1
termcolor==2.3.0
terminado==0.17.1
thop==0.1.1.post2209072238
tifffile==2023.8.30
timm==0.9.2
tinycss2==1.2.1
tokenizers==0.13.3
tomesd==0.1.3
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
torch==2.0.1+cu118
torchaudio==2.0.2+cu118
torchdiffeq==0.2.3
torchmetrics==1.1.1
torchsde==0.2.5
torchvision==0.15.2+cu118
tornado==6.3.3
tqdm==4.66.1
traitlets==5.9.0
trampoline==0.1.2
transformers==4.30.2
triton==2.0.0
typing_extensions==4.7.1
tzdata==2023.3
ultralytics==8.2.8
uri-template==1.3.0
urllib3==1.26.16
uvicorn==0.23.2
wcwidth==0.2.6
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.2
websockets==11.0.3
Werkzeug==2.3.7
widgetsnbextension==4.0.8
xformers==0.0.21
yacs==0.1.8
yapf==0.40.1
yarl==1.9.2
zipp==3.16.2
ZipUnicode==1.1.1

I am NOT getting this error with a fresh venv installed with the most recent versions available to pip.
So you could close this bug report as fixed - but you might get this report again, because the venv posted above is in active use on many machines - it is the venv that stable diffusion A1111 webui creates.