[Community] The "lpw_stable_diffusion_xl" pipeline doesn't support new run params

[T145]

Describe the bug

Passing args like clip_skip or cfg_scale to a pipeline instantiated with the "lpw_stable_diffusion_xl" pipeline cause a crash.

Reproduction

from diffusers import *
from diffusers.utils import make_image_grid
import torch
import numpy as np

universal_args = {
	'cache_dir': 'data',
	'torch_dtype': torch.float16,
}
model_args = {
	'variant': 'fp16',
}
model_args.update(universal_args)

vae = AutoencoderKL.from_pretrained(
	'madebyollin/sdxl-vae-fp16-fix', # https://huggingface.co/madebyollin/sdxl-vae-fp16-fix
	**universal_args,
)

model_id = 'stabilityai/stable-diffusion-xl-base-1.0'

scheduler = DPMSolverMultistepScheduler.from_pretrained(
	model_id,
	subfolder='scheduler',
	use_karras_sigmas=True,
	algorithm_type='sde-dpmsolver++',
	**universal_args
)

# https://huggingface.co/docs/diffusers/v0.25.0/en/api/schedulers/overview
pipe = DiffusionPipeline.from_pretrained(
 model_id,
	vae=vae,
	scheduler=scheduler,
	custom_pipeline="lpw_stable_diffusion_xl",
	**model_args
).to('cuda')

prompt = "(photo-realistic, real photo, masterpiece, best quality, 8k, UHD, RAW photo, levelled out, super resolution, sharp focus, ultra detailed skin, realistic skin, beautiful vivid backgrounds, exquisitely detailed, colorful and intricate details, ray tracing, film lights, spot lights, lustrous skin, physically based rendering, best illustration, best shadow, cinematic lighting, delicate Illustration, official art, aesthetic:1.4), (golden-ratio face, perfect proportioned face, perfectly beautiful), glossy and red lips, (brown eyes), senior high school student, (k-pop idol, miss korea, korean beauty), (short torso, long legs, slim waist, huge hips, huge naturally sagging breasts:1.4), (1girl, solo girl:1.3), (Cinematic view of a wonderful burning castle built in a middle of a dark wood, wide view, fire and ashes, cloudy night:1.3), (Medieval Plate Armour, Full body Armor Suit With Chain Mail :1.3), (Crusader knight, cape, hold a shield and very long sword, fighting, wield a sword:1.4), (carrying a big bow on back:1.2), chastity belt"
prompt2 = ""
prompt = f"{prompt},{prompt2}"
neg_prompt = "(worst quality, low quality, normal quality, low resolution, worst resolution, normal resoultion, collage, bad anatomy of fingers:1.4), korean traditional building, tattoo, watermark, (Dutch angle, Dutch tilt, canted angle, vortex angle, oblique angle:1.3), tiara, (one more girl, nipples:1.4)"

generator = torch.Generator(device='cuda').manual_seed(3469586480)

image = pipe(
	# prompt_embeds=conditioning,
	# pooled_prompt_embeds=pooled,
	# negative_prompt_embeds=negative_conditioning,
	# negative_pooled_prompt_embeds=negative_pooled,
	prompt=prompt,
	negative_prompt=neg_prompt,
	generator=generator,
	num_inference_steps=40,
	clip_skip=2,
	cfg_scale=20,
).images[0]

image.save('test.png')

Logs

Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 10.87it/s]
Traceback (most recent call last):
 File "-\test.py", line 46, in <module>
 image = pipe(
 ^^^^^
 File "-\torch\utils\_contextlib.py", line 115, in decorate_context
 return func(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^
TypeError: SDXLLongPromptWeightingPipeline.__call__() got an unexpected keyword argument 'clip_skip'

System Info

name: sd
channels:
 - pytorch
 - nvidia
 - conda-forge
 - defaults
dependencies:
 - accelerate=0.25.0=pyhd8ed1ab_0
 - aiohttp=3.9.1=py311ha68e1ae_0 
 - aiosignal=1.3.1=pyhd8ed1ab_0 
 - attrs=23.1.0=pyh71513ae_1
 - aws-c-auth=0.7.8=h6bf0135_3
 - aws-c-cal=0.6.9=h7f0e5be_2
 - aws-c-common=0.9.10=hcfcfb64_0
 - aws-c-compression=0.2.17=h7f0e5be_7
 - aws-c-event-stream=0.3.2=h51e6447_8
 - aws-c-http=0.7.15=h80119a0_0
 - aws-c-io=0.13.36=ha737126_3
 - aws-c-mqtt=0.10.0=h9e0d744_1
 - aws-c-s3=0.4.6=he94bf34_0
 - aws-c-sdkutils=0.1.13=h7f0e5be_0
 - aws-checksums=0.1.17=h7f0e5be_6
 - aws-crt-cpp=0.25.0=h1554317_4
 - aws-sdk-cpp=1.11.210=h0441c79_5
 - blas=2.120=mkl
 - blas-devel=3.9.0=20_win64_mkl
 - brotli-python=1.1.0=py311h12c1d0e_1
 - bzip2=1.0.8=hcfcfb64_5
 - c-ares=1.24.0=hcfcfb64_0
 - ca-certificates=2023.12.12=haa95532_0
 - certifi=2023.11.17=py311haa95532_0
 - charset-normalizer=3.3.2=pyhd8ed1ab_0
 - colorama=0.4.6=pyhd8ed1ab_0
 - cuda-cccl=12.3.101=0
 - cuda-cudart=12.1.105=0
 - cuda-cudart-dev=12.1.105=0
 - cuda-cupti=12.1.105=0
 - cuda-libraries=12.1.0=0
 - cuda-libraries-dev=12.1.0=0
 - cuda-nvrtc=12.1.105=0
 - cuda-nvrtc-dev=12.1.105=0
 - cuda-nvtx=12.1.105=0
 - cuda-opencl=12.3.101=0
 - cuda-opencl-dev=12.3.101=0
 - cuda-profiler-api=12.3.101=0
 - cuda-runtime=12.1.0=0
 - datasets=2.16.0=pyhd8ed1ab_0
 - diffusers=0.25.0=pyhd8ed1ab_0
 - dill=0.3.7=pyhd8ed1ab_0
 - filelock=3.13.1=pyhd8ed1ab_0
 - freetype=2.12.1=hdaf720e_2
 - frozenlist=1.4.1=py311ha68e1ae_0
 - fsspec=2023.10.0=pyhca7485f_0
 - huggingface_hub=0.20.0=pyhd8ed1ab_0
 - idna=3.6=pyhd8ed1ab_0
 - importlib-metadata=7.0.1=pyha770c72_0
 - intel-openmp=2023.2.0=h57928b3_50497
 - jinja2=3.1.2=pyhd8ed1ab_1
 - krb5=1.21.2=heb0366b_0
 - lcms2=2.16=h67d730c_0
 - lerc=4.0.0=h63175ca_0
 - libabseil=20230802.1=cxx17_h63175ca_0
 - libarrow=14.0.2=h1048771_0_cpu
 - libarrow-acero=14.0.2=h63175ca_0_cpu
 - libarrow-dataset=14.0.2=h63175ca_0_cpu
 - libarrow-flight=14.0.2=h53b1db0_0_cpu
 - libarrow-flight-sql=14.0.2=h78eab7c_0_cpu
 - libarrow-gandiva=14.0.2=hb2eaab1_0_cpu
 - libarrow-substrait=14.0.2=hd4c9904_0_cpu
 - libblas=3.9.0=20_win64_mkl
 - libbrotlicommon=1.1.0=hcfcfb64_1
 - libbrotlidec=1.1.0=hcfcfb64_1
 - libbrotlienc=1.1.0=hcfcfb64_1
 - libcblas=3.9.0=20_win64_mkl
 - libcrc32c=1.1.2=h0e60522_0
 - libcublas=12.1.0.26=0
 - libcublas-dev=12.1.0.26=0
 - libcufft=11.0.2.4=0
 - libcufft-dev=11.0.2.4=0
 - libcurand=10.3.4.101=0
 - libcurand-dev=10.3.4.101=0
 - libcurl=8.5.0=hd5e4a3a_0
 - libcusolver=11.4.4.55=0
 - libcusolver-dev=11.4.4.55=0
 - libcusparse=12.0.2.55=0
 - libcusparse-dev=12.0.2.55=0
 - libdeflate=1.19=hcfcfb64_0
 - libevent=2.1.12=h3671451_1
 - libexpat=2.5.0=h63175ca_1
 - libffi=3.4.2=h8ffe710_5
 - libgoogle-cloud=2.12.0=h39f2fc6_4
 - libgrpc=1.59.3=h5bbd4a7_0
 - libhwloc=2.9.3=default_haede6df_1009
 - libiconv=1.17=hcfcfb64_2
 - libjpeg-turbo=3.0.0=hcfcfb64_1
 - liblapack=3.9.0=20_win64_mkl
 - liblapacke=3.9.0=20_win64_mkl
 - libnpp=12.0.2.50=0
 - libnpp-dev=12.0.2.50=0
 - libnvjitlink=12.1.105=0
 - libnvjitlink-dev=12.1.105=0
 - libnvjpeg=12.1.1.14=0
 - libnvjpeg-dev=12.1.1.14=0
 - libparquet=14.0.2=h7ec3a38_0_cpu
 - libpng=1.6.39=h19919ed_0
 - libprotobuf=4.24.4=hb8276f3_0
 - libre2-11=2023.06.02=h8c5ae5e_0
 - libsqlite=3.44.2=hcfcfb64_0
 - libssh2=1.11.0=h7dfc565_0
 - libthrift=0.19.0=ha2b3283_1
 - libtiff=4.6.0=h6e2ebb7_2
 - libutf8proc=2.8.0=h82a8f57_0
 - libuv=1.44.2=hcfcfb64_1
 - libwebp-base=1.3.2=hcfcfb64_0
 - libxcb=1.15=hcd874cb_0
 - libxml2=2.11.6=hc3477c8_0
 - libzlib=1.2.13=hcfcfb64_5
 - lz4-c=1.9.4=hcfcfb64_0
 - m2w64-gcc-libgfortran=5.3.0=6
 - m2w64-gcc-libs=5.3.0=7
 - m2w64-gcc-libs-core=5.3.0=7
 - m2w64-gmp=6.1.0=2
 - m2w64-libwinpthread-git=5.0.0.4634.697f757=2
 - markupsafe=2.1.3=py311ha68e1ae_1
 - mkl=2023.2.0=h6a75c08_50497
 - mkl-devel=2023.2.0=h57928b3_50497
 - mkl-include=2023.2.0=h6a75c08_50497
 - mpmath=1.3.0=pyhd8ed1ab_0
 - msys2-conda-epoch=20160418=1
 - multidict=6.0.4=py311ha68e1ae_1
 - multiprocess=0.70.15=py311ha68e1ae_1
 - networkx=3.2.1=pyhd8ed1ab_0
 - numpy=1.26.2=py311h0b4df5a_0
 - openjpeg=2.5.0=h3d672ee_3
 - openssl=3.2.0=hcfcfb64_1
 - orc=1.9.2=hf0b6bd4_0
 - packaging=23.2=pyhd8ed1ab_0
 - pandas=2.1.4=py311hf63dbb6_0
 - peft=0.7.1=pyhd8ed1ab_0
 - pillow=10.1.0=py311h4dd8a23_0
 - pip=23.3.2=pyhd8ed1ab_0
 - psutil=5.9.7=py311ha68e1ae_0
 - pthread-stubs=0.4=hcd874cb_1001
 - pthreads-win32=2.9.1=hfa6e2cd_3
 - pyarrow=14.0.2=py311h6a6099b_0_cpu
 - pyarrow-hotfix=0.6=pyhd8ed1ab_0
 - pysocks=1.7.1=pyh0701188_6
 - python=3.11.7=h2628c8c_1_cpython
 - python-dateutil=2.8.2=pyhd8ed1ab_0
 - python-slugify=8.0.1=pyhd8ed1ab_2
 - python-tzdata=2023.3=pyhd8ed1ab_0
 - python-xxhash=3.4.1=py311ha68e1ae_0
 - python_abi=3.11=4_cp311
 - pytorch=2.1.2=py3.11_cuda12.1_cudnn8_0
 - pytorch-cuda=12.1=hde6ce7c_5
 - pytorch-mutex=1.0=cuda
 - pytz=2023.3.post1=pyhd8ed1ab_0
 - pyyaml=6.0.1=py311ha68e1ae_1
 - re2=2023.06.02=hcbb65ff_0
 - regex=2023.12.25=py311ha68e1ae_0
 - requests=2.31.0=pyhd8ed1ab_0
 - safetensors=0.3.3=py311hc37eb10_1
 - scipy=1.11.4=py311h0b4df5a_0
 - setuptools=68.2.2=pyhd8ed1ab_0
 - six=1.16.0=pyh6c4a22f_0
 - snappy=1.1.10=hfb803bf_0
 - sympy=1.12=pyh04b8f61_3
 - tbb=2021.11.0=h91493d7_0
 - text-unidecode=1.3=pyhd8ed1ab_1
 - tk=8.6.13=h5226925_1
 - tokenizers=0.15.0=py311h91c4a10_1
 - tqdm=4.66.1=pyhd8ed1ab_0
 - transformers=4.36.2=pyhd8ed1ab_0
 - typing-extensions=4.9.0=hd8ed1ab_0
 - typing_extensions=4.9.0=pyha770c72_0
 - tzdata=2023d=h0c530f3_0
 - ucrt=10.0.22621.0=h57928b3_0
 - urllib3=2.1.0=pyhd8ed1ab_0
 - vc=14.3=hcf57466_18
 - vc14_runtime=14.38.33130=h82b7239_18
 - vs2015_runtime=14.38.33130=hcb4865c_18
 - wheel=0.42.0=pyhd8ed1ab_0
 - win_inet_pton=1.1.0=pyhd8ed1ab_6
 - xorg-libxau=1.0.11=hcd874cb_0
 - xorg-libxdmcp=1.1.3=hcd874cb_0
 - xxhash=0.8.2=hcfcfb64_0
 - xz=5.2.6=h8d14728_0
 - yaml=0.2.5=h8ffe710_2
 - yarl=1.9.3=py311ha68e1ae_0
 - zipp=3.17.0=pyhd8ed1ab_0
 - zstd=1.5.5=h12be248_0
 - pip:
 - compel==2.0.2
 - pyparsing==3.1.1
 - torchao==0.0.1
 - torchaudio==2.1.2
 - torchvision==0.16.2

Who can help?

@patrickvonplaten @yiyixuxu @sayakpaul

[sayakpaul]

Can you please tag the original contributor of that pipeline?

[a-r-r-o-w]

Pinging @xhinker.

I recently worked on img2img and inpaint support for lpw_xl. Andrew also spoke about adding support for IP Adapters and other similar additions to diffusers recently. I can take a look into adding support for clip_skip as well, or would you like to take this up as a PR?

[a-r-r-o-w]

As for cfg_scale, isn't it the same as the guidance_scale parameter?

[T145]

Perhaps, but at least on sites like CivitAI you see cfg_scale referenced far more. They're also independent values, so I'd say just to have support to pass it on would be nice. I think GS is more akin to a conscience and CS is like a coach: they each can create similar results but one definitely has more control. I just tested strength and cross_attention_kwargs which work just fine, so there aren't any other parameters to test that come to me.

If you're already doing work on a related feature go wild my friend!

[T145]

I just tested:

pipe.fuse_qkv_projections()
pipe.enable_freeu(b1=1.3, b2=1.4, s1=0.9, s2=0.2)

And both caused crashes.

Traceback (most recent call last):
 File "-\test.py", line 49, in <module>
 pipe.enable_freeu(b1=1.3, b2=1.4, s1=0.9, s2=0.2)
 ^^^^^^^^^^^^^^^^^
 File "-\diffusers\configuration_utils.py", line 142, in __getattr__
 raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'SDXLLongPromptWeightingPipeline' object has no attribute 'enable_freeu'