What API design would you like to have changed or added to the library? Why?
It would be great to have IP-adapters work with t2i-adapter as they're usually faster and less heavy on the image generation process than controlnet.
What use case would this enable or better enable? Can you give us a code example?
import torch
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, LCMScheduler
from diffusers.utils import load_image
model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
lcm_lora_id = 'latent-consistency/lcm-lora-sdxl'
t2i_adapter_id = 'TencentARC/t2i-adapter-depth-midas-sdxl-1.0'
adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16')
pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
pipe.load_lora_weights(lcm_lora_id)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
Current error:
AttributeError Traceback (most recent call last)
Cell In[6], line 12
9 adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16').to('cuda')
11 pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
---> 12 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
13 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
14 pipe.load_lora_weights(lcm_lora_id)
File ~/anaconda3/envs/diffusers2/lib/python3.9/site-packages/diffusers/configuration_utils.py:138, in ConfigMixin.getattr(self, name)
135 deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
136 return self._internal_dict[name]
--> 138 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'StableDiffusionXLAdapterPipeline' object has no attribute 'load_ip_adapter'
What API design would you like to have changed or added to the library? Why?
It would be great to have IP-adapters work with t2i-adapter as they're usually faster and less heavy on the image generation process than controlnet.
What use case would this enable or better enable? Can you give us a code example?
Current error:
AttributeError Traceback (most recent call last)
Cell In[6], line 12
9 adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16').to('cuda')
11 pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
---> 12 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
13 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
14 pipe.load_lora_weights(lcm_lora_id)
File ~/anaconda3/envs/diffusers2/lib/python3.9/site-packages/diffusers/configuration_utils.py:138, in ConfigMixin.getattr(self, name)
135 deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
136 return self._internal_dict[name]
--> 138 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'StableDiffusionXLAdapterPipeline' object has no attribute 'load_ip_adapter'