Skip to content

update IP-adapter code in UNetMotionModel - #6828

Merged
DN6 merged 1 commit into
mainfrom
ip-motion
Feb 5, 2024
Merged

update IP-adapter code in UNetMotionModel#6828
DN6 merged 1 commit into
mainfrom
ip-motion

Conversation

@yiyixuxu

Copy link
Copy Markdown
Collaborator

We updated UNet2dCondition when introducing multi-ip-adapter #6573. However, we forgot to make the same change to UnetMotionModel, so currently ip-adapter + AnimteDiff is not working

this PR fix IP-adapter + AnimateDiff

importtorchfromdiffusersimportAnimateDiffPipeline, DDIMScheduler, MotionAdapterfromdiffusers.utilsimportexport_to_giffromdiffusers.utilsimportload_imageadapter=MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2", torch_dtype=torch.float16)
pipeline=AnimateDiffPipeline.from_pretrained("emilianJR/epiCRealism", motion_adapter=adapter, torch_dtype=torch.float16)
scheduler=DDIMScheduler.from_pretrained(
"emilianJR/epiCRealism",
subfolder="scheduler",
clip_sample=False,
timestep_spacing="linspace",
beta_schedule="linear",
steps_offset=1,
)
pipeline.scheduler=schedulerpipeline.enable_vae_slicing()
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
pipeline.enable_model_cpu_offload()
ip_adapter_image=load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_inpaint.png")
output=pipeline(
prompt="A cute gummy bear waving",
negative_prompt="bad quality, worse quality, low resolution",
ip_adapter_image=ip_adapter_image,
num_frames=16,
guidance_scale=7.5,
num_inference_steps=50,
generator=torch.Generator(device="cpu").manual_seed(0),
)
frames=output.frames[0]
export_to_gif(frames, "yiyi_test_7_animation1.gif")

yiyi_test_7_animation1

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

DN6
DN6 approved these changes Feb 5, 2024

@DN6DN6 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@DN6
DN6 merged commit 64909f1 into mainFeb 5, 2024
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
fix
Co-authored-by: yiyixuxu <yixu310@gmail,com>
@yiyixuxu
yiyixuxu deleted the ip-motion branch July 14, 2024 19:27
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@yiyixuxu@HuggingFaceDocBuilderDev@DN6