[AnimateDiff+Controlnet] Fix multicontrolnet support

[a-r-r-o-w]

What does this PR do?

Multicontrolnet did not work on community AnimateDiff+Controlnet pipeline due to incorrect check_inputs() implementation and a few missing statements. This PR fixes it.

Context: #5928 (comment)

Now you can do this:

Code

importtorchfromdiffusersimportAutoencoderKL, ControlNetModel, MotionAdapterfromdiffusers.pipelinesimportDiffusionPipelinefromdiffusers.schedulersimportDPMSolverMultistepSchedulerfromPILimportImagemotion_id="guoyww/animatediff-motion-adapter-v1-5-2"adapter=MotionAdapter.from_pretrained(motion_id)
controlnet1=ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_openpose", torch_dtype=torch.float16)
controlnet2=ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
vae=AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse", torch_dtype=torch.float16)
model_id="SG161222/Realistic_Vision_V5.1_noVAE"pipe=DiffusionPipeline.from_pretrained(
model_id,
motion_adapter=adapter,
controlnet=[controlnet1, controlnet2],
vae=vae,
custom_pipeline="pipeline_animatediff_controlnet",
).to(device="cuda", dtype=torch.float16)
pipe.scheduler=DPMSolverMultistepScheduler.from_pretrained(
model_id, subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", steps_offset=1, beta_schedule="linear",
)
pipe.enable_vae_slicing()
defload_video(file_path: str):
images= []
iffile_path.startswith(('http://', 'https://')):
# If the file_path is a URLresponse=requests.get(file_path)
response.raise_for_status()
content=BytesIO(response.content)
vid=imageio.get_reader(content)
else:
# Assuming it's a local file pathvid=imageio.get_reader(file_path)
forframeinvid:
pil_image=Image.fromarray(frame)
images.append(pil_image)
returnimagesvideo=load_video("dance.gif")
# You need to install it using `pip install controlnet_aux`fromcontrolnet_aux.processorimportProcessorp1=Processor("openpose_full")
cn1= [p1(frame) forframeinvideo]
p2=Processor("canny")
cn2= [p2(frame) forframeinvideo]
prompt="astronaut in space, dancing"negative_prompt="bad quality, worst quality, jpeg artifacts, ugly"result=pipe(
prompt=prompt,
negative_prompt=negative_prompt,
width=512,
height=768,
conditioning_frames=[cn1, cn2],
num_inference_steps=20,
)
fromdiffusers.utilsimportexport_to_gifexport_to_gif(result.frames[0], "result.gif")

Who can review?

@DN6@yiyixuxu@fkjkey

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[charchit7]

Looks good!

[patrickvonplaten]

Great job!