[Community] Experimental AnimateDiff Image to Video (open to improvements)

[a-r-r-o-w]

What does this PR do?

Experimental img2vid support for AnimateDiff. Moved it here from #6328 due to the other being a core addition and it should make it easier to review vid2vid.

Colab

Fixes#6123.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@DN6

[a-r-r-o-w]

Some results can be found here: #6328 (comment)

[DN6]

Excellent 👍🏽

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

Nice!

[a-r-r-o-w]

Looks like the IP Adapter FaceID PR would require fixing in the community README. @DN6 Should I fix it here?

Edit: There's a few things that we could refactor in the community README. I think I'll take it up in a separate PR after this.

[DN6]

@a-r-r-o-w I think separate PR for the README works.

[Skquark]

I ran into an issue using pipeline_animatediff_img2video worth reporting.. Getting this import error:

File "/root/.cache/huggingface/modules/diffusers_modules/git/pipeline_animatediff_img2video.py", line 28, in <module>
from diffusers.models.unet_motion_model import MotionAdapter
ModuleNotFoundError: No module named 'diffusers.models.unet_motion_model'

To fix it, you'd change that line to:
from diffusers.models.unets.unet_motion_model import MotionAdapter
Looks like it was missing the .unets part as I see in animatediff_controlnet.py.. Tested in my custom at "AlanB/pipeline_animatediff_img2video_mod" and got it working in DiffusionDeluxe.com app. I also notice the output was grainy, thinking the DDIM scheduler code in the example may be incorrect too, gonna play with that some more to make sure. Thanks.

[DN6]

cc: @a-r-r-o-w

[a-r-r-o-w]

@Skquark Thanks for reporting the error! I believe this PR overlapped with the refactoring of the UNet modules and so this broke. If you'd like to, you can submit a PR fixing it otherwise I'll take it up sometime soon. Regarding the quality, AnimateDiff motion checkpoints are sensitive to the beta_schedule="linear" parameter so make sure that you set it when initializing the scheduler. The output quality here is not that great because of how the latents are created from an image (it is a method that the community figured out that produces ~handwavy results). If you're looking into it more deeply to improve result quality, maybe this comment can provide some useful information. I'd love if you can figure out how generation quality could be improved, and congrats on building what you have! But, I do not think much can be done to improve generation quality with a hacky trick like how the latents are created here, or atleast don't know of any such technique (maybe @jon-chuang could share his thoughts), and so I'd recommend using a model geared towards img2vid such as Personalized Image Animator or MotionCtrl SVD.

[Skquark]

Thanks, simple enough fix for an understandable refactoring break. I'd prefer it if you or someone else does that PR since it's an easy one-liner and my git fork is a mess, but I can try if preferred.. I also changed the way I loaded the scheduler to be the same as the video2video example using .from_pretrained, and it worked much better without the defective graininess, so may want to change that too in the usage example.. Worked right with this setting:
pipe.scheduler = DDIMScheduler.from_pretrained(model_id, subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", beta_schedule="linear", steps_offset=1)
instead of DDIMScheduler(beta_schedule="linear".... Another easy fix, got some pretty fun test results so far, looking forward to see what we can do with it next. I've got a nice collection of most of the good opensource Video AIs in my app, including PIA, I2VGen-XL, Video-to-Video, Controlnet Vid2Vid, SVD, Enhanced AnimateDiff from cli mod instead of HF, and all the others I could find. Will add MotionCtrl SVD when it's merged, and more as they come, they're all fun to play with... Now I'm anxious to see what SD3 will do for videos in the next leap. Anyways, here's my screenshots of this pipeline's implementation..