FEAT: Animate between two prompts using CLIP interpolation followed by latent space interpolation

[morganavr]

Playground:
https://replicate.com/andreasjansson/stable-diffusion-animation

Source code:
https://github.com/andreasjansson/cog-stable-diffusion/tree/animate

@andreasjansson GitHub user created a feature that works this way:

First a number of latents are generated by stable diffusion over an interpolation in CLIP space. Then the latents are interpolated and decoded into images and combined to a GIF.

I think this feature can be more powerful if it allows interpolation between two images that user already generated previously and saved their full info for recreation (prompt + parameters + seed).

Use case would be that user generates tons of images without this plugin. Then he picks two images that he likes the most and using this feature he could interpolate between them. For it to work full prompt info is required to get access to those 2 images.

Also I'd love the option for all intermediate images to be saved as separate files (possibly in separate folder for easier organization) - I am not really interested in one GIF file but to use this feature as a way to get interesting variations between two images.

On SD Discord at #community-research an author (andreas) mentioned a limitation of this feature:

i did some experiments with animating SD, interpolating in CLIP space: https://replicate.com/andreasjansson/stable-diffusion-animation. curious if anyone has ideas on how to make it less jumpy between frames!

[janekm]

This repo already stores the full prompt info in the PNG, so this is definitely possible and I think an interesting idea!

[bakkot]

This is a specific instance of a more general ability to interpolate between two sets of parameters, which I've suggested previously but not gotten around to implementing. If I'm reading the code correctly, the "latent space interpolation" is exactly this PR (except that my PR uses spherical coordinates, which probably works a little better), and the "CLIP interpolation" is the existing prompt weighting in this repo.

[andreasjansson]

I updated my demo to use spherical coordinates, indeed it works better!

[Dschogo]

Found another Implementation of this feature: https://github.com/nateraw/stable-diffusion-videos
Just as another Input, to help bringing this idea to life.