Is your feature request related to a problem? Please describe.
I would like to be able to guide the diffusion process with a classifier (or any other method) for controllable generation (note: not to be confused with classifier-free guidance). In my case specifically, I can't work out how to guide Stable Diffusion using a separate model, though there are examples of CLIP guidance and so on in Diffusers (as standalone pipes).
Describe the solution you'd like
# Function produces guidance for latents e.g. gradients from a classifier applied to vae.encode(latents)defguidance(latents, step, ...):
...
returnlatents+changepipe(prompts, guidance_fn=guidance)
Something along these lines; a flexible API allowing for arbitrary guidance. Alternatively, guidance_fn could return just change, and a separate guidance_fn_weight parameter (for example) determines the multiplier for this change.
Describe alternatives you've considered
I have looked around in the API and don't believe this is possible without somehow running the pipe for a single step, applying updates outside the pipe, running the pipe for a step again etc. I think the proposed solution is a lot cleaner and shouldn't be hard to implement (though perhaps it exists somewhere already, apologies if so).
Additional context
In the context of diffusion models, guidance using other models is a fairly common approach to controlled generation.
Is your feature request related to a problem? Please describe.
I would like to be able to guide the diffusion process with a classifier (or any other method) for controllable generation (note: not to be confused with classifier-free guidance). In my case specifically, I can't work out how to guide Stable Diffusion using a separate model, though there are examples of CLIP guidance and so on in Diffusers (as standalone pipes).
Describe the solution you'd like
Something along these lines; a flexible API allowing for arbitrary guidance. Alternatively,
guidance_fncould return justchange, and a separateguidance_fn_weightparameter (for example) determines the multiplier for this change.Describe alternatives you've considered
I have looked around in the API and don't believe this is possible without somehow running the pipe for a single step, applying updates outside the pipe, running the pipe for a step again etc. I think the proposed solution is a lot cleaner and shouldn't be hard to implement (though perhaps it exists somewhere already, apologies if so).
Additional context
In the context of diffusion models, guidance using other models is a fairly common approach to controlled generation.