Possibility to add filters to the generation process (modifying latents at each step)

[TimothyAlexisVass]

Is your feature request related to a problem? Please describe.
I've added filters to the inference process with a workaround solution.
It happens in the latent space.
It would be great if there was some similar solutio as callback called something like modifying_callback which would modify the latents at each step.

Additional context
The example below works similar to an unsharp mask.
I think the settings can be improved upon greatly (since using the ImageFilter.UnsharpMark from PIL will often give a better result), but it's promising.

The importance would be to open up the possibility to add filters to the process.
I think OpenCV may be a great candidate for this.

Example comparing sharpness = 2 and sharpness = 8 https://youtu.be/e2uf1_oQmGA

Example with manually applied unsharp mask on the resulting png where sharpness was set to 2
This was done with Pillow (PIL) ImageFilter image.filter(ImageFilter.UnsharpMask(radius = 0.777, percent = 345, threshold = 0))

[patrickvonplaten]

Agree that we need this! Very much related to: #4778 and #5266

[yiyixuxu]

@patrickvonplaten

I'm not sure this is related to the other two PRs, which I agree that we can solve by allowing users to set a schedule of guidance scale/prompt_embeddings

here we need to apply some transformation on the Unet output at each step and it seems quite different than the other two use cases, i.e. the guidance_scale and the new sharpness parameters are not dynamic and fixed for each timestep - let me know if I missed something here?

[patrickvonplaten]

It's mostly that we should think about how we can enable functionality in diffusers that allow the user to change different variables / states (guidance scale, prompt embeddings, unet output, ...) after each denoising step

[TimothyAlexisVass]

I don't suggest adding a sharpness parameter per say, but a sort of patching that could happen at each step.
Then, one way to use that could be to implement the sampling sharpness filter, which may not actually be the best example as it has been currently implemented in Fooocus though. I have gotten better result by just using Unsharp mask and sharpness from the Pillow library on the resulting PIL data.

But, it would open up the possibility to implement sharpness, among other things, on the latents, rather than the decoded result.

[TimothyAlexisVass]

I think this is as simple as this:

Just like we have callback we could have a modifier_callbackor whatever you want to call it... something like this:

def gaussian_blur(latents, timestep, blur):
 t = 1.0 - (timestep / 999.0)

 alpha = 0.001 * blur * t
 return gaussian_filter(latents) * alpha + latents * (1.0 - alpha)

def modifier_callback_function(steps, timestep, latents):
 return gaussian_blur(latents, timestep, 2) # The modifier_callback returns modified latents

pipe("LOVE", modifier_callback=modifier_callback_function)

But instead of only performing some stuff in parallel with the pipeline, it modifies the latents before they are used in the next step.

[TimothyAlexisVass]

Here is an example which I made with Diffusers and the workaround method:
https://github.com/huggingface/diffusers/assets/55708319/a49823a8-69b2-4cbc-8e75-a9776f27ba62

It's changing between large values to illustrate the filter effect clearly.

[TimothyAlexisVass]

And here is another example using the filter in a different way:
Smooth:

Regular(ish)

Extreme

[patrickvonplaten]

cc @yiyixuxu here

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[TimothyAlexisVass]

Thank you @yiyixuxu, @sayakpaul and @patrickvonplaten for this implementation. 😄
https://youtu.be/jjdrhIgLDvQ