[feature] implement TaylorSeer

[sayakpaul]

Currently, we support PAB and FasterCache. More details: https://huggingface.co/docs/diffusers/main/en/optimization/cache.

It'd be cool to support Accelerating Diffusion Models with TaylorSeers as it's quite neat and significantly better than others. Code is here:
https://github.com/Shenyi-Z/TaylorSeer

Before proceeding with a PR, please propose a design first in this thread. We can take it from there.

Hi @sayakpaul! I am lookin forward to it. But I've a few questions to understand the project:

1. Which pipelines should be prioritized? (SDXL, SD 1.5, Flux or all ?)
2. Should this follow the same pattern as PAB/FasterCache ?
3. Roughly how many files/pipelines would need modification? Is this a small, med or large contribution?
4. Also, can TayloSeer work alongside existing optimizations like PAB, FasterCache?

Based on your answers, I'll propose a design. Thanks!

[sayakpaul]

1. Very good question. In general, our methods tend to be very pipeline-agnostic. For example, FasterCache should work for most models. But we can prioritize the pipelines you mentioned (SDXL, Flux, QwenImage, etc.).
2. Yes, following a uniform structure is always helpful.
3. It's a significant contribution. You can get a sense of the needed modifications from this PR: [core] FasterCache #10163.
4. That's an interesting question. I think we can tackle that later. First priority, IMO, should be to have the method natively intrgrated.

Thanks @sayakpaul! This sounds like a great project. Let me review PR #10163 and the TaylorSeer code to understand the project. I'll get back to you in a few days on whether I can commit to this. Is that okay?

[sayakpaul]

Sounds perfect!

[toilaluan]

Hi @sayakpaul @Aznix07
I created a PR for this feature
I have written proposed design in the PR description as well as init code diff can desribe a bit
I expect to have it functionally done in 2 to 3 days for image models

[sayakpaul]

Could you check with @Aznix07 as well to see if they would be interested in collaborating? This is so everyone gets a fair chance to contribute something.

[toilaluan]

Yeah, since this Pr will be huge, include adding for video models, let's collab on it @Aznix07
The result from paper is very promising and hope we can onboard it soon to users
Lemme know your thoughts

[sayakpaul]

We can see till this week for a response from @Aznix07 :) We can also work in part PRs.

Thanks for waiting and for the collaboration offer! I've been reviewing the FasterCache PR and TaylorSeer implementation to understand the patterns.

@toilaluan, i'd be happy to collaborate! Since you're already started on image models, I can help with,

• Video model integration (once image models are done)
• Testing and documentation
• Or any other parts you'd like to split

Let me know what parts you'd like me to take on. Happy to review your current implementation and contribute where it helps most!

[toilaluan]

@Aznix07 hey sorry i haven't check this issue, i was focusing on the PR
The current pr state is ready for review, but i haven't done visualization benchmarking with video models but it should work well
And currently I have to manually reset hook state every time we run new pipeline.call, i think you can help with this as well
I also implemented lite version suggested by author

[toilaluan]

@sayakpaul Could you please review the PR?