feat: qkv fusing

[llcnt]

Description

This PR introduces a new family of algorithm: FUSER FACTORIZER !
The general idea behind matrix fusing is to concatenate $n$ matrices together (eg $W_{large}=[W_{small}, \dots, W_{small}]$) and perform a single $W_{large} \cdot x$ product instead of $n$ smaller $W _{small} \cdot x$ products.
There is no memory gains. And, in general, there is no latency gains.
However, when combined with quantization, matrix fusing can enable speedups (see huggingface/diffusers#9185)

The main changes in the code are:

• Add a new FUSERFACTORIZER in the new fusingfactorizing folder;
• Add a new fuser factorizer for qkv matrices for diffusers models;
• Add a new check function that look at the models in the diffusers package that are compatible with fusing (ie have some FusedAttention processors);
• Add qkv_fusing for all compatible algorithms;
• Delete the original model (fixture) loaded during the unit test to avoid OOM errors.

Related Issue

Fixes #(issue number)

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

I have added a unit test for the new qkv fuser on sdv1.4.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Additional Notes

[nifleisch]

LGTM! 🙌

[sharpenb]

Thanks for the PR! I left some comments :)

[nifleisch]

Still awesome PR! I only have one small remark about the ModelContext.

[sharpenb]

Looks very good. It can be merged after renaming and fixing the safe_memory_cleanup :)

[davidberenstein1957]

@sharpenb@llcnt wrote a fix here. It seemed to be something broader that also impacted general loading.

#119

[sharpenb]

I only had a last comment :)

[sharpenb]

Super cool, let's merge it :)