[Core] introduce PeftAdapterMixin module.

[sayakpaul]

What does this PR do?

Delegate the peft related methods residing under ModelMixin to a separate class called PeftAdapterMixin following what's done in transformers: https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/peft.py#L43.

It also introduces a new module called integrations, again, following transformers. I see it playing to our advantage:

• modeling_utils.py becomes cleaner.
• Clear separation of concerns.
• If we want to add new integrations (such as with bitsandbytes), we'd have a better structure.

I don't suspect anything introduced in this PR to be backwards-breaking but happy to be proven wrong and fix it.

[DN6]

Makes sense to me. Think we could also do this for xformers.

[sayakpaul]

Think we could also do this for xformers.

xformers -- I think not as it's dealt similarly as SDPA, by means of an attention processor.

[patrickvonplaten]

Nice idea to factor out PEFT into a Mixin!
Two questions:

• 1.) It was already the case before, but do we really want to allow PEFT adaptors for all models. E.g. does it make sense that a VAE can also add LoRA layers?
• 2.) Can we maybe move the Mixin to src/diffusers/loaders ?

[sayakpaul]

It was already the case before, but do we really want to allow PEFT adaptors for all models. E.g. does it make sense that a VAE can also add LoRA layers?

This is what I think too. So, instead of adding the PEFT mixin to ModelMixin we can be selective and only add to the models that we know are used in LoRA training (from our official scripts). WDYT?

Can we maybe move the Mixin to src/diffusers/loaders ?

I am okay. I just followed what transformers did to have a unified structure. LMK.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

This is what I think too. So, instead of adding the PEFT mixin to ModelMixin we can be selective and only add to the models that we know are used in LoRA training (from our official scripts). WDYT?

I like this idea. It also keeps ModelMixin from getting bloated.

[sayakpaul]

This is out of the condition because:

• It's gonna be directly used in the core models (wherever applicable).
• In our core test environment, we don't install peft. So, if we add the above under is_peft_available(), we will run into import errors.

Can move it under is_torch_available(), though.

@DN6 let me know if you have a better way of handling this.

[sayakpaul]

Same as https://github.com/huggingface/diffusers/pull/6416/files#r1440090191.

[sayakpaul]

For the failing test: #6426 (comment).

[sayakpaul]

@patrickvonplaten addressed your comments:

• Moved PeftAdapterMixin to src/diffusers/loaders.
• Added PeftAdapterMixin as an inheritance to only a few selected models that are used in our training scripts.
• Added a doc.

This is my open question: #6416 (comment)

@DN6 LMK your thoughts too.

[patrickvonplaten]

Nice that works for me!

[sayakpaul]

@stevhliu feel free to edit the docs introduced in this PR in a follow-up.

[younesbelkada]

This is very clean thanks @sayakpaul and team!