[core] reuse AttentionMixin for compatible classes

[sayakpaul]

What does this PR do?

Many models use "# Copied from ..." implementations of attn_processors and set_attn_processor. They are basically the same as what we have implemented in

diffusers/src/diffusers/models/attention.py

Line 39 in 693d8a3

classAttentionMixin:

This PR makes those models inherit from AttentionMixin and removes the copied-over implementations.

I decided to leave fuse_qkv_projections and unfuse_qkv_projections out of this PR because some models don't have attention processors implemented in a way that would make this seamless. But the methods removed in this PR should be very harmless.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[dg845]

Perhaps it's out of the scope for this PR, but I see that a lot of models additionally have a set_default_attn_processor method, usually # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.set_default_attn_processor. Do you think it makes sense to add this method to AttentionMixin?

[sayakpaul]

IMO, not yet since AttentionMixin is fairly agnostic to the model-type but set_default_attn_processor relies on some custom attention processor types. For UNet2DConditionModel, we have:

diffusers/src/diffusers/models/unets/unet_2d_condition.py

Lines 762 to 769 in fa468c5

	ifall(proc.__class__inADDED_KV_ATTENTION_PROCESSORSforprocinself.attn_processors.values()):
	processor=AttnAddedKVProcessor()
	elifall(proc.__class__inCROSS_ATTENTION_PROCESSORSforprocinself.attn_processors.values()):
	processor=AttnProcessor()
	else:
	raiseValueError(
	f"Cannot call `set_default_attn_processor` when attention processors are of type {next(iter(self.attn_processors.values()))}"
	)

However, for AutoencoderKL Temporal Decoder:

diffusers/src/diffusers/models/autoencoders/autoencoder_kl_temporal_decoder.py

Lines 269 to 274 in fa468c5

	ifall(proc.__class__inCROSS_ATTENTION_PROCESSORSforprocinself.attn_processors.values()):
	processor=AttnProcessor()
	else:
	raiseValueError(
	f"Cannot call `set_default_attn_processor` when attention processors are of type {next(iter(self.attn_processors.values()))}"
	)

I'd be down to the refactoring, though. Cc: @DN6

[dg845]

Looks good to me! I think AuraFlowTransformer2DModel and AudioLDM2UNet2DConditionModel have their attn_processor/set_attn_processor methods deleted but are missing the corresponding change to inherit from AttentionMixin.

[sayakpaul]

Thanks for those catches, @dg845. Should have been fixed by now.

[dg845]

LGTM :)

[sayakpaul]

@DN6 okay to go?

[sayakpaul]

@DN6 a gentle ping.