[LoRA] support non-diffusers hidream loras

[sayakpaul]

What does this PR do?

Fixes: #11383

To check:

fromdiffusersimportDiffusionPipelinefromtransformersimportPreTrainedTokenizerFast, LlamaForCausalLMimporttorchrepo_id="HiDream-ai/HiDream-I1-Full"tokenizer_4=PreTrainedTokenizerFast.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
text_encoder_4=LlamaForCausalLM.from_pretrained(
"meta-llama/Meta-Llama-3.1-8B-Instruct",
output_hidden_states=True,
output_attentions=True,
torch_dtype=torch.bfloat16,
)
pipeline=DiffusionPipeline.from_pretrained(
repo_id, tokenizer_4=tokenizer_4,
text_encoder_4=text_encoder_4,
torch_dtype=torch.bfloat16
).to("cuda")
pipeline.load_lora_weights("sayakpaul/different-lora-from-civitai", weight_name="486_hidream.safetensors")

Cc: @vladmandic

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

thanks!