Quantized Flux with IP-Adapter

[hlky]

What does this PR do?

#10337 (comment)

Code (4bit)

fromdiffusersimportBitsAndBytesConfigasDiffusersBitsAndBytesConfigfromtransformersimportBitsAndBytesConfigasTransformersBitsAndBytesConfigimporttorchfromdiffusersimportFluxTransformer2DModel, FluxPipelinefromtransformersimportT5EncoderModelfromdiffusers.utilsimportload_imagequant_config=TransformersBitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
)
text_encoder_2_4bit=T5EncoderModel.from_pretrained(
"black-forest-labs/FLUX.1-dev",
subfolder="text_encoder_2",
quantization_config=quant_config,
torch_dtype=torch.float16,
)
quant_config=DiffusersBitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
)
transformer_4bit=FluxTransformer2DModel.from_pretrained(
"black-forest-labs/FLUX.1-dev",
subfolder="transformer",
quantization_config=quant_config,
torch_dtype=torch.float16,
)
pipe=FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
transformer=transformer_4bit,
text_encoder_2=text_encoder_2_4bit,
torch_dtype=torch.float16,
)
image=load_image(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flux_ip_adapter_input.jpg"
).resize((1024, 1024))
pipe.load_ip_adapter(
"XLabs-AI/flux-ip-adapter",
weight_name="ip_adapter.safetensors",
image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14",
torch_dtype=torch.float16,
)
pipe.enable_model_cpu_offload()
pipe.set_ip_adapter_scale(1.0)
image=pipe(
width=1024,
height=1024,
prompt="wearing sunglasses",
negative_prompt="",
true_cfg_scale=4.0,
generator=torch.Generator().manual_seed(4444),
ip_adapter_image=image,
).images[0]
image.save("flux_ip_adapter_output.jpg")

4bit	8bit

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

let's do this?

self.encoder_hid_proj.to(dtype=self.dtype, device=self.device)

if this works, can we apply this to other ip-adapter loader as well?

[hlky]

I don't think we need to cast it?

[yiyixuxu]

oh I think you're right!

[yiyixuxu]

can we do the same for the other ip-adapters?