Improve load_ip_adapter RAM Usage

[CyberVy]

Loading a model with torch_dtype is None using transformers.modeling_utils.PreTrainedModel will result in additional RAM usage because of data type conversion.
There is no conversion if the data type of the model weights is the same as torch_dtype.
And different conversions have different impacts on RAM usage which is mentioned in #10679.
When torch_dtype is None, PreTrainedModel will regard it as torch.float32, which causes a lot more RAM usage.

This PR is to improve this issue.

@asomoza

[asomoza]

nice! do you measure the ram savings?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hlky]

@bot /style

[github-actions]

Style fixes have been applied. View the workflow run here.

[CyberVy]

nice! do you measure the ram savings?

Yep! The peak CPU RAM usage decreases from 8.9G to 2.1G when I load an FP16 SDXL IP Adapter to GPU.

[CyberVy]

It's very easy to measure RAM savings in Colab.

Here are two minimal reproductions.

Now

fromtransformersimportCLIPVisionModelWithProjectionimporttorchmodel=CLIPVisionModelWithProjection.from_pretrained("eramth/ip-adapter",subfolder="sdxl_models/image_encoder",torch_dtype=torch.float16)
# The CPU RAM usage is about 2.3G.

model=model.to("cuda")
# The CPU RAM usage is still about 2.3G. And the VRAM usage is about 3.8G

# The below code can return the released CPU RAM to the system, so that we can observe the CPU RAM usage easily.importctypesctypes.CDLL("libc.so.6").malloc_trim(0)
# The CPU RAM usage is about 2.1G.

Before

fromtransformersimportCLIPVisionModelWithProjectionimporttorchmodel=CLIPVisionModelWithProjection.from_pretrained("eramth/ip-adapter",subfolder="sdxl_models/image_encoder")
# The CPU RAM usage is about 8.9G.

model=model.to("cuda",dtype=torch.float16)
# The CPU RAM usage is still about 6.4G. And the VRAM usage is about 3.8G

# The below code can return the released CPU RAM to the system, so that we can observe the CPU RAM usage easily.importctypesctypes.CDLL("libc.so.6").malloc_trim(0)
# The CPU RAM usage is about 2.2G.

Also you can get the same memory usage results directly loading an IP Adapter in diffusers.

[asomoza]

thanks a lot, I don't have a low RAM system so this is hard to catch for me at least. We have some errors in the tests which I believe aren't related to this PR. @hlky can you review this too please, there's a lot of failed tests so just to be sure.

[hlky]

Thanks @CyberVy. Nice improvement for low RAM users.

Failing tests are unrelated.