Fix failing CogVideoX LoRA fuse test

[a-r-r-o-w]

Context: https://huggingface.slack.com/archives/C065E480NN9/p1734937335392109?thread_ts=1734935953.846629&cid=C065E480NN9

Fixes test: https://github.com/huggingface/diffusers/actions/runs/12461857058/job/34781970659#step:6:6623

This went uncaught in #10270 because it only happens when .to() is not called on a pipeline, or torch_dtype is not specified while loading transformer component. The failing lora test does neither, so the alternative branch of code was never tested.

pos_embedding is torch.int64, which casts embeds to torch.int64 as well unless casted correctly in both branches

[sayakpaul]

I know you have run the concerned test on a GPU but just double-checking.

[a-r-r-o-w]

Yep

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.