[refactor] sincos and rope embeddings in torch instead of numpy

[a-r-r-o-w]

What does this PR do?

Internal discussion: https://huggingface.slack.com/archives/C065E480NN9/p1727418894443269

Code

importnumpyasnpimporttorchfromdiffusers.models.embeddingsimportget_1d_sincos_pos_embed_from_grid, aryan_get_1d_sincos_pos_embed_from_gridfromdiffusers.models.embeddingsimportget_2d_sincos_pos_embed_from_grid, aryan_get_2d_sincos_pos_embed_from_gridfromdiffusers.models.embeddingsimportget_2d_sincos_pos_embed, aryan_get_2d_sincos_pos_embedfromdiffusers.models.embeddingsimportget_3d_sincos_pos_embed, aryan_get_3d_sincos_pos_embedfromdiffusers.models.embeddingsimportget_2d_rotary_pos_embed, aryan_get_2d_rotary_pos_embedfromdiffusers.models.embeddingsimportget_3d_rotary_pos_embed, aryan_get_3d_rotary_pos_embed@torch.no_grad()deftest__get_1d_sincos_pos_embed_from_grid():
base_size=16interpolation_scale=1.0forembed_dimin [128, 256, 1024]:
forgrid_sizein [16, 32, 64, 128]:
numpy_grid=np.arange(grid_size, dtype=np.float32) / (grid_size/base_size) /interpolation_scalesincos_pos_embed_numpy=get_1d_sincos_pos_embed_from_grid(embed_dim, numpy_grid)
fordevicein ["cpu", "cuda"]:
torch_grid=torch.from_numpy(numpy_grid).to(device)
sincos_pos_embed_torch=aryan_get_1d_sincos_pos_embed_from_grid(embed_dim, torch_grid).cpu().numpy()
print(f"===== testing {test__get_1d_sincos_pos_embed_from_grid.__name__}({embed_dim=}, {grid_size=}, {device=}) =====")
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).max())
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).sum())
print("==========")
assertnp.allclose(sincos_pos_embed_numpy, sincos_pos_embed_torch, rtol=1e-12, atol=1e-12)
@torch.no_grad()deftest__get_2d_sincos_pos_embed_from_grid():
base_size=16interpolation_scale=1.0forembed_dimin [128, 256, 1024]:
forgrid_sizein [(32, 32), (64, 64), (128, 64), (64, 128)]:
grid_h=np.arange(grid_size[0], dtype=np.float32) / (grid_size[0] /base_size) /interpolation_scalegrid_w=np.arange(grid_size[1], dtype=np.float32) / (grid_size[1] /base_size) /interpolation_scalegrid=np.meshgrid(grid_w, grid_h)
grid=np.stack(grid)
grid_numpy=grid.reshape(2, 1, *grid_size)
sincos_pos_embed_numpy=get_2d_sincos_pos_embed_from_grid(embed_dim, grid_numpy)
fordevicein ["cpu", "cuda"]:
grid_torch=torch.from_numpy(grid_numpy).to(device)
sincos_pos_embed_torch=aryan_get_2d_sincos_pos_embed_from_grid(embed_dim, grid_torch).cpu().numpy()
print(f"===== testing {test__get_2d_sincos_pos_embed_from_grid.__name__}({embed_dim=}, {grid_size=}, {device=}) =====")
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).max())
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).sum())
print("==========")
assertnp.allclose(sincos_pos_embed_numpy, sincos_pos_embed_torch, rtol=1e-12, atol=1e-12)
@torch.no_grad()deftest__get_2d_sincos_pos_embed():
forbase_sizein [16, 32, 64]:
forinterpolation_scalein [1.0, 2.0]:
forembed_dimin [256, 1024]:
forgrid_sizein [(64, 64), (128, 64), (64, 128)]:
forcls_tokenin [False, True]:
forextra_tokensin [0, 16]:
sincos_pos_embed_numpy=get_2d_sincos_pos_embed(
embed_dim=embed_dim,
grid_size=grid_size,
cls_token=cls_token,
extra_tokens=extra_tokens,
interpolation_scale=interpolation_scale,
base_size=base_size,
)
fordevicein ["cpu", "cuda"]:
sincos_pos_embed_torch=aryan_get_2d_sincos_pos_embed(
embed_dim=embed_dim,
grid_size=grid_size,
cls_token=cls_token,
extra_tokens=extra_tokens,
interpolation_scale=interpolation_scale,
base_size=base_size,
device=device,
).cpu().numpy()
print(f"===== testing {test__get_2d_sincos_pos_embed.__name__}({base_size=}, {interpolation_scale=}, {embed_dim=}, {grid_size=}, {cls_token=}, {extra_tokens=}, {device=}) =====")
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).max())
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).sum())
print("==========")
assertnp.allclose(sincos_pos_embed_numpy, sincos_pos_embed_torch, rtol=1e-12, atol=1e-12)
@torch.no_grad()deftest__get_3d_sincos_pos_embed():
forembed_dimin [256, 1024]:
forspatial_sizein [(64, 64), (128, 64), (64, 128)]:
fortemporal_sizein [8, 16]:
forspatial_interpolation_scalein [1.0, 2.0]:
fortemporal_interpolation_scalein [1.0, 2.0]:
sincos_pos_embed_numpy=get_3d_sincos_pos_embed(
embed_dim=embed_dim,
spatial_size=spatial_size,
temporal_size=temporal_size,
spatial_interpolation_scale=spatial_interpolation_scale,
temporal_interpolation_scale=temporal_interpolation_scale,
)
fordevicein ["cpu", "cuda"]:
sincos_pos_embed_torch=aryan_get_3d_sincos_pos_embed(
embed_dim=embed_dim,
spatial_size=spatial_size,
temporal_size=temporal_size,
spatial_interpolation_scale=spatial_interpolation_scale,
temporal_interpolation_scale=temporal_interpolation_scale,
device=device,
).cpu().numpy()
print(f"===== testing {test__get_2d_sincos_pos_embed.__name__}({embed_dim=}, {spatial_size=}, {temporal_size=}, {spatial_interpolation_scale=}, {temporal_interpolation_scale=}) =====")
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).max())
print(np.abs(sincos_pos_embed_numpy-sincos_pos_embed_torch).sum())
print("==========")
assertnp.allclose(sincos_pos_embed_numpy, sincos_pos_embed_torch, rtol=1e-12, atol=1e-12)
@torch.no_grad()deftest__get_2d_rotary_pos_embed():
forembed_dimin [256, 1024]:
forcrops_coordsin [[(0, 0), (8, 8)], [(0, 0), (16, 16)]]:
forgrid_sizein [(64, 64), (128, 64), (64, 128)]:
rope_numpy=get_2d_rotary_pos_embed(
embed_dim=embed_dim,
crops_coords=crops_coords,
grid_size=grid_size,
)
fordevicein ["cpu", "cuda"]:
rope_torch=aryan_get_2d_rotary_pos_embed(
embed_dim=embed_dim,
crops_coords=crops_coords,
grid_size=grid_size,
)
rope_torch=rope_torch[0].cpu().numpy(), rope_torch[1].cpu().numpy()
print(f"===== testing {test__get_2d_sincos_pos_embed.__name__}({embed_dim=}, {crops_coords=}, {grid_size=}, {device=}) =====")
print(np.abs(rope_numpy[0] -rope_torch[0]).max())
print(np.abs(rope_numpy[1] -rope_torch[1]).max())
print(np.abs(rope_numpy[0] -rope_torch[0]).sum())
print(np.abs(rope_numpy[1] -rope_torch[1]).sum())
print("==========")
assertnp.allclose(rope_numpy, rope_torch, rtol=1e-12, atol=1e-12)
@torch.no_grad()deftest__get_3d_rotary_pos_embed():
forembed_dimin [256, 1024]:
forcrops_coordsin [[(0, 0), (8, 8)], [(0, 0), (16, 16)]]:
forgrid_sizein [(64, 64), (128, 64), (64, 128)]:
fortemporal_sizein [8, 16]:
rope_numpy=get_3d_rotary_pos_embed(
embed_dim=embed_dim,
crops_coords=crops_coords,
grid_size=grid_size,
temporal_size=temporal_size,
)
fordevicein ["cpu", "cuda"]:
rope_torch=aryan_get_3d_rotary_pos_embed(
embed_dim=embed_dim,
crops_coords=crops_coords,
grid_size=grid_size,
temporal_size=temporal_size,
device=device,
)
# ============== ============== ============== ============== ============== ============== ============== # NOTE/TODO: NOT SURE WHY THIS NEEDS HIGHER TOLERANCE EVEN THOUGH ALL THE OPERATIONS ARE SIMILAR-ISH TO THE 2D CASE# - IT IS EXACTLY ZERO ON CPU# - BUT ON CUDA, IT IS SLIGHTLY NUMERICALLY DIFFERENT# ============== ============== ============== ============== ============== ============== ============== rope_torch=rope_torch[0].cpu().numpy(), rope_torch[1].cpu().numpy()
print(f"===== testing {test__get_3d_sincos_pos_embed.__name__}({embed_dim=}, {crops_coords=}, {grid_size=}, {temporal_size=}, {device=}) =====")
print(np.abs(rope_numpy[0] -rope_torch[0]).max())
print(np.abs(rope_numpy[1] -rope_torch[1]).max())
print(np.abs(rope_numpy[0] -rope_torch[0]).sum())
print(np.abs(rope_numpy[1] -rope_torch[1]).sum())
print("==========")
assertnp.allclose(rope_numpy, rope_torch, rtol=1e-12, atol=1e-6)
test__get_1d_sincos_pos_embed_from_grid()
test__get_2d_sincos_pos_embed_from_grid()
test__get_2d_sincos_pos_embed()
test__get_3d_sincos_pos_embed()
test__get_2d_rotary_pos_embed()
test__get_3d_rotary_pos_embed()

importnumpyasnpassertnp.finfo(np.float32).eps==np.float32(1.1920929e-07)

Most of the numerical differences that are seen are okay to have when comparing numpy array with tensor-to-numpy array, and are within the magnitude of float32.eps (in fact, similar on the order of 1e-12 so these changes should be safe to make). They are sometimes even 0 if you do the comparison in tensors instead of numpy. The only peculiar case is the 3D rope embeddings, which seem to require a much higher tolerance (1e-6) instead of 1e-12-1e-15. I'm trying to figure out why :(

Once we finalize and decide to go forward with this, I'll rename the functions (currently prefixed), remove the older numpy based functions, and pass the tensor device and all usage occurrences so that all sincos/rope embedding creation occurs on device directly.

---

To those who don't have access to the internal link, you might wonder why we need this change? It's because using torch.compile leads to a graph break and cudaMemSync on models that create sincos/rope positional embeddings on-the-fly. This is due to creating numpy arrays and converting to cpu pytorch tensors and then moving to accelerator device. It's usually not a problem when prepared inside the pipeline, but for multiresolution training and specific models, you have to create it on the fly. These changes are to ensure that we remain in tensor land and don't have to deal with numpy arrays anywhere in the execution path.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu@sayakpaul

[sayakpaul]

Thanks! Seems to be in the right direction.

[sayakpaul]

We could compare this and the numpy implementation on the same inputs and see if there's any divergence. Usually a good start to try to localize if there are any problems.

[a-r-r-o-w]

It seems to match for all cases except the 3D sincos ones, based on my test code above. I'll give it a look again soon

[sayakpaul]

In case of an np.array, the device will be a CPU, right? Would we incur any device placement penalty for that in case we're on a non-CPU device?

[a-r-r-o-w]

If an np.array is passed, what you said is right. I did not want to redo this function because it was already mostly torch land, and now it supports the extra torch.Tensor case (previously, it only supported np.ndarray and int)

[sayakpaul]

We could move to raise ValueError() here and elsewhere applicable.

[a-r-r-o-w]

Sounds good!

[sayakpaul]

From the code, where we're calling these functions, I think it would be useful to always pass the right device. That way, we won't have to incur any device placement costs.

[a-r-r-o-w]

Yes ofcourse, sounds good!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

Closing in favor of @hlky's PRs