Support H2D/D2D/D2H SMV copy by stiepan · Pull Request #1367 · NVIDIA/cuda-python

stiepan · 2025-12-11T23:46:02Z

Description

This PR implements copy_from/copy_to methods for StridedMemoryView.

For D2D we either run memcopyasync or launch a simple cuda kernel to elementwise copy between the views.
- Layouts are first permuted so that dst is as close as to C-stride as possible.
- Layouts are squeezed, flattened and vecotrized if possible.
- Each specialization is compiled once per arch and stored in a in-process cache.
For H2D/D2H, ideally we just run memcopyasync. If necessary, one or both H2H and D2D copies are run to coalesce, transpose, or scatter the data. The H2H copy is run with numpy.
If extra copies are needed and user did not specified allocators, we allocate: for device - from Device().memory_resource, for host - simply creating numpy array. Otherwise we use specified allocator, for host, the memory is ten wrapped into a numpy array.

Time-permittion or as a follow-up:

Add examples
Migrate tiled kernel from nvmath to speed-up transposing D2D copies.
When available (CTK 13), use default pinned memory pool instead of numpy as default host allocator (when Add PinnedMemoryResource and ManagedMemoryResource #1364 is merged)

closes

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2025-12-11T23:46:06Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

cpcloud · 2025-12-12T14:08:06Z

+ if hasattr(_tls, "strided_copy_include_dir"):
+ return _tls.strided_copy_include_dir


Suggested change
ifhasattr(_tls, "strided_copy_include_dir"):
return_tls.strided_copy_include_dir
if(strided_copy_include_dir :=getattr(_tls, "strided_copy_include_dir", None)) isnotNone:
return strided_copy_include_dir
This is one attribute access versus two (one inside hasattr, and then the attribute access itself).
Feel free to not use the := walrus operator if you don't like it.

cpcloud · 2025-12-12T14:08:52Z

+ if hasattr(_tls, "strided_copy_include_dir"):
+ return _tls.strided_copy_include_dir
+ cdef str current_dir = os.path.dirname(os.path.abspath(__file__))
+ cdef str copy_kernel_dir = os.path.normpath(os.path.join(current_dir, "..", "include", "strided_copy"))


Suggested change
cdef str copy_kernel_dir = os.path.normpath(os.path.join(current_dir, "..", "include", "strided_copy"))
cdef str copy_kernel_dir = os.path.normpath(os.path.join(current_dir, os.pardir, "include", "strided_copy"))
Probably not strictly necessary, but avoids a hardcoded string.

cpcloud · 2025-12-12T14:12:32Z

+ return True
+
+
+cdef inline bint _base_equal_strides(BaseLayout& a, BaseLayout& b) noexcept nogil:


Why is one underscore-prefixed (_base_equal_strides) and the other not (base_equal_shapes)?

It's because the shapes check the ndim equality first, while strides don't. I needed the shape equality for the copy, while strides not really. Maybe for completness it makes sense to add "public" version with ndim check too. :D

cpcloud · 2025-12-12T14:16:32Z

 return 0
+
+
+cdef inline copy_into(


inline-ing here seems a bit pointless and since many of these would be transitively inline'd (because many of the methods/functions are themselves inline), the code will be bigger.
Do we know if inline actually helps anything here? My preference would be to leave inline off unless it can be demonstrated that it actually meaningfully helps performance and doesn't bloat shared objects. It's very low risk to leave it off.

Yeah, here it's probably just unnecessary. Leaving definitions in layout.pxd file for basic shape/strides utilities actually made a perf difference, but for a function definied in the same pyx file I would guess compiler doesn't care if the inline is there. Removed it.

cpcloud

Mostly just nits, slight preference to avoid so much inlining unless we can demonstrate its effectiveness in a realistic benchmark.

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

leofang · 2025-12-13T01:07:46Z

/ok to test c761437

github-actions · 2025-12-13T01:17:15Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1367/
https://nvidia.github.io/cuda-python/pr-preview/pr-1367/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1367/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1367/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

…Device.memory_resource in custom allocator tests Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

oleksandr-pavlyk · 2025-12-15T03:32:55Z

/ok to test 5ceea4d

stiepan · 2025-12-15T10:34:51Z

Error: Unexpected HTTP response: 504

Looks like some (transient?) connectivity issues

rparolin · 2025-12-15T16:23:35Z

+{
+
+template <typename T, typename _coords_t>
+struct array_view


I wouldn't overload the name array_view since the std::array_view was the historical name for std::span. Given that this is implemented multi-dimensions I'd call it something like mdspan which would allow us to ultimately remove the implementation after the std::mdspan is usable in our code base.
https://en.cppreference.com/w/cpp/container/mdspan.html

rparolin · 2025-12-15T16:26:46Z

+};
+
+template <bool B, class T = void>
+struct enable_if


What is the benefit or reimplementing these core meta programming primitives? https://en.cppreference.com/w/cpp/types/enable_if.html

rparolin · 2025-12-15T16:28:25Z

+template <>
+struct min_val<int32_t>
+{
+ static constexpr int32_t value = -2147483648;


Why is std::numeric_limits not being used instead?
https://en.cppreference.com/w/cpp/types/numeric_limits.html
auto min_val = std::numeric_limits<int32_t>::min();

leofang · 2025-12-15T17:33:22Z

As per #674 (comment) let us close this PR. Thanks @stiepan for all the works, and sorry that we decided to not accept it in cuda.core.

Removed preview folders for the following PRs: - PR #1367

leofang assigned stiepan Dec 12, 2025

leofang added P1 Medium priority - Should do feature New feature or request cuda.core Everything related to the cuda.core module labels Dec 12, 2025

leofang added this to the cuda.core beta 10 milestone Dec 12, 2025

leofang added P0 High priority - Must do! and removed P1 Medium priority - Should do labels Dec 12, 2025

leofang reviewed Dec 12, 2025
View reviewed changes

Comment threadcuda_core/pyproject.toml Outdated

stiepan added 3 commits December 12, 2025 14:20

Support H2D/D2D/D2H SMV copy
a3a8bd3
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Add missing copyright header
5f2480c
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

More tests, typos, more concise pyproject includes
a22f2f6
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan force-pushed the support_memview_strided_copy branch from 8dad27c to a22f2f6Compare December 12, 2025 13:25

stiepan marked this pull request as ready for review December 12, 2025 13:27

cpcloud reviewed Dec 12, 2025
View reviewed changes

Comment threadcuda_core/cuda/core/experimental/_strided_copy/_jit.pyx Outdated

cpcloud reviewed Dec 12, 2025
View reviewed changes

Comment threadcuda_core/cuda/core/experimental/_memoryview.pyx

cpcloud reviewed Dec 12, 2025
View reviewed changes

stiepan added 2 commits December 12, 2025 21:30

Addressing review comments
012c31b
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Fix typos and spelling
c761437
Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan added 2 commits December 15, 2025 01:26

Fix tests: indicate device when creating cupy's ExternalStream, wrap …
a15aef0
…Device.memory_resource in custom allocator tests Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Merge branch 'main' into support_memview_strided_copy
5ceea4d

rparolin reviewed Dec 15, 2025
View reviewed changes

leofang closed this Dec 15, 2025

github-actionsBot pushed a commit that referenced this pull request Dec 16, 2025
Clean up PR preview folders for 1 closed/merged PRs
ca7f274
Removed preview folders for the following PRs: - PR #1367

fbusato mentioned this pull request Jan 22, 2026
[PoC] SoL Tensor Copy NVIDIA/cccl#7277
Open
1 task

leofang mentioned this pull request Feb 10, 2026
Cythonize _program.py #1565
Merged
4 tasks

		if hasattr(_tls, "strided_copy_include_dir"):
		return _tls.strided_copy_include_dir

-ifhasattr(_tls, "strided_copy_include_dir"):
-return_tls.strided_copy_include_dir
+if(strided_copy_include_dir :=getattr(_tls, "strided_copy_include_dir", None)) isnotNone:
+return strided_copy_include_dir

	cdef str copy_kernel_dir = os.path.normpath(os.path.join(current_dir, "..", "include", "strided_copy"))
	cdef str copy_kernel_dir = os.path.normpath(os.path.join(current_dir, os.pardir, "include", "strided_copy"))

		return True


		cdef inline bint _base_equal_strides(BaseLayout& a, BaseLayout& b) noexcept nogil:

Uh oh!

Conversation

stiepan commented Dec 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-botBot commented Dec 11, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cpcloud left a comment

Choose a reason for hiding this comment

Uh oh!

leofang commented Dec 13, 2025

Uh oh!

github-actionsBot commented Dec 13, 2025

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

oleksandr-pavlyk commented Dec 15, 2025

Uh oh!

stiepan commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leofang commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

stiepan commented Dec 11, 2025•
edited
Loading