CI: run cuda.bindings examples on Linux and Windows by rwgk · Pull Request #1517 · NVIDIA/cuda-python

rwgk · 2026-01-20T22:26:39Z

This PR is motivated by nvbug 5808967 / #1525 — currently there is no automatic testing for the cuda_bindings/examples at all, therefore it is possible that the QA team is side-tracked unnecessarily by failures that we can discover in the CI here automatically.

This PR enables running the cuda_bindings/examples in wheel-based test environments, which is both a gain and a simplification (see changes in cuda_bindings/examples/common/common.py).

Non-goal for this PR: structural changes to run the examples in various environments or in different ways.

Note that scripts/run_tests.sh runs the examples by default. This PR makes scripts/run_tests.sh succeed in local wheel-based environments (except for one unrelated failure in cuda_core, please ignore for the purpose of this PR):

before this PR: scripts_run_tests_main_log_2026-01-22+093402.txt

$ grep 'FAILED examples' scripts_run_tests_main_log_2026-01-22+093402.txt
FAILED examples/2_Concepts_and_Techniques/streamOrderedAllocation_test.py::main
FAILED examples/3_CUDA_Features/simpleCudaGraphs_test.py::main - AssertionError
FAILED examples/0_Introduction/clock_nvrtc_test.py::main - AssertionError
FAILED examples/0_Introduction/simpleZeroCopy_test.py::main - AssertionError
FAILED examples/0_Introduction/simpleCubemapTexture_test.py::main - Assertion...
FAILED examples/0_Introduction/systemWideAtomics_test.py::main - AssertionError
FAILED examples/0_Introduction/vectorAddDrv_test.py::main - AssertionError
FAILED examples/0_Introduction/vectorAddMMAP_test.py::main - AssertionError

with this PR: scripts_run_tests_pr1517_log_2026-01-22+100143.txt

$ grep 'FAILED examples' scripts_run_tests_pr1517_log_2026-01-22+100143.txt

(no output)

For completeness: I was hoping that our CI reproduces the Python 3.14t failure reported under nvbug 5808967, but it does not. But at least we know now, which helps in the search for the root cause.

copy-pr-bot · 2026-01-20T22:26:42Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

rwgk · 2026-01-20T22:27:41Z

/ok to test

Analysis: - examples were invoked via `python -m pytest` from within `cuda_bindings` so the repo checkout was on sys.path and imports resolved to the source tree - `setuptools_scm` generates `cuda/bindings/_version.py` only in the built wheel, so the source tree lacks this file and `from cuda.bindings._version import __version__` fails during example collection - running `pytest` via the installed entrypoint avoids CWD precedence and keeps imports coming from the installed wheel, which includes the generated version file Change: - switch Linux and Windows example steps to call `pytest` entrypoint

rwgk · 2026-01-20T23:20:30Z

/ok to test

mdboom

This only gets the example tests running in CI, not all the other ways that tests get run (calling pytest directly, or pixi run test etc.) It would be preferable to do this at a higher level -- maybe it's possible to make a symlink from cuda_bindings/tests/ to cuda_bindings/examples -- so that they will be included even for local development.

…es KernelHelper

…irely under cuda_bindings/examples/

rwgk · 2026-01-22T16:45:31Z

/ok to test

leofang · 2026-01-22T21:50:28Z

It would be preferable to do this at a higher level -- maybe it's possible to make a symlink from cuda_bindings/tests/ to cuda_bindings/examples -- so that they will be included even for local development.

Let's follow what I set up in cuda-core. All cuda-core examples are run as part of the regular tests, either locally or in the CI.

leofang · 2026-01-22T21:57:00Z

@rwgk for reproducing the bug, try replacing 3.13 here

cuda-python/ci/test-matrix.yml

Line 33 in 78ade5c

 - { ARCH: 'amd64', PY_VER: '3.13', CUDA_VER: '13.1.0', LOCAL_CTK: '1', GPU: 'h100', GPU_COUNT: '2', DRIVER: 'latest' }

with 3.14t so that we use free-threading Python.

rwgk · 2026-01-22T22:22:31Z

Let's follow what I set up in cuda-core. All cuda-core examples are run as part of the regular tests, either locally or in the CI.

CC @rparolin since this is about priorities: This is turning into a bigger project than expected. What should I do?

I agree the cuda-core approach is much better than what we have now, but this PR provides immediate CI coverage without blocking that direction.

I started this work expecting it'd be quick and help inform dealing with QA failures (nvbug 5808967).
Now a very different direction is requested, even though this PR is very small and doesn't create difficulties for the different direction.
It's an unknown to me how long it'll take to make the structural changes, especially at the moment: I have not gotten a complete CI run even for this easy PR, opened two days ago; currently rtxpro6000 jobs sit in the queue for hours.

My preference: Merge this PR and create a new issue for future proper prioritization: Rework organization of cuda_bindings/examples

rwgk · 2026-01-22T22:30:47Z

/ok to test

rwgk · 2026-01-22T22:34:06Z

for reproducing the bug, try replacing 3.13 here
cuda-python/ci/test-matrix.yml
Line 33 in 78ade5c
- { ARCH: 'amd64', PY_VER: '3.13', CUDA_VER: '13.1.0', LOCAL_CTK: '1', GPU: 'h100', GPU_COUNT: '2', DRIVER: 'latest' }
with 3.14t so that we use free-threading Python.

Done: commit dbfa3db

leofang · 2026-01-22T23:06:06Z

Here is our reproducer 🙂
https://github.com/NVIDIA/cuda-python/actions/runs/21267230884/job/61209868639?pr=1517#step:27:266

rwgk · 2026-01-22T23:20:41Z

Here is our reproducer 🙂 https://github.com/NVIDIA/cuda-python/actions/runs/21267230884/job/61209868639?pr=1517#step:27:266

Awesome! I'll work on taking care of that now, to get the nvbug out of limbo.

Keep pointer arrays alive through launches to avoid free-threaded Python misaligned-address failures caused by temporary argument buffers.

rwgk · 2026-01-23T00:08:49Z

/ok to test

rwgk · 2026-01-23T01:16:30Z

Wow, so many unrelated flakes! Five, I looked at all of them. The one we're most interested in just says "This job failed" with a big X, it didn't even start up.

Currently there is no job running, only 34 queued. I'll cancel and rerun. Nothing lost, but hopefully we'll get more lucky with the infrastructure.

rwgk · 2026-01-23T18:16:25Z

commit 7e03dd5 fixes nvbug 5808967 / #1525:

https://github.com/NVIDIA/cuda-python/actions/runs/21269516981/job/61302919967?pr=1517#step:27:129

examples/extra/isoFDModelling_test.py::main Checking for multiple GPUs...
CUDA-capable device count: 2
Checking GPU(s) for support of peer to peer memory access...
> Peer access from b'NVIDIA H100 NVL' (GPU0) -> b'NVIDIA H100 NVL' (GPU1) : Yes
> Peer access from b'NVIDIA H100 NVL' (GPU1) -> b'NVIDIA H100 NVL' (GPU0) : Yes
dt= 1.6666666666666667 delta= 25.0 nt= 600 freq max= 35.0
init object for device 0
init object for device 1
creating source on device 0
running create velocity on device 0
creating source on device 1
running create velocity on device 1
this code generates 90.05918964491383 GPoints/sec / device nz= 384 nx= 768
Done
PASSED
examples/extra/jit_program_test.py::main PASSED

copy-pr-bot · 2026-01-23T18:18:15Z

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

rwgk · 2026-01-23T18:19:28Z

@seberg is there a chance that you could review this one file? Did I apply your idea (#1525) correctly?

https://github.com/NVIDIA/cuda-python/pull/1517/changes#diff-61778604948d1f94800d64d8c1092fe04ac2fb8ba7129d7686b4ba97802bc41d

seberg

(only the _test.py changes)

Yeah, this is what I had in mind, since it is minimal and makes sure args are all alive at kernel launch time.

🤞 that this is the (only) issue underlying the nvbug.

github-actions · 2026-01-23T22:21:06Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

* ci: run cuda.core examples on Linux * ci: run cuda.core examples on Windows Add pytest step to run cuda.core example tests following the same pattern as cuda.bindings examples from PR NVIDIA#1517. The step runs `pytest -ra -s -vv tests/example_tests/` from within the cuda_core directory, matching the pattern used in QA testing. Trigger for this change: NVIDIA#1718 (comment) Made-with: Cursor

* ci: run cuda.core examples on Linux * ci: run cuda.core examples on Windows Add pytest step to run cuda.core example tests following the same pattern as cuda.bindings examples from PR #1517. The step runs `pytest -ra -s -vv tests/example_tests/` from within the cuda_core directory, matching the pattern used in QA testing. Trigger for this change: #1718 (comment) Made-with: Cursor

rwgk added 2 commits January 20, 2026 14:19

ci: run cuda.bindings examples on Linux
833e625

ci: run cuda.bindings examples on Windows
dbd6464

rwgk self-assigned this Jan 20, 2026

This comment has been minimized.
Sign in to view

mdboom requested changes Jan 22, 2026
View reviewed changes

rwgk added 2 commits January 22, 2026 08:35

use pathfinder.find_nvidia_header_directory() in cuda_bindings exampl…
95024c4
…es KernelHelper

Remove pytest_skipif_cuda_include_not_found() and get_cuda_home() ent…
7fa3f76
…irely under cuda_bindings/examples/

rwgk force-pushed the ci_cuda_bindings_examples branch from e34a864 to 7fa3f76Compare January 22, 2026 16:41

Merge branch 'main' into ci_cuda_bindings_examples
78ade5c

Replace py3.13 2-GPU job with 3.14t 2-GPU job
dbfa3db

rwgk added 2 commits January 22, 2026 16:06

Fix kernel arg lifetimes in isoFDModelling example
7e03dd5
Keep pointer arrays alive through launches to avoid free-threaded Python misaligned-address failures caused by temporary argument buffers.

Merge branch 'main' into ci_cuda_bindings_examples
403bba8

rwgk mentioned this pull request Jan 23, 2026
[BUG]: isoFDModelling_test.py use of temporaries is incorrect (undefined behavior) #1525
Closed
1 task

rwgk marked this pull request as ready for review January 23, 2026 18:18

seberg approved these changes Jan 23, 2026
View reviewed changes

Merge branch 'main' into ci_cuda_bindings_examples
aa5fd32

leofang approved these changes Jan 23, 2026
View reviewed changes

Comment threadcuda_bindings/examples/common/common.py

leofang merged commit 13de2c2 into NVIDIA:mainJan 23, 2026
1 check passed

leofang added bug Something isn't working P0 High priority - Must do! CI/CD CI/CD infrastructure cuda.bindings Everything related to the cuda.bindings module labels Jan 23, 2026

leofang added this to the cuda-python 13.1.2 & 12.9.6 milestone Jan 23, 2026

rwgk mentioned this pull request Jan 23, 2026
[FEA]: Avoid importing pytest in all cuda_bindings/examples #1530
Open
1 task

rwgk deleted the ci_cuda_bindings_examples branch January 23, 2026 23:12

leofang linked an issue Jan 24, 2026 that may be closed by this pull request
[BUG]: isoFDModelling_test.py use of temporaries is incorrect (undefined behavior) #1525
Closed
1 task

kkraus14 reviewed Jan 26, 2026
View reviewed changes

Comment threadcuda_bindings/examples/extra/isoFDModelling_test.py

leofang modified the milestones: cuda.bindings 13.1.2 & 12.9.6, cuda.bindings next Feb 17, 2026

rwgk mentioned this pull request Mar 4, 2026
cuda-core: add pixi examples env and fix runnable examples #1718
Merged
5 tasks

rwgk mentioned this pull request Mar 4, 2026
[CI]: run cuda.coreexample_tests on Linux and Windows #1722
Merged

rwgk mentioned this pull request Mar 20, 2026
Examples: Standardize example testing #1797
Closed

Uh oh!

Conversation

rwgk commented Jan 20, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-botBot commented Jan 20, 2026

Uh oh!

rwgk commented Jan 20, 2026

Uh oh!

This comment has been minimized.

rwgk commented Jan 20, 2026

Uh oh!

mdboom left a comment

Choose a reason for hiding this comment

Uh oh!

rwgk commented Jan 22, 2026

Uh oh!

leofang commented Jan 22, 2026

Uh oh!

leofang commented Jan 22, 2026

Uh oh!

rwgk commented Jan 22, 2026

Uh oh!

rwgk commented Jan 22, 2026

Uh oh!

rwgk commented Jan 22, 2026

Uh oh!

leofang commented Jan 22, 2026

Uh oh!

rwgk commented Jan 22, 2026

Uh oh!

rwgk commented Jan 23, 2026

Uh oh!

rwgk commented Jan 23, 2026

Uh oh!

rwgk commented Jan 23, 2026

Uh oh!

copy-pr-botBot commented Jan 23, 2026

Uh oh!

rwgk commented Jan 23, 2026

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rwgk commented Jan 20, 2026•
edited
Loading