CI: Run some tests with compute-sanitizer by carterbox · Pull Request #566 · NVIDIA/cuda-python

carterbox · 2025-04-22T18:06:39Z

Description

Runs python 3.12 pytests in the context of compute-sanitizer to check for memory issues and errors from the CUDA API.

closes#565
closes#562

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2025-04-22T18:06:43Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

leofang · 2025-04-22T19:24:56Z

FYI right now we use the mini-CTK approach in the CI:

local CTK: we only grab what we need with a custom GHA

cuda-python/.github/actions/fetch_ctk/action.yml

Lines 85 to 92 in d425a88

 populate_cuda_path cuda_nvcc
 populate_cuda_path cuda_cudart
 populate_cuda_path cuda_nvrtc
 populate_cuda_path cuda_profiler_api
 populate_cuda_path cuda_cccl
 if [[ "$(cut -d '.' -f 1 <<< ${{ inputs.cuda-version }})" -ge 12 ]]; then
 populate_cuda_path libnvjitlink
 fi

wheel: same is done via the optional dependencies
cuda-python/.github/workflows/test-wheel-linux.yml
Lines 218 to 222 in d425a88
if [[ "${{ inputs.local-ctk }}" == 1 ]]; then
pip install *.whl
else
pip install $(ls *.whl)[all]
fi
conda: not yet implemented (CI: Test conda-based workflows #280) but should be straightforward as you know

So compute-sanitizer is currently not available in the CI. But I assume it can be grabbed easily.

@cryos is refactoring our CI (#555). I suggest we perhaps add another standalone pipeline for running compute sanitizer?

cryos · 2025-04-22T19:58:28Z

I was going to look at tools like this next, that is a great point and something I can factor in. Looking at the proposal here picking out a test run would be reasonable, I know there are other tools we would like to run too.

carterbox · 2025-04-22T22:16:15Z

/ok to test f27e1f4

leofang · 2025-04-22T22:25:46Z

I doubt commit f27e1f4 would work -- we'll see: #571.

carterbox · 2025-04-23T17:34:24Z

/ok to test 05a70689

carterbox · 2025-04-23T21:15:16Z

/ok to test dde857b

carterbox · 2025-04-23T21:57:10Z

Yay! The linux-64 tests are failing for the correct reason! (the compute sanitizer returns non-zero because it has detected issues).

https://github.com/NVIDIA/cuda-python/actions/runs/14628267586/job/41045781037?pr=566

Windows tests are failing because I have disabled them partially.

carterbox · 2025-04-24T18:26:05Z

/ok to test a1ea51e

There is no compute-sanitizer wheel, so we can only run when the ctk is installed system-wide

Because the sanitizer commands depend on the version of the sanitizer we need to be able to run the sanitzer to set the sanitizer cmd. Thus, we need to setup the sanitzer after it is installed.

carterbox · 2025-04-24T20:28:24Z

/ok to test 0430930

carterbox · 2025-04-25T20:27:12Z

/ok to test 9c25910

carterbox · 2025-04-25T21:47:34Z

/ok to test 7fb013e

copy-pr-bot · 2025-04-25T22:48:23Z

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

leofang · 2025-04-26T03:29:26Z

Let's get #562 (comment) addressed and merge!

…sanitizer-to-ci

carterbox · 2025-04-28T16:30:09Z

/ok to test 675c41f

carterbox · 2025-04-28T17:42:38Z

/ok to test 5abd6e3

rwgk

Awesome, I'm really happy that we'll be routinely testing with the compute sanitizer.

carterbox · 2025-04-28T20:59:02Z

/ok to test 53a01cb

rwgk

Thanks @carterbox!

leofang

Very neat, thanks!

@cryos are you OK if we merge this now, or if you'd rather wait until we merge the CI refactoring (#555)?

kkraus14 · 2025-04-29T14:46:08Z

@cryos are you OK if we merge this now, or if you'd rather wait until we merge the CI refactoring (#555)?

Marcus is on PTO until May 5 so lets merge this and we can sync with him once he's back 😄

leofang · 2025-04-29T16:27:08Z

Right he told me about it but I forgot... let's merge now.

github-actions · 2025-04-29T16:46:00Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

leofang requested review from cryos and leofang April 22, 2025 19:25

leofang assigned carterbox Apr 22, 2025

leofang added P1 Medium priority - Should do CI/CD CI/CD infrastructure labels Apr 22, 2025

This comment has been minimized.
Sign in to view

cryos linked an issue Apr 23, 2025 that may be closed by this pull request
CI: Refactor the CI matrix #576
Closed

cryos removed a link to an issue Apr 23, 2025
CI: Refactor the CI matrix #576
Closed

carterbox added 10 commits April 24, 2025 15:25

CI: Add compute-sanitizer paths to linux test environment
843130f

CI: Run pytest in the context of compute-sanitizer
c8df0dc

CI: Only use compute-sanitizer for tests of one python version
4ac79d0

CI: Add non-zero exitcode to compute-sanitizer
be912c9

CI: Only run compute-sanitzer when testing against local ctk
865eeb5
There is no compute-sanitizer wheel, so we can only run when the ctk is installed system-wide

CI: Delay CUDA_HOME variable expansion
ddd0714

CI: Move compute-sanitzer setup into own step after CTK setup
5079605
Because the sanitizer commands depend on the version of the sanitizer we need to be able to run the sanitzer to set the sanitizer cmd. Thus, we need to setup the sanitzer after it is installed.

CI: Optionally skip tests that raise CUDA API errors
4a03465

CI: Add sanitizer skip environment variable to CI
bd88039

DOC: Fix spelling of CI step name
0430930

carterbox force-pushed the dching/add-compute-sanitizer-to-ci branch from a1ea51e to 0430930Compare April 24, 2025 20:26

leofang added the cuda.bindings Everything related to the cuda.bindings module label Apr 25, 2025

BUG: Correctly spell "version"
7fb013e

carterbox marked this pull request as ready for review April 25, 2025 22:48

carterbox requested a review from leofang April 25, 2025 22:48

leofang reviewed Apr 26, 2025
View reviewed changes

Comment threadcuda_bindings/docs/source/environment_variables.md Outdated
Comment thread.github/workflows/test-wheel-linux.yml Outdated

rwgk reviewed Apr 26, 2025
View reviewed changes

Comment thread.github/workflows/test-wheel-linux.yml Outdated
Comment threadcuda_bindings/tests/test_cuda.py Outdated

carterbox added 5 commits April 28, 2025 10:43

DOC: Fix spelling of sanitizer
2632bbc

TST: Define new test skip in conftest instead of copy-paste
2bfd402

Merge remote-tracking branch 'upstream/main' into dching/add-compute-…
3f7a79b
…sanitizer-to-ci

TST: Cleanup post-merge
02559a3

CI: Remove COMPUTE_SANITIZER_VERSION variable
675c41f

carterbox requested review from leofang and rwgk April 28, 2025 16:30

TST: Skip another post-merge error
5abd6e3

rwgk previously approved these changes Apr 28, 2025
View reviewed changes

Comment thread.github/workflows/test-wheel-linux.yml Outdated

TST: Use consistent name for environment variable and test skip function
53a01cb

carterbox dismissed rwgk’s stale review via 53a01cbApril 28, 2025 20:58

rwgk approved these changes Apr 28, 2025
View reviewed changes

leofang approved these changes Apr 29, 2025
View reviewed changes

leofang merged commit 034ffbf into NVIDIA:mainApr 29, 2025

leofang mentioned this pull request Apr 29, 2025
Doc: mention about compute-sanitizer errors #589
Closed

Uh oh!

Conversation

carterbox commented Apr 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-botBot commented Apr 22, 2025

Uh oh!

leofang commented Apr 22, 2025

Uh oh!

cryos commented Apr 22, 2025

Uh oh!

carterbox commented Apr 22, 2025

Uh oh!

leofang commented Apr 22, 2025

Uh oh!

This comment has been minimized.

carterbox commented Apr 23, 2025

Uh oh!

carterbox commented Apr 23, 2025

Uh oh!

carterbox commented Apr 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carterbox commented Apr 24, 2025

Uh oh!

carterbox commented Apr 24, 2025

Uh oh!

carterbox commented Apr 25, 2025

Uh oh!

carterbox commented Apr 25, 2025

Uh oh!

copy-pr-botBot commented Apr 25, 2025

Uh oh!

leofang commented Apr 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carterbox commented Apr 28, 2025

Uh oh!

carterbox commented Apr 28, 2025

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carterbox commented Apr 28, 2025

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

leofang left a comment

Choose a reason for hiding this comment

Uh oh!

kkraus14 commented Apr 29, 2025

Uh oh!

leofang commented Apr 29, 2025

Uh oh!

github-actionsBot commented Apr 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

carterbox commented Apr 22, 2025•
edited
Loading

carterbox commented Apr 23, 2025•
edited
Loading