Make populating the internal symbol table thread-safe by leofang · Pull Request #835 · NVIDIA/cuda-python

leofang · 2025-08-13T14:00:12Z

Description

TODO

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2025-08-13T14:00:17Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

leofang · 2025-08-13T14:31:36Z

/ok to test 3fe6a3c

kkraus14 · 2025-08-13T19:16:35Z

/ok to test 67e543c

kkraus14 · 2025-08-13T19:21:47Z

+ cdef int err, driver_ver
+ with gil, __symbol_lock:
+ # Load driver to check version
+ handle = dlopen('libcuda.so.1', RTLD_NOW | RTLD_GLOBAL)


Unrelated to this PR, but... this handle is used to get the driver version, which is fed into the load_library call which doesn't use the driver version. This is likely a codegen issue, but we should probably just remove this?

Either way, wanted to call it out here but we can defer it to a future PR

kkraus14 · 2025-08-13T19:45:49Z

/ok to test 095999a

kkraus14 · 2025-08-13T20:51:17Z

Stopped CI because they all deadlocked at the start of testing. Likely caused a lock ordering issue here that we need to triage.

…rocAddress in the init function...

kkraus14 · 2025-08-14T00:56:21Z

/ok to test

copy-pr-bot · 2025-08-14T00:56:24Z

/ok to test

@kkraus14, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

kkraus14 · 2025-08-14T00:56:41Z

/ok to test 47c1c52

kkraus14 · 2025-08-14T19:24:21Z

/ok to test 5f4125e

…erlying C APIs

leofang · 2025-08-15T16:09:48Z

/ok to test 8e79272

kkraus14 · 2025-08-15T16:19:43Z

+ if __cuPythonInit:
+ return 0


With this outside of the lock if you have a bunch of threads trying to run this initially at the same time, you can end up in the situation where they all don't hit this early exit and all wait for the symbol lock and the reinitialize all of the symbols. I had done a quick and dirty local benchmark with a single thread acquiring a lock and doing nothing to understand the overhead in the early exit case and it was ~50ns.

Yeah I thought about this too. Good to know it's only ~50 ns. We should fix the codegen so that we check if symbol is null to avoid re-initialization, but it can be done later.

leofang · 2025-08-15T17:42:09Z

/ok to test 1672f40

leofang · 2025-08-18T11:29:01Z

Merging as per #836 (comment).

github-actions · 2025-08-18T11:43:00Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

github-project-automationBot added this to CCCL Aug 13, 2025

github-project-automationBot moved this to Todo in CCCL Aug 13, 2025

This comment was marked as outdated.
Sign in to view

leofang force-pushed the symbol_lock branch from 29be4b1 to 9031970Compare August 13, 2025 14:05

protect cuPythonInit in driver
3fe6a3c

leofang force-pushed the symbol_lock branch from 9031970 to 3fe6a3cCompare August 13, 2025 14:31

This comment has been minimized.
Sign in to view

leofang requested a review from kkraus14 August 13, 2025 14:55

leofang added this to the cuda-python parking lot milestone Aug 13, 2025

leofang added enhancement Any code-related improvements triage Needs the team's attention cuda.bindings Everything related to the cuda.bindings module labels Aug 13, 2025

leofang changed the title ~~WIP: Make internal symbol table thread-safe~~WIP: Make populating the internal symbol table thread-safeAug 13, 2025

leofang mentioned this pull request Aug 13, 2025
[Backport] Make populating the internal symbol table thread-safe #836
Merged
2 tasks

pentschev mentioned this pull request Aug 13, 2025
RuntimeError: Function "cuCtxGetCurrent" not foundrapidsai/rapidsmpf#434
Closed

add lock for all modules
67e543c

kkraus14 added 2 commits August 13, 2025 15:37

fixes
cca86c0

fix identation, make consistent
095999a

kkraus14 reviewed Aug 13, 2025
View reviewed changes

relocate setting __cuPythonInit to avoid deadlock since we use cuGetP…
47c1c52
…rocAddress in the init function...

kkraus14 reviewed Aug 14, 2025
View reviewed changes

Comment threadcuda_bindings/cuda/bindings/_bindings/cydriver.pyx.in Outdated

move init check inside lock
5f4125e

kkraus14 changed the title ~~WIP: Make populating the internal symbol table thread-safe~~Make populating the internal symbol table thread-safeAug 14, 2025

leofang assigned kkraus14 and leofang Aug 15, 2025

leofang added P0 High priority - Must do! and removed triage Needs the team's attention labels Aug 15, 2025

leofang commented Aug 15, 2025
View reviewed changes

Comment threadcuda_bindings/cuda/bindings/_bindings/cynvrtc.pyx.in Outdated
Comment threadcuda_bindings/cuda/bindings/_bindings/cydriver.pyx.in
Comment threadcuda_bindings/cuda/bindings/_bindings/cynvrtc.pyx.in
Comment threadcuda_bindings/cuda/bindings/_bindings/cydriver.pyx.in Outdated

make cuPythonInit reentrant + ensure GIL is released when calling und…
8e79272
…erlying C APIs

kkraus14 reviewed Aug 15, 2025
View reviewed changes

kkraus14 previously approved these changes Aug 15, 2025
View reviewed changes

github-project-automationBot moved this from Todo to In Review in CCCL Aug 15, 2025

kkraus14 reviewed Aug 15, 2025
View reviewed changes

Comment threadcuda_bindings/cuda/bindings/_bindings/cydriver.pyx.in Outdated

fix indentation
1672f40

leofang dismissed kkraus14’s stale review via 1672f40August 15, 2025 17:40

kkraus14 approved these changes Aug 15, 2025
View reviewed changes

leofang modified the milestones: cuda-python parking lot, cuda-python 13.0.1 & 12.9.2 Aug 15, 2025

leofang mentioned this pull request Aug 15, 2025
Bump version and add release notes #841
Merged
2 tasks

leofang merged commit db85867 into NVIDIA:mainAug 18, 2025
48 checks passed

github-project-automationBot moved this from In Review to Done in CCCL Aug 18, 2025

leofang deleted the symbol_lock branch August 18, 2025 11:29

pentschev mentioned this pull request Aug 18, 2025
[BUG]: The symbol table requires protection for thread-safety #852
Closed
1 task

leofang linked an issue Aug 18, 2025 that may be closed by this pull request
[BUG]: The symbol table requires protection for thread-safety #852
Closed
1 task

leofang mentioned this pull request Aug 19, 2025
Improve #789: Remove cyclical dependency between cuda.bindings.{driver|runtime} and c.b.utils #840
Merged

Uh oh!

Conversation

leofang commented Aug 13, 2025• edited by kkraus14 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-botBot commented Aug 13, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

leofang commented Aug 13, 2025

Uh oh!

This comment has been minimized.

kkraus14 commented Aug 13, 2025

Uh oh!

kkraus14Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

kkraus14Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

kkraus14 commented Aug 13, 2025

Uh oh!

kkraus14 commented Aug 13, 2025

Uh oh!

kkraus14 commented Aug 14, 2025

Uh oh!

copy-pr-botBot commented Aug 14, 2025

Uh oh!

kkraus14 commented Aug 14, 2025

Uh oh!

Uh oh!

kkraus14 commented Aug 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leofang commented Aug 15, 2025

Uh oh!

kkraus14Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

leofangAug 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leofang commented Aug 15, 2025

Uh oh!

leofang commented Aug 18, 2025

Uh oh!

Uh oh!

github-actionsBot commented Aug 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leofang commented Aug 13, 2025•
edited by kkraus14
Loading