Skip to content

Reduce overhead of bindings requiring cuPythonInit() - #894

Merged
mdboom merged 3 commits into
NVIDIA:mainfrom
mdboom:faster_cupythoninit
Aug 26, 2025
Merged

Reduce overhead of bindings requiring cuPythonInit()#894
mdboom merged 3 commits into
NVIDIA:mainfrom
mdboom:faster_cupythoninit

Conversation

@mdboom

@mdboommdboom commented Aug 22, 2025

Copy link
Copy Markdown
Contributor

Description

This reduces the calling overhead of binding functions that require cuPythonInit or cudaPythonInit to be called.

This reduces the time it takes to call driver.cuDeviceGet(0) (for example) by about 50ns (on my machine):

  • Base: Mean +- std dev: 149 ns +- 8 ns
  • This PR: Mean +- std dev: 101 ns +- 2 ns

Note that these times include the work done by the actual underlying cuDeviceGet call, not just the function call overhead.

Why this works

The cuPythonInit function is extremely large, so no C compiler is likely to ever inline it. By creating a small wrapper function just to check the init flag and then delegate to the big function, the C compiler inlines it. This not only removes a C function call, but probably helps out the branch predictor when checking the flag.

closes

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@mdboom
mdboom requested review from Copilot and leofangAugust 22, 2025 16:36

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes the initialization check for CUDA Python bindings by reducing function call overhead. The optimization splits the existing cuPythonInit() and cudaPythonInit() functions into two parts: a small wrapper function that checks if initialization has already occurred, and a larger function that performs the actual initialization work.

  • Refactors initialization functions to use a small wrapper pattern for better compiler inlining
  • Moves the initialization flag check to a separate small function to enable C compiler optimization
  • Applies the same pattern across three binding files for consistency

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

FileDescription
cyruntime.pyx.inSplits cudaPythonInit() into wrapper and implementation functions
cynvrtc.pyx.inSplits cuPythonInit() into wrapper and implementation functions
cydriver.pyx.inSplits cuPythonInit() into wrapper and implementation functions
Comments suppressed due to low confidence (1)

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@mdboom

Copy link
Copy Markdown
ContributorAuthor

/ok to test

@github-actions

This comment has been minimized.

@leofangleofang added enhancement Any code-related improvements P1 Medium priority - Should do cuda.bindings Everything related to the cuda.bindings module to-be-backported Trigger the bot to raise a backport PR upon merge labels Aug 22, 2025
@leofangleofang added this to the cuda-python parking lot milestone Aug 22, 2025

@leofangleofang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also add a release note to 13.X.Y. (I want to backport it to 12.9.X, so perhaps also a good idea to touch its release note. Note that we only generate docs on the main branch #809.)

Also we need to backport this to the codegen 🙂

Comment threadcuda_bindings/cuda/bindings/_bindings/cydriver.pyx.in
@mdboom

Copy link
Copy Markdown
ContributorAuthor

Let's also add a release note to 13.X.Y. (I want to backport it to 12.9.X, so perhaps also a good idea to touch its release note. Note that we only generate docs on the main branch #809.)

👍

Also we need to backport this to the codegen 🙂

Sure, will do. The upstream of the generator at git@github.com:NVIDIA/cuda-python-private.git doesn't seem to have the free-threading fix yet (so running the generator for this change reverts all that). Is that just because that fix hasn't been implemented in the generator yet, or am I using the wrong branch or something on my end?

@leofang

Copy link
Copy Markdown
Member

I guess you're blocked by me now... let me get to it asap

@github-project-automationgithub-project-automationBot moved this from Todo to In Review in CCCLAug 22, 2025
@leofang

Copy link
Copy Markdown
Member

/ok to test 4c6a057

@mdboom
mdboom merged commit 8fb7a88 into NVIDIA:mainAug 26, 2025
48 checks passed
@github-project-automationgithub-project-automationBot moved this from In Review to Done in CCCLAug 26, 2025
@github-actions

Copy link
Copy Markdown

Backport failed for 12.9.x, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 12.9.x
git worktree add -d .worktree/backport-894-to-12.9.x origin/12.9.x
cd .worktree/backport-894-to-12.9.x
git switch --create backport-894-to-12.9.x
git cherry-pick -x 8fb7a885ea61ec0bad4c34587011370077c54ea2

@github-actions

This comment has been minimized.

1 similar comment
@github-actions

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

mdboom added a commit to mdboom/cuda-python that referenced this pull request Aug 28, 2025
@mdboommdboom mentioned this pull request Aug 28, 2025
2 tasks
kkraus14 pushed a commit that referenced this pull request Aug 28, 2025
@cpcloudcpcloud mentioned this pull request Oct 8, 2025
@mdboom
mdboom deleted the faster_cupythoninit branch December 9, 2025 16:13
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.bindingsEverything related to the cuda.bindings moduleenhancementAny code-related improvementsP1Medium priority - Should doto-be-backportedTrigger the bot to raise a backport PR upon merge

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants

@mdboom@leofang