Skip to content

cuda.core: validate ctypes host callback signatures against CUhostFn - #2525

Merged
Andy-Jost merged 5 commits into
NVIDIA:mainfrom
Andy-Jost:ajost/validate-ctypes-host-callback
Aug 7, 2026
Merged

cuda.core: validate ctypes host callback signatures against CUhostFn#2525
Andy-Jost merged 5 commits into
NVIDIA:mainfrom
Andy-Jost:ajost/validate-ctypes-host-callback

Conversation

@Andy-Jost

Copy link
Copy Markdown
Contributor

Summary

Closes#2439.

cuda.core accepted any ctypes._CFuncPtr as a host callback and cast its address to CUhostFn without checking the declared prototype. CUDA requires void (*)(void*); a mismatched restype, argument list, or calling convention is undefined behavior when the driver invokes the callback.

_resolve_host_callback now validates ctypes prototypes against that ABI before the pointer reaches CUDA. Construction (GraphNode.callback / GraphBuilder.callback) and update paths share the check. Valid CFUNCTYPE(None, c_void_p) callbacks keep working; on Windows WINFUNCTYPE is also accepted. Docs, stubs, and the 1.2.0 release notes are updated accordingly.

Changes

  • Reject incompatible ctypes host-callback signatures with TypeError (message names CUhostFn and suggests a conforming declaration or a Python callable)
  • Document the required ABI on callback / update APIs
  • Add construction, update, and stream-capture rejection tests (plus Windows WINFUNCTYPE accept, skipped on Linux)
  • Note the stronger checking in cuda_core/docs/source/release/1.2.0-notes.rst

Reject incompatible ctypes prototypes before CUDA sees them, document
the required ABI, and note the stronger checking in the 1.2.0 release notes.
Use getattr for private ctypes calling-convention constants so the
regenerated _host_callback.pyi type-checks cleanly.
@Andy-JostAndy-Jost added this to the cuda.core next milestone Aug 6, 2026
@Andy-JostAndy-Jost added bug Something isn't working P1 Medium priority - Should do cuda.core Everything related to the cuda.core module labels Aug 6, 2026
@Andy-JostAndy-Jost self-assigned this Aug 6, 2026
@Andy-Jost
Andy-Jost requested a review from juenglinAugust 6, 2026 21:44
@github-actions

This comment has been minimized.

The previous check inspected ctypes' private _flags_ bits to identify the
calling convention. That is wrong on Windows: CPython defines
FUNCFLAG_STDCALL as 0, so a bitwise test can never match WINFUNCTYPE, and
every win-64 test job rejected a valid callback. The 0x2 fallback used when
_ctypes.FUNCFLAG_STDCALL is absent is FUNCFLAG_HRESULT, not stdcall.
Drop the calling-convention check rather than repair the bit arithmetic.
ctypes only honors stdcall when building a callback on 32-bit x86 Windows,
which cuda.core does not support, and FUNCFLAG_PYTHONAPI is never consulted
on the callback path, so CFUNCTYPE, WINFUNCTYPE, and PYFUNCTYPE all yield the
same FFI_DEFAULT_ABI thunk. That leaves the declared result and argument
types, which are reachable through the public restype/argtypes attributes.
Reading those public attributes also lets a function pointer taken from a
shared library be accepted once its restype and argtypes are declared, which
the class-level lookup could never see.
…-host-callback
# Conflicts:
#	cuda_core/docs/source/release/1.2.0-notes.rst
@Andy-Jost
Andy-Jost merged commit 0e6d282 into NVIDIA:mainAug 7, 2026
207 of 209 checks passed
@Andy-Jost
Andy-Jost deleted the ajost/validate-ctypes-host-callback branch August 7, 2026 20:38
@github-actions

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugSomething isn't workingcuda.coreEverything related to the cuda.core moduleP1Medium priority - Should do

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cuda.core: validate ctypes host callback signatures

2 participants

@Andy-Jost@juenglin