Add CUDA version compatibility check by Andy-Jost · Pull Request #1412 · NVIDIA/cuda-python

Andy-Jost · 2025-12-18T19:06:15Z

Summary

Adds a version compatibility check that warns users when cuda-bindings was compiled against a newer CUDA major version than the installed driver supports.

Changes

cuda-bindings

Added check_cuda_version_compatibility() function in cuda/bindings/utils/_version_check.py
Compares compile-time CUDA_VERSION vs runtime cuDriverGetVersion()
Exported from cuda.bindings.utils
Added comprehensive unit tests in tests/test_version_check.py

cuda-core

Device.__new__ calls check_cuda_version_compatibility() after cuInit succeeds
Imports the function directly from cuda.bindings.utils

Rationale

When cuda-bindings is built against CUDA 13 headers but the user's driver only supports CUDA 12, many features will silently fail or behave unexpectedly. This check provides early, clear feedback:

UserWarning: cuda-bindings was built against CUDA 13.0, but the installed driver only supports CUDA 12.8. Some features may not work correctly. Consider updating your NVIDIA driver. Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to suppress this warning.

Design

Provided by cuda-bindings: The version check implementation lives in cuda.bindings.utils since it checks cuda-bindings' compile-time version
Invoked by cuda-core: Called when Device first triggers CUDA initialization
Runs once: Uses a module-level flag to ensure the check runs only once per process
Non-blocking: Warning only, does not prevent operation
Suppressible: Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to disable
Major version only: No warning for minor version differences (handled by graceful degradation per PR Handle unsupported device attributes gracefully #1409)

Future Work

We could not find a suitable place to invoke the version check automatically within cuda-bindings itself (e.g., hooking into cuInit), so the check is currently triggered by cuda-core. This may be revisited in the future.

Test Coverage

7 tests in cuda-bindings covering:

No warning when driver is newer
No warning when same major version
Warning when compile major > driver major
Warning only issued once per process
Suppression via environment variable
Silent handling of driver errors
Silent handling of missing CUDA_VERSION attribute

Related Work

Complements PR Handle unsupported device attributes gracefully #1409 which handles graceful degradation for minor version mismatches

copy-pr-bot · 2025-12-18T19:06:18Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Andy-Jost · 2025-12-18T19:06:28Z

/ok to test 7ce325c07b559e9a66472f5ba7a99fe980e17c09

Andy-Jost · 2025-12-18T19:08:17Z

/ok to test 1962e351a233d65f2288fd6d16a43f0b0b75d6ad

leofang · 2025-12-18T20:28:01Z

@@ -9,6 +9,8 @@ Runtime Environment Variables

 - ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

+- ``CUDA_PYTHON_DISABLE_VERSION_CHECK`` : When set to 1, suppresses the warning that is issued when ``cuda.core`` detects that ``cuda-bindings`` was compiled against a newer CUDA major version than the installed driver supports. This warning helps identify version mismatches that may cause features to not work correctly.


This env var is added to cuda-bindings docs, but the actual change is made to cuda-core? Can we move them to the right place?
Actually, thinking about this we probably should do this check in cuda-bindings instead of cuda-core, since it's what actually matters. The question is if there is a good place that would allow lazy-init. Remember, both cuda-core and cuda-bindings can be installed and imported on CPU-only machines, without driver or a physical GPU installed. I can't think of an obvious solution off top of my head.

What about cuInit? That seems like a logical place for the version check, though the change might be more complex.

As discussed, cuda-bindings provides the check; cuda-core invokes it. Other consumers of cuda-bindings can also invoke the check.

Andy-Jost · 2025-12-19T00:10:25Z

/ok to test 73e5a4379fad3392c24499e245a521c86c53db42

Andy-Jost · 2025-12-19T00:48:39Z

/ok to test ddd92bd

rparolin

We should issue a warning to the user if we can't fetch the driver version.

Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

# Conflicts: # cuda_core/pixi.lock

Andy-Jost · 2026-01-09T21:30:52Z

/ok to test acadb31

Integrate RAII handles architecture from main while preserving the version compatibility check feature. Resolved conflict in _device.pyx by keeping the refactored helper functions and updating field names to match RAII object structure.

Andy-Jost · 2026-01-13T19:42:51Z

/ok to test 3a5c210

Andy-Jost · 2026-01-14T01:12:48Z

/ok to test 424a113

Andy-Jost · 2026-01-14T01:42:35Z

/ok to test 6071609

leofang

Sorry for my late review.

leofang · 2026-01-14T03:20:20Z


 - ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

+- ``CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING`` : When set to 1, suppresses warnings about CUDA major version mismatches between ``cuda-bindings`` and the installed driver.


TBH I am not sure if this env var is really useful, because the added API is entirely opt-in?

The version check can be triggered indirectly from the end user's POV. For example, by using cuda-core. Without this, each library that calls the version-check function would need its own opt-out, and users might need to set them all. To me it makes sense to place the opt-out alongside the implementation and there is little/no cost.

I insist that this is useless because this is listed in the cuda-bindings docs, not cuda-core. But cuda-bindings does not call this function anywhere, whereas cuda-core users don't necessarily look at cuda-bindings docs. Can we please create a new environment_variables.rst for cuda-core if we think this is helpful there?

xref: #1412 (comment)

Even better: Document this env var in both docs.

Good point. The same argument applies to all cuda.bindings runtime variables, so I:
duplicated both CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM and CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING in the cuda.core documentation; and
added a note referencing the cuda.bindings runtime variables and linking to the relevant documentation page.

Andy-Jost · 2026-01-14T17:49:45Z

/ok to test 96532b226

Andy-Jost · 2026-01-14T17:51:15Z

/ok to test 973af9b7a

Replace setup_method/teardown_method with a pytest fixture that uses monkeypatch to properly save and restore the original value of _major_version_compatibility_checked after each test. Minor change to Cython cdef inline helper function signature.

Andy-Jost · 2026-01-14T18:03:55Z

/ok to test 7f23b08

Document runtime environment variables that affect cuda.core behavior: - CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM - CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING Include note linking to cuda-bindings environment variables documentation.

Andy-Jost · 2026-01-23T19:26:26Z

/ok to test a368e0f

github-actions · 2026-01-23T23:29:45Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

Andy-Jost added this to the cuda.core beta 11 milestone Dec 18, 2025

Andy-Jost added P0 High priority - Must do! feature New feature or request cuda.core Everything related to the cuda.core module labels Dec 18, 2025

Andy-Jost self-assigned this Dec 18, 2025

Andy-Jost requested review from leofang, rparolin and rwgk December 18, 2025 19:06

Andy-Jost force-pushed the runtime-version-check branch from 7ce325c to 1962e35Compare December 18, 2025 19:08

mdboom approved these changes Dec 18, 2025
View reviewed changes

This comment has been minimized.
Sign in to view

Andy-Jost mentioned this pull request Dec 18, 2025
fix: derive CUDA major version from headers for build #1395
Merged
2 tasks

leofang requested changes Dec 18, 2025
View reviewed changes

Comment threadcuda_core/cuda/core/_utils/cuda_utils.pyx Outdated

leofang reviewed Dec 18, 2025
View reviewed changes

Comment threadcuda_core/cuda/core/_utils/cuda_utils.pyx Outdated

Andy-Jost force-pushed the runtime-version-check branch 2 times, most recently from 8a0d248 to 73e5a43Compare December 19, 2025 00:09

Andy-Jost force-pushed the runtime-version-check branch from 73e5a43 to ddd92bdCompare December 19, 2025 00:47

leofang self-requested a review December 19, 2025 01:22

rparolin reviewed Dec 19, 2025
View reviewed changes

Comment threadcuda_bindings/cuda/bindings/utils/_version_check.py Outdated

rparolin reviewed Dec 19, 2025
View reviewed changes

Comment threadcuda_bindings/cuda/bindings/utils/_version_check.py Outdated

rparolin requested changes Dec 20, 2025
View reviewed changes

rwgk reviewed Dec 26, 2025
View reviewed changes

Comment threadcuda_bindings/cuda/bindings/utils/_version_check.py Outdated
Comment threadcuda_bindings/cuda/bindings/utils/_version_check.py Outdated
Comment threadcuda_bindings/cuda/bindings/utils/_version_check.py Outdated
Comment threadcuda_bindings/cuda/bindings/utils/_version_check.py Outdated

Andy-Jost added 2 commits January 7, 2026 13:09

Add CUDA major version compatibility check
d999f40
Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

Merge remote-tracking branch 'origin/main' into runtime-version-check
62dfcca
# Conflicts: # cuda_core/pixi.lock

rparolin reviewed Jan 9, 2026
View reviewed changes

Comment threadcuda_bindings/tests/test_version_check.py Outdated

rparolin approved these changes Jan 9, 2026
View reviewed changes

kkraus14 approved these changes Jan 13, 2026
View reviewed changes

Merge branch 'main' into runtime-version-check
3a5c210
Integrate RAII handles architecture from main while preserving the version compatibility check feature. Resolved conflict in _device.pyx by keeping the refactored helper functions and updating field names to match RAII object structure.

Andy-Jost force-pushed the runtime-version-check branch from acadb31 to 3a5c210Compare January 13, 2026 19:42

Merge branch 'main' into runtime-version-check
424a113

Merge branch 'main' into runtime-version-check
6071609

rwgk approved these changes Jan 14, 2026
View reviewed changes

leofang requested changes Jan 14, 2026
View reviewed changes

Merge branch 'main' into runtime-version-check
7925693

Andy-Jost force-pushed the runtime-version-check branch from 96532b2 to 973af9bCompare January 14, 2026 17:51

Andy-Jost force-pushed the runtime-version-check branch from 973af9b to 27732f2Compare January 14, 2026 17:56

Andy-Jost force-pushed the runtime-version-check branch from 27732f2 to 7f23b08Compare January 14, 2026 18:03

rwgk approved these changes Jan 15, 2026
View reviewed changes

Comment threadcuda_core/cuda/core/_device.pyx

Andy-Jost requested a review from leofang January 22, 2026 19:47

Andy-Jost added 2 commits January 23, 2026 11:25

Add environment variables documentation for cuda.core
4f20ae0
Document runtime environment variables that affect cuda.core behavior: - CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM - CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING Include note linking to cuda-bindings environment variables documentation.

Merge origin/main into runtime-version-check
a368e0f

leofang approved these changes Jan 23, 2026
View reviewed changes

leofang merged commit 8d8f29b into NVIDIA:mainJan 23, 2026
160 of 161 checks passed

Andy-Jost deleted the runtime-version-check branch January 23, 2026 23:21

		@@ -9,6 +9,8 @@ Runtime Environment Variables

		- ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

		- ``CUDA_PYTHON_DISABLE_VERSION_CHECK`` : When set to 1, suppresses the warning that is issued when ``cuda.core`` detects that ``cuda-bindings`` was compiled against a newer CUDA major version than the installed driver supports. This warning helps identify version mismatches that may cause features to not work correctly.


		- ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

		- ``CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING`` : When set to 1, suppresses warnings about CUDA major version mismatches between ``cuda-bindings`` and the installed driver.

Uh oh!

Conversation

Andy-Jost commented Dec 18, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

cuda-bindings

cuda-core

Rationale

Design

Future Work

Test Coverage

Related Work

Uh oh!

copy-pr-botBot commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

This comment has been minimized.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-JostDec 18, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Uh oh!

Uh oh!

rparolin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost commented Jan 9, 2026

Uh oh!

Andy-Jost commented Jan 13, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

leofang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

Uh oh!

Andy-Jost commented Jan 23, 2026

Andy-Jost commented Dec 18, 2025•
edited
Loading

Andy-JostDec 18, 2025•
edited
Loading