Skip to content

CI: retry arm64 GH200 nightly-standard row + add GB300 - #2296

Merged
leofang merged 10 commits into
NVIDIA:mainfrom
leofang:leofang/retry-gh200-nightly
Jul 14, 2026
Merged

CI: retry arm64 GH200 nightly-standard row + add GB300#2296
leofang merged 10 commits into
NVIDIA:mainfrom
leofang:leofang/retry-gh200-nightly

Conversation

@leofang

@leofangleofang commented Jul 2, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Re-enable the gh200nightly-standard row in ci/test-matrix.yml (previously disabled in 4c70cfa because the runner hung on cudaMallocAsync). Runner team indicated the pool-side issue has been fixed, so let's give it another CI-visible run.

Also added access to gb300, which exposes a minor issue in the way we generate object/library files for testing.

Also fixed a few nightly CI issues due to the recent numba-cuda-mlir v0.4.1 release

leofang added 2 commits July 2, 2026 15:01
@leofang
ci/test-matrix.yml: re-enable arm64 gh200 nightly-standard row
4568e9c
Reverts the disable in 4c70cfa now that the runner team has fixed the
pool-side hang on stream-ordered memory allocator calls.
@leofang
Temporarily add push trigger to ci-nightly.yml for testing
8d51cf7
Remove before merging.
@copy-pr-bot

copy-pr-botBot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@leofang

leofang commented Jul 2, 2026

Copy link
Copy Markdown
MemberAuthor

/ok to test 8d51cf7

@github-actionsgithub-actionsBot added the CI/CD CI/CD infrastructure label Jul 2, 2026
@github-actions

github-actionsBot commented Jul 2, 2026

Copy link
Copy Markdown
Doc Preview CI

🚀 View preview at
https://nvidia.github.io/cuda-python/pr-preview/pr-2296/

https://nvidia.github.io/cuda-python/pr-preview/pr-2296/cuda-core/

https://nvidia.github.io/cuda-python/pr-preview/pr-2296/cuda-bindings/

https://nvidia.github.io/cuda-python/pr-preview/pr-2296/cuda-pathfinder/


Preview will be ready when the GitHub Pages deployment is complete.

@kkraus14kkraus14 mentioned this pull request Jul 2, 2026
Closed
@leofang
CI: deselect test_get_bar_size_in_kb on gh200 runners
f148214
The cufile BAR-size query returns CUDA_ERROR_NOT_SUPPORTED on
Grace+Hopper (unified memory, no discrete PCIe BAR). Tracked in NVIDIA#2299;
remove this deselect once the test skipif is fixed upstream.
Uses PYTEST_ADDOPTS so no changes to run-tests are needed.
@leofang

leofang commented Jul 2, 2026

Copy link
Copy Markdown
MemberAuthor

/ok to test f148214

leofang
leofang commented Jul 13, 2026
View reviewed changes
Comment threadci/test-matrix.yml Outdated
@leofangleofang changed the title CI: retry arm64 gh200 nightly-standard rowCI: retry arm64 GH200 nightly-standard row + add GB300Jul 13, 2026
@leofang

leofang commented Jul 13, 2026

Copy link
Copy Markdown
MemberAuthor

/ok to test 5bbfa09

@leofangleofang added enhancement Any code-related improvements P1 Medium priority - Should do labels Jul 13, 2026
@leofang
cuda.core tests: build saxpy RDC fixture with -arch=all (covers sm_10…
9362c3a
…3 / GB300)
-arch=all-major only emits X.0 cubins (sm_100, sm_90, ...), so nvJitLink returns ERROR_INVALID_INPUT when the RDC saxpy fixture is linked on GB300 (sm_103): Blackwell no longer serves an sm_10x cubin from the sm_100 image. -arch=all additionally emits sm_103 (and other minor archs) so the linker finds a usable input. The fixture is built in-job by build_test_binaries.py, so there is no committed artifact to regenerate.
@github-actionsgithub-actionsBot added the cuda.core Everything related to the cuda.core module label Jul 13, 2026
@leofang
CI: install curand/cublas + cupy for numba-cuda-mlir nightly; extend …
6a89eaa
…deselect gate to 0.4.1
test_fortran_contiguous calls cp.random, which dlopens libcurand at runtime; the mlir nightly install didn't provide it, unlike the numba-cuda branch (which installs cuda-toolkit[curand,cublas] + cupy). Mirror that here.
Bump the version-gated mlir deselect from <=0.4.0 to <=0.4.1: 0.4.1 still fails the cuobjdump-invoking nvjitlink test (wheel envs lack cuobjdump) and the NVIDIA#135 pytest-contamination tests, so keep them deselected. cu12 rows remain red due to the separate NVIDIA#2320 LTO-IR regression (not in the deselect list).
@leofang
nit: align columns
7f44b44
@leofang

leofang commented Jul 13, 2026

Copy link
Copy Markdown
MemberAuthor

/ok to test 7f44b44

leofang
leofang commented Jul 13, 2026
View reviewed changes
Comment threadcuda_core/tests/test_binaries/build_test_binaries.py
@leofangleofang self-assigned this Jul 13, 2026
@leofangleofang added bug Something isn't working P0 High priority - Must do! and removed P1 Medium priority - Should do labels Jul 13, 2026
@leofang

leofang commented Jul 13, 2026

Copy link
Copy Markdown
MemberAuthor

/ok to test 4b61b3c

leofang
leofang commented Jul 13, 2026
View reviewed changes
Comment threadci/test-matrix.yml
# allocator (cudaMallocAsync); runner pool needs fixing first.
# - { ARCH: 'arm64', PY_VER: '3.14', CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'gh200', GPU_COUNT: '1', DRIVER: 'latest', ENV: { MODE: 'nightly-standard' } }
- { ARCH: 'arm64', PY_VER: '3.13', CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'gh200', GPU_COUNT: '1', DRIVER: 'latest', ENV: { MODE: 'nightly-standard' } }
- { ARCH: 'arm64', PY_VER: '3.14', CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'gb300', GPU_COUNT: '1', DRIVER: 'latest', ENV: { MODE: 'nightly-standard' } }

@leofangleofangJul 13, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two lines are real, non-cosmetic changes (gh200/gb300).

@leofang
leofang marked this pull request as ready for review July 13, 2026 23:06
lijinf2
lijinf2 approved these changes Jul 14, 2026
View reviewed changes

@lijinf2lijinf2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for the fix.

Comment thread.github/workflows/test-wheel-linux.yml
Comment threadci/tools/run-tests
@leofang

leofang commented Jul 14, 2026

Copy link
Copy Markdown
MemberAuthor

/ok to test a31c2cd

@leofang
leofang enabled auto-merge (squash) July 14, 2026 01:51
@leofang
leofang merged commit 046cefa into NVIDIA:mainJul 14, 2026
103 checks passed
github-actionsBot pushed a commit that referenced this pull request Jul 14, 2026
cuda-python-bot
Clean up PR preview folders for 2 closed/merged PRs
69f7bcb
Removed preview folders for the following PRs:
- PR #2296
- PR #2312
@lijinf2

lijinf2 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Thank you Leo!

@leofang
leofang deleted the leofang/retry-gh200-nightly branch July 16, 2026 02:31
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lijinf2lijinf2lijinf2 approved these changes

Assignees

@leofangleofang

Labels

bugSomething isn't workingCI/CDCI/CD infrastructurecuda.coreEverything related to the cuda.core moduleenhancementAny code-related improvementsP0High priority - Must do!

Projects

None yet

Milestone

cuda.bindings 13.4.0 & 12.9.8

Development

Successfully merging this pull request may close these issues.

2 participants

@leofang@lijinf2