Skip to content

Add multi-GPU CI runners (t4 and h100 with 2 GPUs) - #1505

Merged
leofang merged 5 commits into
NVIDIA:mainfrom
Andy-Jost:multi-gpu-ci
Jan 16, 2026
Merged

Add multi-GPU CI runners (t4 and h100 with 2 GPUs)#1505
leofang merged 5 commits into
NVIDIA:mainfrom
Andy-Jost:multi-gpu-ci

Conversation

@Andy-Jost

@Andy-JostAndy-Jost commented Jan 15, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Add support for multi-GPU CI testing by introducing GPU_COUNT field to the test matrix and adding t4 and h100 2-GPU configurations.

Changes

  • ci/test-matrix.yml:

    • Added GPU_COUNT field to all entries for consistency
    • Added two new multi-GPU entries: t4 and h100 with GPU_COUNT: '2'
    • Removed special_runners section - entries now integrated directly into pull-request matrix
    • Aligned columns for readability (can be reverted if needed)

  • .github/workflows/test-wheel-linux.yml:

    • Updated runs-on to use ${{ matrix.GPU_COUNT }} instead of hardcoded -1
    • Updated job name to show (x2) suffix for multi-GPU tests (e.g., t4(x2))
    • Removed special_runners handling logic (no longer needed)

  • .github/workflows/test-wheel-windows.yml:

    • Updated runs-on to use ${{ matrix.GPU_COUNT }} for consistency

  • cuda_core/examples/simple_multi_gpu_example.py:

    • Switched from old CuPy RNG (cp.random.random()) to new RNG (cp.random.default_rng()) to avoid requiring libcurand.so

  • cuda_core/tests/test_launcher.py:

    • Switched to new CuPy RNG to avoid libcurand dependency

Test Coverage

  • Multi-GPU runners are now included in the standard PR test matrix
  • Job names clearly indicate GPU count: py3.13, 13.1.0, local, t4(x2)

Closes#1501

@Andy-JostAndy-Jost self-assigned this Jan 15, 2026
@copy-pr-bot

copy-pr-botBot commented Jan 15, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Andy-Jost

Andy-Jost commented Jan 15, 2026

Copy link
Copy Markdown
ContributorAuthor

/ok to test f97ef265bc7f69e978f70ba1241806d9f3ede742

@github-actions

This comment has been minimized.

Sign in to view
@Andy-Jost

Andy-Jost commented Jan 15, 2026

Copy link
Copy Markdown
ContributorAuthor

/ok to test 916380468

leofang
leofang reviewed Jan 15, 2026
View reviewed changes
Comment thread.github/workflows/test-wheel-linux.yml Outdated
@Andy-Jost
Add multi-GPU CI runners (t4 and h100 with 2 GPUs)
f19d167
Add GPU_COUNT field to test matrix to support multi-GPU configurations.
This enables rigorous testing of peer access, device switching, and
other multi-GPU functionality in CI.
ClosesNVIDIA#1501
@Andy-Jost
Remove special_runners, integrate entries into pull-request matrix
cc13ecf
Simplifies CI configuration by moving special runner entries directly
into the pull-request matrix rather than handling them separately.
@Andy-Jost
Fix test to avoid cuRAND dependency
074a6ca
Replace cp.random.default_rng() with cp.arange() to avoid requiring
cuRAND library which may not match the installed CUDA version.
@Andy-Jost

Andy-Jost commented Jan 15, 2026

Copy link
Copy Markdown
ContributorAuthor

/ok to test 074a6ca

@Andy-Jost

Andy-Jost commented Jan 15, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Notes on test-matrix.yml changes:

  • Added GPU_COUNT field to all entries for consistency and to support multi-GPU runners (t4 and h100 with 2 GPUs).
  • Aligned columns for readability. This is optional and can be reverted if anyone objects.

@Andy-Jost

Andy-Jost commented Jan 15, 2026

Copy link
Copy Markdown
ContributorAuthor

/ok to test 3184316

@Andy-Jost
Use CuPy's new RNG to avoid libcurand dependency
4214c2c
Switch from cp.random.random() (old RNG requiring libcurand.so) to
cp.random.default_rng().random() (new RNG with pre-compiled curand
device libs bundled in CuPy).
@Andy-Jost

Andy-Jost commented Jan 15, 2026

Copy link
Copy Markdown
ContributorAuthor

/ok to test 4214c2c

@Andy-JostAndy-Jost changed the title [WIP] Test multi-GPU CI runnersAdd multi-GPU CI runners (t4 and h100 with 2 GPUs)Jan 15, 2026
@Andy-Jost
Andy-Jost marked this pull request as ready for review January 15, 2026 23:50
@kkraus14

kkraus14 commented Jan 16, 2026

Copy link
Copy Markdown
Collaborator

Do we want to run all of our CI jobs on these multi-GPU runners or can we only run multi-GPU specific tests / examples / etc. as needed?

leofang
leofang approved these changes Jan 16, 2026
View reviewed changes

@leofangleofang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to run all of our CI jobs on these multi-GPU runners or can we only run multi-GPU specific tests / examples / etc. as needed?

I suggested offline that we add 2 dual-GPU jobs on the per-PR basis for now, and monitor the usage in the next few days. We don't have the infra for as-needed tests yet (#299 is a good start).

Comment threadci/test-matrix.yml
Comment threadcuda_core/examples/simple_multi_gpu_example.py
@leofangleofang added this to the cuda.core beta 12 milestone Jan 16, 2026
@leofangleofang added P0 High priority - Must do! CI/CD CI/CD infrastructure enhancement Any code-related improvements labels Jan 16, 2026
@mdboom

mdboom commented Jan 16, 2026

Copy link
Copy Markdown
Contributor

🎉 There are a few APIs about to land in cuda.core.system related to multiple GPUs that could benefit from this testing.

@leofang
leofang merged commit 53c8d4a into NVIDIA:mainJan 16, 2026
88 checks passed
@github-actions

github-actionsBot commented Jan 16, 2026

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

@leofangleofang mentioned this pull request Jan 22, 2026
Merged
2 tasks
@Andy-Jost
Andy-Jost deleted the multi-gpu-ci branch January 23, 2026 15:48
@leofangleofang mentioned this pull request Feb 20, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leofangleofangleofang approved these changes

Assignees

@Andy-JostAndy-Jost

Labels

CI/CDCI/CD infrastructureenhancementAny code-related improvementsP0High priority - Must do!

Projects

None yet

Milestone

cuda.core v0.6.0

Development

Successfully merging this pull request may close these issues.

Enable multi-GPU CI testing

4 participants

@Andy-Jost@kkraus14@mdboom@leofang