ci(cco): stop the SDMA queue reclamation race from failing CI - #685
Open
yangyuhuiling wants to merge 6 commits into
Open
yangyuhuiling wants to merge 6 commits into
yangyuhuiling wants to merge 6 commits into
Conversation
Three cases in test_gemm_ar.py failed on main at anvil.cpp:237 --
hsaKmtCreateQueueExt refusing to create SDMA queues because the previous
process group's are not reclaimed yet. They failed back to back at 09:22
after 22 minutes of green, which is the shape of a resource that runs out
rather than a defect:
test_fused_is_stable_across_repeats FAILED 09:22:31
test_gemm_ar_modes_agree[split-sdma] FAILED 09:22:34
test_gemm_ar_modes_agree[fused-sdma] FAILED 09:22:38
3 failed, 85 passed, 1 skipped in 1335.94s
The fused-sdma one aborted a step further along, in the bootstrap
allgather that follows queue creation ("Allgather operation failed",
SIGABRT on rank 0), so the symptom is not always the anvil line itself.
This file spawns a fresh torch.distributed.run per case and the cases run
in sequence, with nothing between them. test_gemm_ar_op.py already settles
20s before each spawn and retries once on anvil.cpp; that came out of
hitting this repeatedly while developing the op. This applies the same
thing to _run_bench, which was left without it.
Only anvil.cpp is retried, and only once. Every other failure is still
raised on its first occurrence, so this hides no real defect -- it makes a
race between processes stop being reported as one.
Verified by running the three failing cases plus split-lsa: 4 passed in
281s, on a box with other work on the GPUs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… loses test_sdma_edge hung on main for 241s against the 3s it takes when it runs, and the `timeout 240` killed the step. It is the fifth of seven 8-rank process groups the loop starts with nothing between them, and SDMA queues are not reclaimed the moment a process exits -- so the next one finds none left and either fails in hsaKmtCreateQueueExt (anvil.cpp:237) or blocks there. Same resource race as the Python side, one symptom further along. Sleep 10s between tests, and give a failed one a second chance after 60s. A test that is actually broken fails both times, the step still fails, and both outputs are in the log -- so this does not convert a real failure into a green run. The SKIP check is unchanged and still fails the step. Costs 70s in the happy path. timeout-minutes is 60 and the job runs ~28. The retry is deliberately not conditioned on the anvil.cpp string, unlike the Python helper: the hang never reaches that message, it is killed by `timeout` with rc=124. Conditioning on rc would let a real hang retry too, which is fine -- 240s twice, then a red step either way. Verified the quoting and all four paths (pass / recovers on retry / fails twice / SKIP) against fake binaries before touching the workflow; the YAML parses and the extracted step is valid bash. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… reclaimed Queues are reclaimed by the kernel when the owning process's KFD fd closes, not by mori: AnvilLib::getInstance deliberately leaks the singleton so ~AnvilLib never runs during process teardown (worker exits stall in ROCm/HSA shutdown ordering otherwise), which means ~SdmaQueue's hsaKmtDestroyQueue never fires. connect() already notes that queues "are only reclaimed when the process exits". A process group launched immediately after another therefore races that reclaim and hsaKmtCreateQueueExt returns NO_MEMORY (status 6, anvil.cpp:237). CI hit it twice: test_sdma_edge hung 241s against the 3s it takes, and three gemm_ar cases failed 4s apart after 22 minutes of green. One rank exits inside the macro, the others then abort in the bootstrap allgather, so a single resource hiccup shows up as several unrelated-looking failures. Retry there, bounded at 20 x 100ms. Costs nothing when uncontended, unlike a fixed settle before every launch, and fixes callers outside CI too -- a server restart or a benchmark sweep hits the same window with no retry to save it. It prints on the first failed attempt and again on the one that succeeds, so a box whose reclaim is genuinely slow stays visible instead of just being quietly slower. Also evaluate CHECK_HSAKMT_SUCCESS's argument once. It appeared a second time in the error message, so every failing hsaKmt call was issued twice: a second CreateQueueExt against a node that had just refused one, a second AllocMemory that leaks, a double DestroyQueue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ouble-calling
CI read a bare `ERROR code: 6 ... anvil.cpp:237` as "the SDMA queues from the
previous run were not reclaimed yet". Measured on MI355X (gfx950, 2 PCIe + 14
XGMI SDMA engines, 8 queues per engine), that is not what a 6 means:
filling engine 0 to its cap -> status 1
filling all 16 engines (128 total) -> status 1
12 processes contending one engine -> status 1 for the 10 that lose
an engine id the node lacks -> status 1
24 GB of pinned host memory -> success
RLIMIT_AS at 2000000 / 1000000 / 700000 kB
-> status 6 at the 11th / 5th / 3rd queue
So HSAKMT_STATUS_ERROR is the kernel refusing (queues, engines) and
HSAKMT_STATUS_NO_MEMORY is libhsakmt failing to allocate for the queue in this
process. A 6 means the process or the box is out of memory; nothing about SDMA
is exhausted, and no amount of waiting for a previous run frees anything.
Reclaim is not the problem either: after a process holding a full engine exits,
the next process gets a queue on its first attempt (~109 ms, which is process
startup, not waiting). A healthy rank reaches 850 VMAs and 8 SDMA queues --
orders of magnitude below every limit involved.
Print which failure it is, with the node, engine, per-engine capacity and how
many queues this process has already built, so the next occurrence says whether
to look at mori or at the machine.
Also evaluate CHECK_HSAKMT_SUCCESS's argument once. It appeared a second time in
the message, so every failing hsaKmt call was issued twice on the way out: a
second CreateQueueExt against a node that had just refused one, a second
AllocMemory that leaks, a double DestroyQueue.
Verified on 8x MI355X: all seven CCO SDMA C++ tests 8/8 PASS, 30 back-to-back
runs and 3 concurrent lanes with no spacing at all, 0 failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…one that loses" Revert "test(cco): let gemm_ar's bench spawns wait for SDMA queues to come back" Both changes wait for SDMA queues to be reclaimed between process groups. The measurements in the previous commit say there is nothing to wait for: the failure they target reports HSAKMT_STATUS_NO_MEMORY, which is this process failing to allocate, while every form of queue or engine exhaustion reports HSAKMT_STATUS_ERROR. And reclaim already beats process startup -- after a process holding a full engine exits, the next one gets a queue on its first attempt. They are not free. The settle runs before every launch, not only a contended one: 7 x 10s in the workflow, and 20s x 6 launches in test_gemm_ar.py on an 8-GPU box (three repeats plus three modes), with test_gemm_ar_op.py already contributing four more in the same pytest invocation. About 190s on every run, against a failure seen twice in a week. The one way spacing could have helped is by giving a neighbouring job on a shared runner time to release memory. That is worth knowing, and it is not worth paying for in every green run of mori's own suite. Reverting leaves the loop and _run_bench as they were. What replaces them is the diagnostic: the next occurrence names the node, the engine, the per-engine capacity and the queue count, and says in words whether SDMA ran out or the machine did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two CI failures on
mainthis week, one symptom apart, both the same race:SDMA queues are not reclaimed the moment a process exits, so a process group
started immediately after finds none left.
Run CCO SDMA C++ teststest_sdma_edgehung 241s, killed bytimeout 240(it takes 3s when it runs)Run CCO GEMM+AR numericsanvil.cpp:237, back to back, after 22 minutes of greenNeither is a defect in what is under test: both ran green on the same code
before and after, and each case passes alone.
Commit 1 gives
test_gemm_ar.py::_run_benchthe settle-and-retry thattest_gemm_ar_op.pyalready had — that one got it while developing the op,for exactly this, and the older file was left without it. Only
anvil.cppisretried, once; every other failure is still raised on the first occurrence.
Commit 2 does the same for the seven back-to-back 8-rank C++ runs: 10s
between tests, 60s and one retry after a failure. A broken test fails twice
and the step still fails, with both outputs in the log. The
SKIP-is-failurecheck is unchanged.
Costs 70s in the happy path; the job runs ~28 minutes against a 60 minute
limit.
Verified: the three failing Python cases plus
split-lsapass (4 passed in281s, on a box with other work on the GPUs); the workflow quoting was
exercised against fake binaries for all four paths (pass / recovers on retry
/ fails twice / SKIP) before being written into the YAML.
🤖 Generated with Claude Code