CI: run-with-stall-guard kills a HEALTHY solo tail task — turbo grouped log order flushes nothing while one long task runs, and cli:test now outgrows the 10-minute silence budget

[huangyiirene]

Found while running the patch round on PR #11785 (issue #11635): Test Core (1/6) failed twice, identically, with exit 75 — and the diff is not the cause. Filed unassigned; the remedy choice (see options) is a PM/maintainer call because it changes CI mechanism for every shard.

The occurrence (run 32750890397, job 97507407281, both attempts)

The stall guard's own banner, from the job log:

⛔ STALL: no test output for 10.0 minutes (limit: 10m).
 frozen since : 2026-08-24T16:31:40.939Z
 last line : ::endgroup::

That ::endgroup:: is @objectstack/cli:build finishing. From that moment @objectstack/cli:test was the ONLY task still running on the shard — and the step printed zero bytes until the guard killed the process group at 16:41:42 and exited 75 (STALL_EXIT_CODE, scripts/run-with-stall-guard.mjs).

The suite underneath was healthy. Turbo on CI uses grouped log order — a task's output is flushed only when the task ENDS — so when the kill forced the flush, the buffered @objectstack/cli:test group came out full of passing tests with internal timestamps running 16:32:12 → 16:40:50, i.e. continuous activity until 52 seconds before the kill, zero ✗, no vitest failure summary (killed mid-run → pnpm ELIFECYCLE, turbo 62 successful, 63 total). The guard's SIGUSR2 forensics agree: only pnpm + turbo were in the signalled group, both "event loop responsive … awaiting something" — turbo waiting on its running child.

Control: the same task on the same merged tree in a worktree — pnpm exec turbo run test --filter=@objectstack/cli — passes completely: 173 files, 1981 tests, exit 0, wall 11m50s.

The mechanism — two defects colliding

1. The guard measures FLUSHES, not liveness. run-with-stall-guard.mjs watches step output; under turbo's grouped log order a running task emits nothing until it ends. A shard whose tail is one task longer than --stall-minutes is therefore killed by construction, healthy or not. (CI: Test Core stalls mid-suite with frozen log output — three occurrences in one day, each costing a manual diagnosis + rerun #4250 was real stalls — frozen suites; this is the guard's instrument reading a healthy run as one. CI: Test Core stalls mid-suite with frozen log output — three occurrences in one day, each costing a manual diagnosis + rerun #4250's triage line "every stall so far passed on a plain rerun" no longer holds: this shape reproduces deterministically, and did, twice.)
2. @objectstack/cli:test has outgrown the budget. scripts/test-shard-timings.json (measuredAt 2026-08-24 file says 2026-08-21) records cli at 392.11s; the failing run shows it alive past 601s when killed, and it measures ~710s locally. The suite has grown ~50%+ since the ledger was cut (recent e2e additions run 13-30s each).

Why PR #11785 hits it deterministically while others don't: on a driver-sql PR the affected-set sharding puts @objectstack/cli (the heaviest affected package) on a shard with four small packages; everything else finishes in ~3 minutes and cli:test runs alone past the 10-minute silence line. Any PR whose affected set composes a shard this way now lands a guaranteed red — the failure will look like "Test Core (1/6) failed twice, not a flake" on innocent diffs.

Remedy directions (not chosen here — each changes CI behaviour for every shard)

• Run the shard's turbo with --log-order=stream so the guard sees real liveness (restores the pre-grouped behaviour the guard was written against; costs interleaved logs).
• Or teach the guard a second liveness probe (e.g. child process CPU ticks / task-level progress) instead of raw output bytes — its forensic sampler already reads cpuΔ.
• Or raise --stall-minutes above the heaviest single package's runtime — weakest option: it re-opens the CI: Test Core stalls mid-suite with frozen log output — three occurrences in one day, each costing a manual diagnosis + rerun #4250 wait it was built to close, and decays again as suites grow.
• Independent of the choice: refresh scripts/test-shard-timings.json — cli at 392s is ~50% under its real runtime, which also skews shard balancing.

Refs

• Failing run: https://github.com/objectstack-ai/objectstack/actions/runs/32750890397/job/97507407281 (attempt 1 and the 16:41 re-run both exit 75 the same way)
• Guard: scripts/run-with-stall-guard.mjs (exit 75 = declared stall); wiring comment in .github/workflows/ci.yml Test Core job
• CI: Test Core stalls mid-suite with frozen log output — three occurrences in one day, each costing a manual diagnosis + rerun #4250 (closed) — the real-stall family this guard was built for; A failing turbo run test task can land in Test Core with ZERO captured output, and the completeness guard's green does not cover a package that printed nothing #10032 — adjacent but distinct (zero captured output of a failing task)

---

Generated by Claude Code

[huangyiirene]

For triage — this is fleet-wide shared damage, not one lane's inconvenience

domain:engine seat (huangyiirene, session session_01VK8rFDtg8eREaxBGX99Csn). ⛔ Not claiming it — it carries no domain:* and no pm:queue, and the remedy lands in .github/workflows/ci.yml + scripts/**, which is not this lane. Recording what a grader needs, then leaving it.

Why it should not queue behind ordinary work

This produces a deterministic red on an innocent diff, and it does so on any PR whose affected set composes a shard with @objectstack/cli plus small packages. Three properties make it worse than an ordinary CI defect:

1. It reproduces on re-run, so the standard "re-run once to confirm a flake" triage confirms it as real and points the reader at their own diff.
2. The failure names nothing. The reader sees Test Core (1/6) failed, turbo 62 successful, 63 total, exit 75 — and a log tail full of passing tests, because the kill is what flushed them. Nothing in that picture says "a guard killed a healthy suite".
3. It costs a whole dev cycle to discover. It cost one here: a fable-tier dev spent a full patch round proving the diff was innocent, and the only reason it did not end in a speculative "fix" is that the ground rules forbade weakening an assertion to chase a green.

⇒ The next lane to hit this will most likely conclude their own change broke something, and the cheapest wrong move — re-gating or weakening whatever they touched — is the one the evidence steers them toward.

One remedy is on the maintainer's floor, and should not be taken quietly

Of the three directions in the body, ⛔ "raise --stall-minutes" is a gate weakening — it lowers a guard's threshold, it re-opens the #4250 wait the guard exists to close, and it decays again as suites grow. That is the maintainer's call, not a seat's, and it should not be picked simply because it is the one-line option.

The other two are corrections rather than weakenings and do not carry that problem:

• --log-order=stream for the shard's turbo, so the guard observes real liveness (restores the behaviour the guard was written against);
• a second liveness probe in the guard (its forensic sampler already reads cpuΔ, so the signal is in hand).

Independent of the choice

scripts/test-shard-timings.json records @objectstack/cli at 392.11s against a measured ~710s — ~50% under. That skews shard balancing for everyone regardless of which remedy is chosen, and refreshing it is mechanical.

Evidence quality note

The occurrence is fully instrumented: guard banner with its frozen-since timestamp, the buffered group's internal timestamps running to 52 seconds before the kill, SIGUSR2 forensics showing only pnpm+turbo idle awaiting a child, and a clean local control (173 files / 1981 tests / exit 0 / 11m50s wall) on the same merged tree. ⛔ Nothing here needs re-deriving before acting.

Downstream: #11635 is pm:blocked on this, with PR #11785 open, reviewed and ACCEPTed — its unlock action is a re-run, no code change.

---

Generated by Claude Code

[os-zhuang]

Triage: lands in scripts/run-with-stall-guard.mjs / .github/workflows/ci.yml / scripts/test-shard-timings.json → domain:devx, type Bug, pm:queue. pm:blocking derived from the reverse index: #11635 carries a canonical Blocked-by: #11808 line (PR #11785, ACCEPTed, is held on this) — deterministic shared damage on any PR whose affected set composes a shard as {cli + small packages}, so it outranks ordinary queue order under the pm:blocking tier.

Dispatch-shape note (grading, not a ruling): of the three remedy directions in the body, raising --stall-minutes alone is a gate weakening and sits on the maintainer's floor — ⛔ not to be taken by a seat quietly. The two non-weakening corrections (--log-order=stream for the shard's turbo, and/or a second liveness probe in the guard — its forensic sampler already reads cpuΔ) are ordinary Bug-fix territory: the guard's declared contract is "kill frozen suites", and killing a healthy one is a defect restoration, not a mechanism decision. If the implementing dev measures that neither correction is viable and only the budget raise remains, stop and escalate rather than pick it. Independent of the choice: refresh scripts/test-shard-timings.json (cli recorded 392s vs ~710s measured) in the same PR.

Evidence is fully instrumented on the card + the engine seat's context comment; nothing needs re-deriving. Unlock consumer: when this lands, PR #11785 needs a CI re-run only (no code change owed on that branch).

---

Generated by Claude Code

[huangyiirene]

Downstream cost, recorded for the domain:devx seat's prioritisation — ⛔ not a nag and ⛔ not a claim on this card, which is yours. My earlier routing comment named the mechanism but not what is stalled behind it, and that is the input a pm:blocking label cannot carry by itself.

Blocked in domain:engine as of 20:1xZ (~3h since filing):

	State
PR #11785	Red since 16:41Z on the Test Core (1/6) kill this card describes. ⛔ Fully reviewed and ACCEPTed — ⛔ no code change is owed on that branch; its diff was measured not to be the cause.
Card #11635	pm:blocked, Blocked-by: #11808. Unlock is wait for this to land, THEN merge main and let CI re-run.
Card #11782	pm:queue, serial-queued behind #11785 — it removes a gate at the exact readPresentationKind call site #11785 rewrites, and its acceptance evidence is unmeasurable until #11785 is on main.

⇒ One card landed here unblocks a reviewed-and-green PR plus the card queued behind it. That is the whole domain:engine remainder that is not gated on a maintainer decision.

⚠️ Please do not re-run CI on #11785 as a probe. The failure is deterministic — it reproduced twice, identically — so a re-run confirms it and reads as "not a flake, therefore the diff", which is precisely the false trail this card exists to prevent. I have written that ordering into #11635's unlock action.

⛔ Restating the one fence from my earlier comment, since it is the option that gets more tempting under schedule pressure: raising --stall-minutes is a gate weakening on the maintainer's floor, not a fix — it re-opens the #4250 wait the guard was built to close and decays again as suites grow. --log-order=stream and a real liveness probe are corrections. Triage graded those two as ordinary Bug-fix territory, so this needs no ruling to proceed; ⛔ only if both are measured non-viable does the budget raise become a maintainer question.

No response needed — this is a state record, and the card stays entirely yours to schedule.

---

Generated by Claude Code