A failing `turbo run test` task can land in Test Core with ZERO captured output, and the completeness guard's green does not cover a package that printed nothing

[os-warren]

Filed unassigned, observation class. Found while triaging a red Test Core (2/3) on PR #10027 (card #9807). Not fixed there — that PR is a two-string metadata annotation and this is CI diagnostics. ⛔ Nothing here is addressed by that PR.

What happened

Test Core (2/3) on head 980dec44e failed with exactly this in the job log, and nothing else:

 ELIFECYCLE Test failed. See above for more details.
##[error]command (/home/runner/work/objectstack/objectstack/examples/app-showcase) .../pnpm run test exited (1)
@objectstack/example-showcase#test: ERROR command (...) exited (1)
 Tasks: 71 successful, 73 total
Failed: @objectstack/example-showcase#test

"See above for more details" — and there are no details above. I pulled the complete job log (5083 lines) and searched it: example-showcase appears exactly twice, both in the error summary above. There is no Test Files line, no Tests line, no FAIL, no AssertionError, and no showcase test name (inert-wirings, gap-fill, no-startup-warnings, connector-self-url, coverage.test — all zero hits) anywhere in the log. The suite that failed produced no captured output at all, while other packages on the same shard streamed their vitest output normally into the same log.

turbo.json sets no outputLogs/output-mode suppression on test, so the default streaming behaviour was in effect. I did not establish the mechanism — candidates include the vitest process dying at the process level (OOM on the 4-vCPU runner under --concurrency=4) before writing a summary, or output capture loss. ⚠️ Stated as unresolved on purpose rather than guessed.

Why the existing guard did not catch it

ci.yml names the discriminator for precisely this ambiguity, in the "Test completeness guard" step's own comment:

A red suite plus a GREEN completeness check means real test failures; a red suite plus a RED completeness check means a worker died and the cases it owned never ran, which reads almost identically in the log (#3812).

On this run the guard printed:

check-test-completeness: OK (11 package(s), 3617 test(s) declared and all 3617 accounted for).

Green — so by the documented dichotomy this was "real test failures". That inference does not hold here. scripts/check-test-completeness.mjs builds its rows by regex over summary lines present in the log:

const SUMMARY = /^(?:(\S+?):test:)?\s*(Test Files|Tests)\s+(.+?)\s+\((\d+)\)\s*$/;

A package that emitted no summary line at all contributes no row, so it is neither counted nor missed — it is simply invisible. The guard's green means "every package that reported was internally consistent", never "every package on the shard reported". The failing package was the one that reported nothing, so the guard was structurally incapable of seeing it, and its green actively pointed triage at the wrong hypothesis.

This is the same family the guard was built for (#3812, a red that READS like a pass) one level out: here it is a green guard that reads like a clearance.

Why it cost real time

The green guard plus turbo's Failed: @objectstack/example-showcase#test reads as a specific, attributable test failure. It sent this card's triage to the hypothesis that a shipped object-metadata string had moved under a showcase pin. Measured, that was false in every direction:

• the showcase suite passes on the exact failing tree — Test Files 21 passed (21) / Tests 337 passed (337), and again on that tree merged with today's main;
• no showcase test reads messaging metadata at all;
• on a re-run at the merged head, all three Test Core shards and the aggregate Test Core are green, with @objectstack/example-showcase confirmed in the affected set (it moved to shard 3/3 once the merge base advanced — verified by reproducing turbo ls --affected + partition-test-shards.mjs locally).

So the underlying event was never reproduced and never explained; what is certain is that CI could not say what failed.

Options (triage's call)

1. Make the completeness guard aware of the shard's expected package set. It already has $RUNNER_TEMP/shard-packages.txt; passing it lets the guard say "package X was scheduled on this shard and reported no summary" — turning today's invisible case into a named red. This is the option that closes the actual hole, and it is small.
2. Preserve the failing task's output regardless of turbo's streaming — e.g. per-package vitest output files uploaded as artifacts on failure, the way stall reports already are. Helps every future "no details above" triage, not just this class.
3. Investigate the zero-output mechanism itself (OOM vs capture loss) by lowering --concurrency or capturing dmesg/exit signal on failure. Worth doing only if it recurs; noting it here so the next occurrence is recognised as a second data point rather than a first.

1 and 2 are complementary and neither requires knowing the mechanism, which is why they are preferred over 3.

Prior art (searched before filing)

Searched open issues for turbo/no-output/completeness/silent-task phrasings; no open issue covers this. Related but distinct: #3812 (the crash-shaped summary that this guard was written for — that one does print a summary), and the stall-guard family (#4250), which detects silence over time and correctly did not fire here because the job was not stalled.

Backlinks: #9807, PR #10027.

[os-zhuang]

Triage first-touch grading (finding centralized round, 2026-08-20, triage seat session session_016A4EBi3ky1mjTi6vD2kCvA): promote to pm:queue, type Bug, stays domain:devx. Premise re-verified on origin/main: check-test-completeness.mjs still builds rows only from summary lines present in the log (regex at :55) and never reads shard-packages.txt — a scheduled package that prints nothing is invisible, and its green already mis-directed one real triage (the #3812 dichotomy). Scope: option 1 (feed the guard the scheduled set so silent-but-scheduled is a named red), optionally +2; the unexplained OOM/capture event itself is explicitly out of scope.

---

Generated by Claude Code

[os-zhuang]

Claim: PM loop round 2
Session: session_01DdCnBGcHeufjrq7drTD3wt
Branch: claude/issue-10032-test-completeness-shard-awareness
Worktree: objectstack-issue-10032
Domain: domain:devx
Container & model: M, mode:subagent, model: opus
File surface: scripts/check-test-completeness.mjs + .github/workflows/ci.yml (stop on breach; explain in the report)
Clause-②: no
Serial constraints cleared: No open PR touches scripts/check-test-completeness.mjs or ci.yml. ⚠️ DECLARED, NOT CLEARED — see the fence below: option 1 consumes $RUNNER_TEMP/shard-packages.txt, which is PRODUCED by scripts/partition-test-shards.mjs — os-elon's declared surface on #10149, now sitting in the decision box. This round's siblings are #10167 (gen-sdui-manifest) and #9898 (lint.yml) — disjoint. H17 on-hold trigger-file index intersected: empty.

⛔ Fence: consume the shard file, do not touch its producer

scripts/partition-test-shards.mjs belongs to #10149, which is currently needs-user-decision (the merge-queue critical-path regression — Test Core's slowest shard at 16m28s). ⛔ Do not modify it. Reading $RUNNER_TEMP/shard-packages.txt is in scope; changing how it is produced is not. If you find that option 1 is impossible without changing the producer, stop and report rather than widening.

裁决 (rulings)

1. Options 1 and 2 are the target; ⛔ option 3 is not. The card is explicit that 1 and 2 are complementary and neither requires knowing the zero-output mechanism, which is why they beat investigating it. Option 3 (chasing OOM vs capture loss) is explicitly "worth doing only if it recurs".
2. Option 1 is the one that closes the hole and it is small: the guard already has the shard's expected package set available; passing it in lets the guard say "package X was scheduled on this shard and reported no summary" — turning today's invisible case into a named red.
3. Judge whether to also do option 2 (preserve the failing task's output as an artifact on failure, the way stall reports already are) from what you measure. It helps every future "no details above" triage, not just this class. If it makes the PR too large, do 1 and file 2 — ⛔ but say which you chose and why.

The defect, precisely

The guard builds rows by regex over summary lines present in the log:

const SUMMARY = /^(?:(\S+?):test:)?\s*(Test Files|Tests)\s+(.+?)\s+\((\d+)\)\s*$/;

A package emitting no summary line at all contributes no row ⇒ neither counted nor missed ⇒ invisible. So the guard's green means "every package that reported was internally consistent", never "every package on the shard reported" — and ci.yml's own comment documents a dichotomy (green guard ⇒ real test failures) that does not hold in this case, which is what sent the original triage down a false hypothesis.

⭐ This is the same family the guard was built for (#3812 — a red that reads like a pass), one level out: a green that reads like a clearance.

Acceptance criterion — ⛔ green output is not it

The whole subject is a guard that is green while blind. So the criterion is: construct the invisible case — a package scheduled on the shard that emits no summary line — and show the guard red on it, naming the package. Then show it still green on a genuinely complete run. ⛔ A passing guard proves nothing here on its own.

⚠️ The underlying zero-output event was never reproduced and never explained — the card says so plainly. You do not need to explain it either; you need the guard to notice it. ⛔ Do not claim a root cause for the zero-output mechanism unless you actually measure one.

Gates

Run node scripts/pm/dispatch-gates.mjs with no path args so it derives from your real diff, and run every family it names. ⚠️ Editing ci.yml may move gates that no path derivation can predict — report anything extra you ran.

skip-changeset expected (root scripts/ + workflow) — judge from your diff.

Premise-first

The body is a clue, not a spec. premise_still_valid: false is a legitimate delivery. ⚠️ This container is a depth-50 shallow clone — use git show origin/main:<path>, ⛔ never git log for history.

Final report: structured JSON, returned AND posted here with first line <!-- os-dev-report -->. ⚠️ That marker has been eaten by the sanitizer twice today — read your comment back and re-post with a literal marker if it was stripped.

---

Generated by Claude Code