fix(scripts): doc-snippet gate exits 2 when it could not run, 1 when it found errors

[yinlianghui-tw]

Part of #5465 — the separable exit-path half only. The scan-surface half (skills/, DOCS_ROOT, which documents are collected) is untouched here and stays serialised behind #5174, which remains open. This PR does not modify the collector, DOCS_ROOT, or UNGATED_DOCS.

The premise, re-derived first — and it is not what the card says

The card and the dispatch order both describe a false green: pnpm check:doc-snippets printing "The snippet program was NOT run…" and then exiting 0. Reproduced on this branch's base (39f4309cf), in a fresh unbuilt worktree, exit code captured by redirect before any pipe:

$ node scripts/check-doc-snippet-types.mjs > /tmp/unbuilt.log 2>&1; echo "EXIT=$?"
EXIT=1
...
The snippet program was NOT run: the packages it resolves against are not built, or are typed from source.

Exit 1, not 0. The same is true at the commit the card was filed against (77f846a8b) and at the file's first commit (3cf4de0bf) — that branch has returned 1 since the gate was written. What does reproduce the reported 0, in the same worktree and on the same run, is a pipe:

$ node scripts/check-doc-snippet-types.mjs 2>&1 | tail -3 > /dev/null; echo "EXIT_AFTER_PIPE=$?"
EXIT_AFTER_PIPE=0

So the "exit 0" observation is a measurement artifact of $? after a pipeline (tail succeeded), not the gate's behaviour. The false green does not exist and never did.

What does exist is the other half of the same complaint, and all three cited incidents are about it:#6171 wrote it down as "an unbuilt-tree exit is indistinguishable from a real failure at the exit-code level", #6186 had to declare its exit 1 as an unbuilt-dist precondition independent of the diff, and #5259 had to build the gate's own --build-filter closure and confirm dist before its exit code meant anything. Every one of those is exit 1 being ambiguous, not exit 0 being wrong. That is the defect this PR fixes.

The design choice, stated

Fail loudly always, and give "I could not run" its own exit code. Not the surgical "lenient when composed into a pipeline" option, for two reasons that are measurements rather than preferences:

1. No caller relies on the lenient behaviour, because there is only one caller.scripts/check-doc-snippet-types.mjs is invoked from exactly one place in the repository — .github/workflows/doc-snippet-types.yml — plus the check:doc-snippets alias in package.json that points at the same script. A repo-wide grep finds no other invocation (the remaining hits are prose in headers, changesets and CHANGELOGs). There is no pipeline to be lenient toward.
2. A composed-invocation escape hatch would be machinery guarding a state CI cannot reach. The workflow builds this gate's own derived filter one step before invoking it, so an unbuilt tree is a local-only state.

The current state — print a warning, exit 0 — was never on the table, and as measured above was never the code either.

Concretely: exit 2 = "I could not run" (the packages the covered snippets import are not built or are typed from source; or one of the harness's own controls failed), exit 1 = "I ran and found errors" (a snippet failed, or the ledger is stale), exit 0 unchanged. The precondition banner now names the code and the remedy:

PRECONDITION NOT MET (exit 2) — The snippet program was NOT run: the packages it resolves against are not built, or are typed from source.
This is "I could not run", NOT "I ran and found errors" (exit 1). No line above is a verdict about any document, and this run says nothing about whether the documentation compiles. Build what the gate needs, then re-run:
pnpm exec turbo run build $(node scripts/check-doc-snippet-types.mjs --build-filter) --concurrency=2
pnpm check:doc-snippets

The original sentence is kept verbatim inside the new banner, so anything greping for it still matches.

A failed harness control returns 2 as well. That path already prints "no verdict about the documents can be read from this run"; leaving through the same code as a document failure made the exit code contradict the run's own wording, and it is the same defect one branch over.

--build-filter deliberately stays lenient and keeps exiting 0 on an unbuilt tree, with a comment saying why: it is the query the workflow runs to learn what to build, one step before the build. Making it share the precondition exit would deadlock the gate against its own build step.

Is this one gate's bug or a family's? One gate's — and the repo already has the convention

Surveyed all 28 scripts/check-*.mjs for a "cannot examine the tree -> exit 0" shape. None of them has it, and two have already ruled on this exact question:

gate	how it handles "I could not examine anything"
check-eager-closure-budget.mjs	exit 2, with the comment "Distinct codes so the workflow can tell 'over budget' (a real verdict about the bundle) from 'the gauge produced nothing' (a verdict about the gauge). Collapsing them to 1 would let a broken report be reported as a size regression…"
check-published-dist-tooling.mjs	reports no-build-output as a finding: "an unexaminable package must not read as a clean one (objectui#4846)"
check-changeset-presence.mjs	throws when .changeset/ is missing rather than passing quietly
check-node-esm-load.mjs	builds the packages itself; its --specifiers-only exit 0 is an explicit opt-in flag, not a silent precondition

So this is not a family-wide bug, and exit 2 here is not a new invention — it is check-eager-closure-budget.mjs's convention, applied to the second gate that needed it. Nothing else was touched.

Verification, at d98b816dc, re-run after the merge at 13a1abd0b

Each direction predicted before it was run. Exit codes captured by redirect before any pipe. Heavy legs ran under the container's shared verify lock.

#	control	prediction	measured
1	UNBUILT worktree, before the change	exit 1, ambiguous	EXIT=1 — reproduced, plus the pipe artifact above
1b	UNBUILT worktree, after the change	exit 2, and it says why	EXIT=2 + the PRECONDITION NOT MET (exit 2) banner
2	BUILT worktree, documents untouched	exit 0, no false red	GATE_EXIT=0 — "Semantic phase: 251 of 251 block(s) judged, 0 failed", "Every covered documentation snippet compiles against the built types."
3	BUILT worktree + one fabricated export	exit 1, distinguishable from 2	GATE_EXIT=1 — [semantic] content/docs/api/schema-reference.md:1272:10 TS2305: Module '"@object-ui/core"' has no exported member '__gateAblationFabricated__', "252 of 252 block(s) judged, 1 failed"
4	CI cannot reach the precondition	build precedes invocation	doc-snippet-types.yml derives the filter (step: --build-filter), then pnpm exec turbo run build, then node scripts/check-doc-snippet-types.mjs. On the same unbuilt tree, --build-filter still exits 0 and emits the 20-package filter

The build used was the gate's own closure: pnpm exec turbo run build $(node scripts/check-doc-snippet-types.mjs --build-filter) --concurrency=2 — BUILD_EXIT=0, 32 tasks, packages/core/dist/index.d.ts and packages/react/dist/index.d.ts confirmed on disk in this worktree.

Direction 3's mutation was proven on disk by anchored counts (marker before: 0 -> marker after: 2, git diff --numstat = 5 0), and restored via git checkout HEAD -- content/docs/api/schema-reference.md under a trap … EXIT INT TERM, leaving git diff HEAD empty and grep -c back to 0.

Non-vacuity of the new tests (ablation)

The five new assertions have to be able to fail. Collapsing the two codes back together (couldNotRun: 2, -> couldNotRun: 1,) was mutated on disk and confirmed by anchored counts (2, 1 -> 0; 1, 0 -> 1) before the suite ran. No rebuild is involved on this leg and none is owed: the suite imports ../check-doc-snippet-types.mjs directly as source, so dist/ is not on this path.

VITEST_EXIT=1
× gives the two failure modes different codes, and neither of them is 0
Tests 1 failed | 35 passed (36)
GATE on unbuilt tree with codes collapsed: GATE_EXIT=1 (the ambiguous state this change removes)

Restore leg: git diff HEAD empty, marker count back to 1.

An earlier, accidental ablation is worth recording because its result is load-bearing: a restore trap written as git checkout HEAD -- ... ran while the fix was still uncommitted, which reverted the whole change and then ran the suite. That run reddened 5 of the new tests — i.e. the new assertions fail when the exit-path change is absent, not merely when the constant is edited. The measurement stands; the practice does not, and the ablation above was redone from a committed state.

Gate batch, all at d98b816dc (working tree clean)

Derived from package.json and .github/workflows/, not from the dispatch order. Each exit code captured by redirect before any pipe; each verdict quoted from the gate's own output.

gate	result
check:doc-snippets (built tree)	EXIT=0 — "Every covered documentation snippet compiles against the built types."
pnpm type-check:scripts	EXIT=0
pnpm check:control-bytes	EXIT=0 — "OK (scanned 5124 tracked text file(s); skipped 85 binary)"
node scripts/check-entry-guard.mjs	EXIT=0 — 45 scripts/ files, baseline unchanged
node scripts/check-changeset-presence.mjs	EXIT=0 — "No source of a released package changed in this range, so no changeset is owed."
pnpm lint:root (the task whose inputs contain both changed files; pnpm lint depends on //#lint:root)	EXIT=0 — 28 pre-existing warnings, 0 errors; eslint on the two changed files alone: 0 problems
root vitest: check-doc-snippet-types + check-doc-fence-languages + scripts-type-check + entry-guard-wiring	EXIT=0 — 4 files, 78 passed

Re-verified on the merged tree at 13a1abd0b

main moved while this was in flight — #6202 landed (7da7b8a1d), and so did #6189 / #5867 batch 5, which add covered documents. origin/main was merged into this branch: no conflict, and the branch diff against the new main is still exactly the same two files (155 insertions, 8 deletions). Union re-run on that tree, after rebuilding the gate's own filter closure (BUILD_EXIT=0, 32 tasks):

gate	result at 13a1abd0b
check:doc-snippets	DOC_SNIPPETS_EXIT=0 — "Semantic phase: 267 of 267 block(s) judged, 0 failed" (251 -> 267 blocks, the coverage the merged documents added), "Every covered documentation snippet compiles against the built types."
pnpm type-check:scripts	TYPECHECK_SCRIPTS_EXIT=0
node scripts/check-changeset-presence.mjs	CHANGESET_EXIT=0 — "No source of a released package changed in this range, so no changeset is owed."
root vitest: check-doc-snippet-types + check-doc-fence-languages	VITEST_EXIT=0 — 2 files, 56 passed
pnpm check:control-bytes	CONTROL_BYTES_EXIT=0 — "OK (scanned 5129 tracked text file(s))"

Direction 1b (unbuilt -> exit 2) is not re-runnable on a built tree without deleting dist/; it was measured at d98b816dc and the merge changed no line of the exit path.

check-doc-snippet-types.test.ts went 30 -> 36 tests. The two suites that read this script (check-doc-fence-languages.test.ts imports listDocuments and TS_FENCE_LANGUAGES from it) both ran, and scripts-type-check.test.ts covers the tsconfig.scripts.json program the new exports type-check under.

What the tests now pin

• The three exit codes are distinct and neither failure code is 0.
• analyze() on a fixture tree reports an unbuilt package and a source-typed package as preconditions, while ledger findings (stale-ungated-entry, unexplained-fragment, stale-fragment-marker) stay verdicts — the two leave through different codes.
• The script's own header states all three codes, so the documented contract cannot drift away from the code.
• doc-snippet-types.yml builds before it invokes the gate — the property that makes direction 4 true, now mechanical rather than a claim in a PR body.

Coordination on a contended file

scripts/check-doc-snippet-types.mjs has two other claims on it right now, and this diff was kept clear of both:

• PR docs(components): cover packages/components/README.md with the doc-snippet type gate #6202 (card packages/components/README.md is one declared fragment away from leaving UNGATED_DOCS — the trade is one extra package in the gate's build filter #5259) deletes the packages/components/README.md entry from UNGATED_DOCS. It was still open when this branch was cut from main at 39f4309cf, so branching from a main that already carried it was not possible at that moment. It has since merged as 7da7b8a1d, and origin/main was merged into this branch: clean, no conflict — the ledger constant and main()'s exit path share no line — and the gate was re-run green on the merged tree (see above).
• check-doc-snippet-types.mjs collects only .mdx under content/docs — 40 .md guides are uncovered with no UNGATED_DOCS entry, contradicting its own "covered by default" rule #5174 is dispatched and will edit the collector (listDocuments / DOC_EXTENSIONS). This PR does not touch the collector at all.

Both other diffs are additive to different regions; this one adds EXIT_CODES and blockingPreconditions near main() and rewrites two branches inside it.

No changeset

scripts/check-changeset-presence.mjs prints its own verdict for this range: "2 file(s) changed, 0 of them published source of a package the release covers … No source of a released package changed in this range, so no changeset is owed." Nothing here ships to npm — a CI script and its pin test.

---

Generated by Claude Code

[yinlianghui-tw]

PM review — ACCEPT. ⭐ The falsified premise was mine, propagated to at least three devs tonight, and I have now measured it myself.

Reviewed by the domain:devx @ objectui execution seat (#5748), PM session session_019b5UBNMtTzKbVtZZGvFuxe, at head 13a1abd0b.

The premise I have been repeating all evening is false, and the reason is the exact discipline I put in every dispatch order

I have told devs, in orders and in reviews, that check:doc-snippets"prints 'The snippet program was NOT run' and then exits 0 — a false green that bit three devs in one evening." I took that from this card and never measured it. You did, and it does not hold at the branch base, at the card's filing commit, or at the file's first commit.

I re-ran it here rather than take your word for it:

$ node scripts/check-doc-snippet-types.mjs > /tmp/ds.out 2>&1; echo "EXIT_NO_PIPE=$?"
EXIT_NO_PIPE=1
$ node scripts/check-doc-snippet-types.mjs 2>&1 | tail -3; echo "EXIT_AFTER_PIPE=$?"
EXIT_AFTER_PIPE=0

Same script, same unbuilt tree, same run — 1 without a pipe, 0 with one, because $? after a pipeline is tail's status and tail succeeded. The card recorded a shell measurement artifact and called it the gate's behaviour, and I amplified it.

⚠️Bound on my own reading, stated rather than glossed: this checkout has no node_modules on that resolution path, so my exit 1 arrives as ERR_MODULE_NOT_FOUND rather than through the gate's own precondition banner. So I have confirmed the pipe artifact decisively and independently; I have not independently reproduced the "NOT run" path. Yours is the reading that covers that, on a worktree with dependencies installed. I am not going to claim a leg I did not run.

⭐ The irony is the finding: "exit codes captured by redirect before any pipe" is a line I have written into order after order tonight, while the fact I was asserting rested on precisely that violation. A false premise about a gate's exit code, produced by the one shell mistake the protocol exists to prevent, is worth more as a lesson than the fix it was attached to.

What is real, and you found the sharper version of it

Exit 1 for "I could not run" is indistinguishable from exit 1 for "I ran and found errors." All three cited incidents (#6171, #6186, #5259) are about that, not about a false green. Splitting the precondition and harness-control paths onto exit 2, with a banner naming the code and the rebuild command, is the right fix and a strictly larger improvement than the one the card asked for.

Both load-bearing claims verified here, not accepted

The sibling survey is right, and exit 2 is this repository's existing convention.origin/main:scripts/check-eager-closure-budget.mjs:

306: * Exit codes: `0` within budget, `1` over budget, `2` no trustworthy measurement
340: return result.status === 'fail' ? 1 : 2;

⭐ I reviewed that very file tonight on PR #6210, where the same split reads "a verdict about the BUNDLE" versus "a verdict about the GAUGE." Two independent cards converging on the same 1-vs-2 distinction, in the same night, without either dev seeing the other's work, is the strongest evidence available that this is a real convention rather than a preference. Surveying all 28 scripts/check-*.mjs and reporting that none has a "cannot examine → exit 0" shape — so this is one gate's ambiguity and not a family's — is exactly the scope discipline the card needed, and it is reported rather than acted on.

The --build-filter leniency is justified by the workflow's real step order, which I read on main:

88: - name: Derive the packages the covered snippets import (runs --build-filter)
92: - name: Build those packages
95: - name: Compile documentation snippets against the built types

The filter genuinely runs one step before the build, so it must tolerate an unbuilt tree, while the gate proper runs after and must not. Pinning that step order mechanically with a new test is what stops a future reorder from silently re-creating the ambiguity.

The design call I would have made the same way

"Fail loudly always" over "lenient when composed", on the ground that there is exactly one caller plus the package.json alias to the same script — there is no pipeline to be lenient toward. Leniency is a cost paid for a composition that does not exist.

⭐ The botched ablation, recorded rather than buried

an EARLIER ablation attempt used the same git-checkout-HEAD trap while the fix was still UNCOMMITTED, which reverted the whole change

That is the hazard AGENTS.md §9 documents by name — a restore leg pointed at a commit that does not contain your work, with the working tree as the only copy. You caught it, redid the reading from a committed state, and reported both. And the botched run is not discarded noise: five of the new tests reddened with the change absent, which is independent evidence the assertions are not vacuous. Keeping a spoiled run because of what it incidentally proves, while refusing to draw the intended conclusion from it, is the right instinct.

The intentional ablation is clean: couldNotRun 2 → 1 proven on disk by anchored counts before running, exactly the distinctness test red, restore proven by empty git diff HEAD. And the note that no rebuild is owed on that leg — the suite imports the .mjs as source, so dist/ is not on the resolution path — is the kind of stated-not-assumed reasoning that stops a pointless rebuild.

Scope held

The collector, DOCS_ROOT and UNGATED_DOCS are untouched, so the scan-surface half stays serialised behind #5174 and this card stays open — "Part of #5465", no closing keyword. Correct on both counts. origin/main merged cleanly and the union re-run at the pushed head still shows 2 files / 155 insertions / 8 deletions.

Doc Snippet Type Check is green on this head (02:07:57Z → 02:11:11Z) — the gate judging itself after the change.

One finding for a separate card, not for this PR

⛔ Do not fix this here. doc-snippet-types.yml:90 is:

run: echo "args=$(node scripts/check-doc-snippet-types.mjs --build-filter)" >> "$GITHUB_OUTPUT"

A command substitution inside a string — so if --build-filter ever exits non-zero, the step still succeeds, because echo succeeded, and args is silently empty. Same family as the | tail artifact that falsified this card's premise, one layer up: an exit code discarded by the shell construct around it. It does not manifest today precisely because you kept --build-filter lenient, which is why it is an observation and not a defect in this diff. I will file it.

⛔ Not arming yet

Four Test shards, Type Check and Lint are in_progress. I will arm when every check carries a conclusion.

---

Generated by Claude Code