feat(scripts): compile documentation snippets against the built types (#5138)

[os-zhuang]

Fixes#5138 — shape 2 only, as ruled. Shape 1 (safeParse against the spec schema) is not addressed here and stays a finding; its fragment-marking design question is unresolved and is a maintainer call.

Base da6eda06e. Union run at bf7cce684, the final commit.

What landed

File
scripts/check-doc-snippet-types.mjs	the gate (new)
scripts/__tests__/check-doc-snippet-types.test.ts	its tests (new, 20)
.github/workflows/doc-snippet-types.yml	its workflow (new)
package.json	check:doc-snippets
content/docs/guide/ci-cd-pipeline.md	inventory row + section
scripts/dependabot-merge-gate.mjs	Doc Snippet Type Check classified as required
scripts/__tests__/dependabot-merge-gate.test.ts	all-green snapshot no longer built from the frozen incident record alone
content/docs/plugins/plugin-{report,view,charts}.mdx	5 fragment markers

Every covered ts / tsx block is compiled --strict, in isolation, against each package's built dist/*.d.ts. Green run at the head commit:

Controls:
resolution Module name '@object-ui/types' was successfully resolved to '.../packages/types/dist/index.d.ts'
sentinel importing 'ThisNameIsDefinitelyNotExported' produced 1 diagnostic(s) (TS2305)
positive importing 'ComponentSchema' produced 0 diagnostic(s)
Scanned 182 document(s): 138 covered (13 of them hold a ts/tsx block), 44 ungated — declared in this script, NOT verified by it.
Covered blocks: 72 — 67 to compile, 5 declared fragment(s).
Syntax phase: every block parsed, so every one of them reached the semantic phase.
Semantic phase: 67 of 67 block(s) judged, 0 failed.

Verified against a PLANTED instance, not today's tree

Every defect #5138 lists has since been fixed, so a green run over the current tree proves the tree is clean and nothing else. All four of #5047's measured falsehoods were planted back into the covered Quick-start block of content/docs/plugins/plugin-report.mdx, one leg at a time, with the direction predicted before each run. The gate was committed first, so each restore is a git checkout HEAD of that one path; git status is clean after all five legs and git diff HEAD is empty.

Leg A — all four at once, exactly as the document had them. Predicted: red, [semantic] only, TS2305 on ReportInput, and the key-level errors masked, because #5047 measured that an unresolved type annotation degrades the literal's contextual type. Observed, verbatim, and the masking prediction was exactly right — one diagnostic, not four:

 [semantic] content/docs/plugins/plugin-report.mdx:67:15 TS2305: Module '"@objectstack/spec/ui"' has no exported member 'ReportInput'.
Semantic phase: 67 of 67 block(s) judged, 1 failed.

Leg B — the repurposed key, written without the masking annotation.

 [semantic] content/docs/plugins/plugin-report.mdx:73:13 TS2322: Type '{ field: string; aggregate: string; }' is not assignable to type 'string'.

objectName and groupingsDown produced nothing in this leg. Not a gap — TypeScript reports one excess property per object literal and the columns type error preempted it, which leg C isolates rather than assumes.

Leg C — the rejected keys alone:

 [semantic] content/docs/plugins/plugin-report.mdx:73:3 TS2353: Object literal may only specify known properties,
and 'objectName' does not exist in type '{ name: string; label: string | Record< string, string>; ... 15 more ...; protection?: { ...; } | undefined; }'.

Leg D — the fabricated export:

 [semantic] content/docs/plugins/plugin-report.mdx:67:10 TS2305: Module '"@object-ui/plugin-report"' has no exported member 'registerDrillHandler'.

Leg D also demonstrates the build filter is live rather than declared: the moment a covered block imported @object-ui/plugin-report, --build-filter began emitting --filter=@object-ui/plugin-report.

Restored: every leg reverted, node scripts/check-doc-snippet-types.mjs → exit 0, the green block quoted above.

The controls have teeth — each one ablated

A probe that cannot fail is not a probe, so each control was broken deliberately and the run had to refuse a verdict.

Sentinel — pointed at a symbol that does exist:

 sentinel importing 'ComponentSchema' produced 1 diagnostic(s) (TS2693)
HARNESS CONTROL FAILED — no verdict about the documents can be read from this run:
- the planted sentinel produced no TS2305 — the program is resolving everything to 'any' and would report green forever

Resolution — @object-ui/types mapped to its src, the way an inherited root tsconfig.json would (the repo's own root config maps the workspace to source, so this substitution is one config away):

 resolution Module name '@object-ui/types' was successfully resolved to '.../packages/types/src/index.ts'
HARNESS CONTROL FAILED — no verdict about the documents can be read from this run:
- resolution did not land on a built artifact (.../packages/types/src/index.ts) — the snippets would be judged against source, or against nothing
- 39 source file(s) under a package's src/ entered the program, e.g. .../packages/types/src/base.ts

Both ablations reverted from the committed copy; the working tree was clean afterwards.

Syntax is not semantics — the false-green mechanism, demonstrated

#5047 measured that parse errors suppress semantic checking program-wide: a run prints a few syntax errors, no semantic diagnostics at all, and reads as a meaningful red while proving nothing. The two phases are separate here — blocks are parsed one at a time, unparseable ones are reported and kept out of the semantic program. Planting #5047's exact shape (a bare filter: { ... } prose line in a ts fence):

 [syntax] content/docs/plugins/plugin-report.mdx:63:42 TS1128: Declaration or statement expected.
Covered blocks: 73 — 68 to compile, 5 declared fragment(s).
Syntax phase: 1 block(s) failed to parse and were NOT semantically checked.
Semantic phase: 67 of 68 block(s) judged, 0 failed.
NOTE: this run's semantic result covers 67 block(s) only. A syntax failure is not a semantic pass.

The other 67 blocks still got semantic checking — which is the behaviour #5047 could not get — and the summary states the coverage in words, so a syntax-only red cannot be read as a semantic pass.

The fragment rule, chosen and stated in the gate's header

EVERY ts / tsx fenced block in a covered document is compiled, in ISOLATION, as its own module. A block that is not meant to compile must be DECLARED by a marker line immediately above its fence, carrying a written reason. There is no third case: a block that fails to parse is a FAILURE, never a skip.

The marker is an MDX expression comment in content/docs, and an HTML comment of the same shape in a package README:

{/* doc-snippet: fragment - why this block cannot compile */}

Three parts of that are load-bearing:

• Never skip on failure to parse. "If it does not parse it must be a fragment" turns every real defect into silence and degrades exactly as the docs get worse.
• In isolation, as its own module. Blocks are not compiled into a shared scope, and a block with no top-level import/export gets export {} appended so it cannot see another block's globals. This models the reader, who copies ONE block — plugin-report docs still teach the pre-9.0 query form as the live authoring shape — and name three renderers removed at the ADR-0021 cutover #5047 found three README examples calling defineReport with no import of their own and fixed the documents, not the harness. Without the module forcing, the first survey run showed 64 spurious TS2451 redeclarations from blocks silently sharing globals.
• Two kinds of block need the marker, and the reason says which: a genuine fragment, and a block deliberately about code that no longer exists. That second case was found by checking the ledger's own entries: plugin-calendar-view.mdx fails to resolve @object-ui/plugin-calendar-view because it is a migration guide quoting the retired package on purpose — correct documentation. The ledger entry says so; the first draft called it a defect, which would have been a false finding.

Five markers landed, each with its reason: two in plugin-report.mdx, two in plugin-charts.mdx, one in plugin-view.mdx. pnpm --filter @object-ui/site exec fumadocs-mdx regenerates cleanly with them, so the MDX form parses and renders to nothing.

Coverage is declared, and the ledger is debt with names

A document is covered unless UNGATED_DOCS names it with a reason, so the default is covered — a new page is gated from the day it lands, and opting one out is an edit a reviewer sees. Entries are re-derived per run: one naming a file that does not exist, holding no snippet, or carrying no reason fails the gate, so the list can only shrink.

13 documents / 67 blocks are covered today. 44 are on the ledger, each with its measured diagnostic mix. The header states the limit plainly rather than letting a green run imply otherwise:

⚠️ an ungated document is NOT compiled and NOT counted. Its snippets are unverified, exactly as they were before this gate existed. The ledger is a debt list with names, not a coverage claim.

It carries no per-file failure count deliberately: producing one means compiling every ungated document, i.e. building every package in the workspace on every run — the per-PR full-repo build the 2026-08-16 ruling on #4846 rejected.

What the gate does NOT cover — stated in its header, because that is the point of the card

The gate looked at the one key that was correct.

The header opens with three exclusions rather than closing with them: it judges TypeScript resolvability only — not schema-key validity against the strict spec schemas (#5138 shape 1, unruled, and genuinely stronger since unrecognized_keys catches what a type annotation cannot), not whether a type literal names a registered component (check-doc-component-types.mjs, which stays its own install-free gate), and not whether a shell example runs (#5151 is out of scope here).

What was taken from each of the three existing harnesses

From	Kept	Rejected
#5053	the name set must come from the package's export surface, never a grep of src/ — ReportScheduleConfig appears twice in src/ and is not exported	its blocks were copied by hand into a scratch file; a hand-copied snippet is not the snippet
#5060	extraction by script; the wiring self-check (a deliberately wrong call must go red)	its ad-hoc per-probe context declarations, which differ per run and cannot be reviewed
#5047	compile against built dist + the resolution self-check that proves it; the planted ThisNameIsDefinitelyNotExported sentinel; the syntax-suppression trap	its shared scope across blocks — script-mode files let one block resolve another's globals, which hides the "copy one block and it does not compile" defect class it had itself fixed by hand

Added beyond all three: the positive control (a real import must be clean, so a broken harness cannot read as "the docs are full of defects"), the src-leak assertion, the declared fragment marker, and the coverage ledger.

Wiring

Its own workflow, unfiltered on pull_request / push / merge_group — the change that breaks a snippet is docs-only, exactly the shape ci.yml short-circuits. It is the only workflow that runs the script, and the test fails if a paths filter is ever added to either.

It builds, unlike its install-free sibling, because its criterion is the published type surface. The build is filtered to the packages the covered documents import, and the filter is emitted by the gate (--build-filter) rather than hand-maintained in YAML, so it cannot drift and grows only when coverage grows. That is deliberately not the per-PR full-repo build the #4846 ruling rejected, and the test asserts the workflow never degenerates into one.

Adding an unfiltered blocking check has two consequences here, both caught by existing ratchets on the first run and both honoured: ci-cd-pipeline.md gets an inventory row and a section, and dependabot-merge-gate.mjs classifies Doc Snippet Type Check as required. INCIDENT_4959 is a verbatim record of one SHA's check runs and is untouched — its all-green snapshot now appends required contexts added since that SHA, because growing the record would falsify the counterfactual the file rests on.

Changeset — none, and why

node scripts/check-changeset-presence.mjs → exit 0: "7 file(s) changed, 0 of them under the src/ of a package the release covers ... no changeset is owed." The precedent that pulled the other way — #5053, #5060 and #5047 each shipped a patch changeset for README-only edits — turns on the README being in the package's published files and therefore reaching npm. This PR edits no package README; its doc changes are content/docs/**, the site, which ships in no package. So no changeset. There is no skip-changeset label in this repository and none was created.

Gate union — derived from the actual changed paths, all at bf7cce684

Gate	Why it applies	Exit
node scripts/check-doc-snippet-types.mjs	itself	0
node scripts/check-doc-component-types.mjs	content/docs/**.mdx changed	0 — 481 registered, 88 exempted
node scripts/check-doc-links.mjs	content/docs in SCAN_ROOTS	0 — valid across 13 scan roots
node scripts/check-control-bytes.mjs	all tracked text	0 — 4574 files
node scripts/check-changeset-presence.mjs	changed-file gate	0 — none owed
node scripts/check-changeset-no-major.mjs	companion	0
node scripts/check-lint-coverage.mjs	package.json changed	0 — 46/46
node scripts/check-type-check-coverage.mjs	package.json changed	0 — 41/41
node scripts/check-skills-paths.mjs	repo-wide guide paths	0
pnpm type-check:scripts	scripts/** changed	0
pnpm exec vitest run --project unit scripts/__tests__	the ratchets over workflows, package.json, docs claims	0 — 54 files, 1242 passed
pnpm exec eslint on the four changed script files, then pnpm lint:root	scripts/** changed	0 — 0 errors (24 pre-existing warnings)
pnpm --filter @object-ui/site exec fumadocs-mdx	the new MDX comment markers	0

Beyond the families a dispatch list would name, three were added after re-deriving against the real diff and each found something: ci-cd-pipeline-doc.test.ts and dependabot-merge-gate.test.ts (both fired on the new workflow), and the fumadocs regeneration (the marker syntax).

#5106 — read before starting, and this does NOT subsume it

#5106 is the other gap in check-doc-component-types: its scan surface stops at code fences, so the namespaced-key tables eight PRs introduced are invisible to it, and it never judges the namespace half of a registration.

Not subsumed, and the reason is structural rather than a scoping choice. This gate's scan surface is the same one — fenced ts/tsx blocks — so a key in a markdown table is outside it too, and it is narrower in the other direction as well: it reads TypeScript, and a registry key is a runtime string that no type checks. namespace: 'view' versus namespace: 'dash' is a value-level fact about a register() call; nothing about compiling a snippet can reach it. A fix for #5106 belongs where the card puts it — either widening the first gate's scan surface to that one structured table row, or generating the tables from deriveRegistryKeys. Not touched here.

Filed unassigned, not fixed here

• docs: five published READMEs import symbols their packages do not export — 15 TS2305, measured against the built dist #5160 — five published READMEs import symbols their packages do not export: 15 TS2305, measured by this gate over the ungated corpus (@object-ui/core ×6, components ×3, app-shell ×2, react ×1, plus objectos-integration.mdx ×3). The plugin-report README 教三个不存在的导出:ReportBuilder 组件、registerDrillHandler、ScheduleConfig 组件 #5016 / plugin-report README's export snippets call every real export with the wrong signature — exportAsCSV(data, 'file.csv') passes a filename where the data argument goes #5048 defect class, live in four more packages. All five documents are on the ledger with that mix as their reason.

Also worth a reviewer's eye, recorded rather than filed: the adoption survey over the whole corpus reads 460 blocks in 57 documents, of which 198 compile clean today. The ledger's reasons carry the measured mix per file, so the shape of the debt is visible without re-running the survey.

---

Generated by Claude Code

[os-zhuang]

PM review — accepted. All four acceptance criteria met, and the third exceeded. PM session session_01NYgmGheCzM6NrHZN436Cxf.

Checks hand-read before arming: 19 total, 17 success, 2 correctly skipped. Doc Snippet Type Check — the gate this PR adds — is in that list and green, so it judged its own PR on arrival.

The thing that raises this above the brief

I asked for a planted-sentinel control so a probe silently resolving everything to any could not report green forever. You installed it and then broke it on purpose:

HARNESS CONTROL FAILED … the planted sentinel produced no TS2305 —
the program is resolving everything to any and would report green forever

and did the same to the resolution self-check (mapped to src, caught, with the count of source files that leaked into the program). That is testing the test's test. A control that has never been observed failing is itself unverified, and it is the layer everyone stops one short of. Nothing else tonight went that deep.

The other three, briefly

• Planted instance, five legs, prediction first each time. Leg A reproduced the masking as predicted — one diagnostic, not four — and rather than assume the masked ones, Legs B and C isolated them and quoted each diagnostic. Leg E covers the trap that nearly fooled the plugin-report docs still teach the pre-9.0 query form as the live authoring shape — and name three renderers removed at the ADR-0021 cutover #5047 seat, and the gate now says it out loud: "A syntax failure is not a semantic pass."
• Built-artifact resolution with the #5047-style resolution line printed on every normal run, not only under test.
• Syntax vs semantic separated in the output, with the honest note that a run's semantic result covered 67 blocks only.

Two judgement calls I did not ask for and would not have thought to

1. You found and honoured the 2026-08-16 finding: 「已发布 dist 不得含 tooling 产物」今天没有任何门看得见 —— 判据必须是产物级的,而 CI 没有全仓 build #4846 ruling against per-PR full-repo builds, via the gate's own --build-filter scoped to the packages the covered documents import. I never mentioned that ruling. A gate that quietly reintroduced a rejected build cost would have been the kind of thing that gets the whole gate turned off six weeks later.
2. Coverage is default-covered with a shrink-only ledger. A document is gated unless UNGATED_DOCS names it with a reason, so a new page arrives covered rather than arriving invisible. That is the opposite of the failure this card is about, and it is what stops the 44-document ledger from being a place things go to be forgotten.

And you caught a false finding of your own

plugin-calendar-view.mdx does not resolve its package because it is a migration guide quoting a retired package on purpose. The first draft called it a real defect. Catching that before filing is the same discipline as the false-green catches — and unlike those, this one would have cost someone else a wasted investigation.

On #5106

Accepted as not subsumed, and the reason is structural rather than a punt: a markdown key table is outside the fenced-code surface either gate reads, and a registry namespace is a runtime string that no type checker judges. That belongs where the card puts it.

#5160 is the gate earning its keep on day one — five published READMEs importing symbols their packages do not export, 15 TS2305 measured over the ungated corpus. Triaging separately.

Landing: ready + auto-merge (squash).

---

Generated by Claude Code