Skip to content

fix(docs-audit): derive the docs-drift advisory from what pages document, not package edges - #9229

Merged
os-project-manager merged 1 commit into
mainfrom
claude/issue-9192-docs-drift-content-map
Aug 17, 2026
Merged

fix(docs-audit): derive the docs-drift advisory from what pages document, not package edges#9229
os-project-manager merged 1 commit into
mainfrom
claude/issue-9192-docs-drift-content-map

Conversation

@os-project-manager

Copy link
Copy Markdown
Collaborator

Fixes#9192

The docs-drift advisory resolved pages by package dependency — "which hand-written docs reference @objectstack/x". That is a dependency-graph proxy answering a semantic question, and the card measured it wrong in both directions on PR #9191: 3 pages listed of which 1 was relevant, while the 2 pages that actually document the changed surface were absent, because they document it through the SDK, which does not depend on the implementing package at all.

Taking the first-touch grading's route: derive from what pages document.

The derivation

A page is listed when it names something the change touched. Three anchor kinds:

anchorderived from
symbolthe documentable declaration enclosing each changed line — a top-level declaration, or a member of a top-level container (class / interface / type / enum / schema object). Both sides of the diff, so a removed export still anchors the pages naming it.
routea wire path on a changed line, plus every route whose registrar handler references a changed symbol.
sdkthe client method the repo's route ledgers bind to an anchor route.

The last two hops are what cross the boundary the package graph cannot: auditMetaItem (changed) → GET /api/v1/meta/:type/:name/audit (rest-server.ts registrar) → meta.getAudit (rest-route-ledger.ts) → the token api/client-sdk.mdx actually contains. Both hops are declared data in the repo, not inference.

A local variable is not documentable surface. That one rule drops the measured false positive: const singular = request.type; inside a method body is not an anchor, so kernel/services-checklist.mdx — whose only singular is a service slot name — is gone. A const object is a container (its keys are metadata property names, which docs do name); a function body is not.

Two guards, both publishing what they removed

The first build of this was, on some PRs, noisier than the proxy it replaced — 134 rows where the old tool gave 26. Two guards fixed that, and both run before the route bridge, because a weak name left in the set does not merely add a noisy row: it mints noisy route and SDK anchors from every registrar handler that mentions it.

  1. Shape — an anchor must be code-shaped (camelCase / PascalCase / snake_case / dotted). label, object, start, locale, sections all arrived as real declarations and matched 82, 113, 43, 13 and 10 of 178 pages. Confining them to code spans does not help; those words live in code spans too. Reported as weakAnchorsDropped.
  2. Corpus share — an anchor matching more than 15% of the corpus is a hub term. ObjectQL is code-shaped, genuinely changed, and named by 59 of 178 pages; it cannot tell an author which page to re-read. Reported as overbroadAnchors, with the count that condemned it.

Plus a cap on the bridge: a symbol wired into more than 3 routes is a cross-cutting helper (sendError and co. pulled six unrelated route families into a REST error-typing change). Reported as crossCuttingSymbols. SCREAMING_SNAKE constants stay out of the bridge — a data table is consulted by handlers, it is not their implementation.

What it cannot see is reported, never implied

anchorlessChanges lists changed files that yielded no anchor at all. The superseded coarse set is still computed and emitted as packageMentionDocs, labelled, so an audit that wants the wide net can still ask for it — and so a reader can tell a narrow list from a blind one. The PR comment renders all of it in a collapsed "What this run could not see" section, and distinguishes "no page names any of the N anchors" from "nothing in this diff resolved to a documentable surface, so this run has no opinion". The card's central point was that the tool never signalled its own limits at the point of use; that is the half this addresses.

Measured, across ten real PRs

Each re-derived at its own merge base with its own docs corpus (docs row counts):

PR / commitoldnew
#9191 — the three metadata read verbs (the card's specimen)43
0668f02a6 fix(rest): closed ErrorCode union on the error responder2614
75b7c240a feat(spec): master_detail + controlled_by_parent11332
07ad42463 fix(cli): os meta resync skip-count explanation220
7a537ce90 feat(spec): strict top-level stack keys11313
445ae4deb fix(auth): auth emails follow the deployment locale133
30b1c636a feat(spec): register 9 REST wire codes1134
650cd3daa fix(objectql): delete-cascade registry reads140
3851f87f0 feat(spec,plugin-security): partial field masking11619
d5156b965 refactor(metadata-protocol): drop dead objects tolerances44

"One PR is not a rate" was the card's own caveat, so this is the answer to it. The #9191 row reads 4 against the card's "three pages" because docs is the full set and the comment partitions releases/v9.mdx into its own read-only section (#6893): 3 editable + 1 release-owned.

On #9191 the change is qualitative, not just smaller — all three previously-listed pages are gone and the two the card measured as missing are back, each carrying the anchor that put it there:

content/docs/api/client-sdk.mdx (via getAudit (sdk), getReferences (sdk), meta.getAudit (sdk), meta.getReferences (sdk))
content/docs/kernel/contracts/metadata-service.mdx (via getHistory (sdk))
content/docs/plugins/adding-a-metadata-type.mdx (via /:type/:name/history (route))

The two zeroes are the honest shape of the trade, not a bug. 07ad42463 derives MetaResync and resyncSkipExplanationLine and no hand-written page names either, so the run says so and points at the coarse set — where the old tool's 22 rows were every page mentioning @objectstack/cli. A CLI command name (os meta resync) is exactly the recall class the shape guard costs us: it is a lowercase word, so it cannot anchor. Stated rather than hidden.

Cost, which the card flagged as never measured

The derivation reads the same 178-page corpus the old one did, plus the 18 route-registrar/ledger sources (~875 KB) and one git show per changed file per side. End to end on the ten PRs above: 85-195 ms before, 114-582 ms after; best-of-3 on the widest diff, 92 ms → 429 ms. Every case stays well under a second against a job that already spends seconds checking out the repo and setting up Node. It is the right default for every PR.

Verification

  • node scripts/docs-audit/check-affected-docs.mjs — the dispatch-gates discovers no family for a workflow verification step whose script is not named check-*docs-drift-check.yml runs one on packages/** and the tool cannot mention it in any bucket #9187 wrapper is kept and extended, not worked around: 115 self-test cases pass, up from 56. The new pins cover the rank rule (a method body anchors on the method, a local does not), container detection, statement heads, the shape guard, route-tail acceptance and matching, -U0 hunk parsing, the registrar and ledger parsers, and an end-to-end bridge assertion over the three fixtures.
  • Reverse verification, from the committed state, three ablations, each in the expected red direction:
    • drop the rank rule (attribute to the innermost declaration) → 2 cases red, the method-vs-class attribution collapses;
    • make function a container → 2 cases red, draftBuffer becomes an anchor;
    • neutralise the shape guard → 4 cases red, and live on 3851f87f0 the list goes 19 → 49 pages.
      Restored each time with git checkout of the branch path; git status clean afterwards.
  • Gate union re-run after the final commit, at 3cb2c9eb8, all green: check:docs-audit-scope, check:node-version, check:required-contexts, check:shard-attestation, check:workflow-status-functions, node scripts/check-shard-attestation.mjs, node scripts/docs-audit/check-affected-docs.mjs, node scripts/check-nul-bytes.mjs. Derived with node scripts/pm/dispatch-gates.mjs against the actual changed paths.
  • The workflow's inline renderer was syntax-checked and driven against real --json output for three shapes (rows, zero-with-anchors, capped-with-limits).

Scope

scripts/** and .github/workflows/** publish nothing ⇒ no changeset; skip-changeset applied. No cross-tool shared rule and no AGENTS.md / dispatch-brief change — the grading scoped this card to this tool's derivation. content/docs/releases/ untouched; the release-owned partition (#4920 / #6893) and --all behaviour are unchanged, and check-audit-scope.mjs, which consumes --all --json, still passes.


Generated by Claude Code

…ent, not package edges
The advisory resolved pages by PACKAGE DEPENDENCY -- "which hand-written docs
reference @objectstack/x". That is a dependency-graph proxy answering a semantic
question, and #9192 measured it wrong in BOTH directions on PR #9191 (the three
metadata read verbs): 3 pages listed of which 1 was relevant, while the 2 pages
that actually document the changed surface were absent, because they document it
through the SDK -- which does not depend on the implementing package at all.
A page is now listed when it NAMES something the change touched. Three anchor
kinds: the documentable declaration enclosing each changed line (top-level, or a
member of a top-level container -- a local inside a function body is not surface,
which is what drops the measured `singular` false positive); the wire routes a
changed symbol's registrar handler serves; and the client method the route
ledgers bind to those routes, which is the hop that carries the derivation across
the surface boundary the package graph cannot cross.
Two guards keep it precision-first, both running BEFORE the route bridge so a
weak name cannot mint route anchors from every handler that mentions it: a shape
guard (a single lowercase word like `label` or `object` matched 82 and 113 of 178
pages) and a corpus-share guard (`ObjectQL` is code-shaped and genuinely changed,
and names 59 of 178 pages). Everything either guard removes is published, as is
every changed file that yielded no anchor -- silence from this tool must never be
readable as absence, which is the whole subject of the card.
Measured before/after across ten real PRs; the PR comment now also states what
the run could not see, at the point of use.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y26DJEHSBhhAQ6wwfsHNza
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cddocumentationImprovements or additions to documentationsize/xlskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants

@os-project-manager@claude