Skip to content

fix(docs-audit): say NOT COVERED in the drift headline when a changed file yields no anchor#11798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
os-steve merged 1 commit into from
Aug 24, 2026
+388 −7
Merged

fix(docs-audit): say NOT COVERED in the drift headline when a changed file yields no anchor #11798

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 65 additions & 4 deletions .github/workflows/docs-drift-check.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -18,6 +18,13 @@ name: Docs Drift Check
# package. The comment now also states what the run could NOT see — the derivation being
# read past its precision, with its silence taken for absence, is what #9192 records.
#
# Since #11357 the same posture reaches the HEADLINE. A README exports no symbol, mounts
# no route and declares no SDK method, so a README-only diff derives no anchor at all and
# the comment answered it with "this run has no opinion" — while the file it could not
# anchor was named only inside the collapsed fold. Two real README defects landed that way
# on one day (#11180, #11262). The fold was right and stays; what changed is that a run
# with an unanchored file now says so where a reviewer reads it, and withholds its ✅.
#
# And since #9519 it states WHICH TREE it read. On `pull_request`, actions/checkout gives
# the MERGE of base and head, so the row set is a fact about a commit that exists on no
# branch a reader can name — while the comment's own re-derivation command sent them to
Expand DownExpand Up@@ -78,6 +85,16 @@ jobs:
- name: Self-test the change → docs mapper
run: node scripts/docs-audit/check-affected-docs.mjs

# The advisory's own TEXT, pinned by rendering it (#11357). The step above answers
# "what did this run find"; this one answers "what does the comment SAY about what
# it did NOT find" — a question no other gate asks, because that wording lives in
# the inline `github-script` block below, which nothing imports and nothing
# type-checks. It runs THAT block against fixture diffs, so a blind-spot notice that
# stopped rendering — or started rendering unconditionally, which is the same defect
# wearing the opposite mask — fails here rather than on a reader.
- name: Self-test the drift comment's wording
run: node scripts/docs-audit/check-drift-comment.mjs

- name: Compute affected docs
id: affected
run: |
Expand DownExpand Up@@ -199,8 +216,39 @@ jobs:
lines.push('</details>');
return lines;
})();
// ── WHAT THIS RUN COULD NOT COVER BELONGS IN THE HEADLINE (#11357) ────
//
// A README carries no `@docs-rule` block, exports no symbol, mounts no route
// and declares no SDK method, so a README-only diff derives NOTHING and lands
// in the zero-anchor arm below. Two such defects landed on one day (#11180 —
// `packages/cli/README.md` advertising `os studio`, a command the CLI does not
// ship; #11262 — `packages/console/README.md` asserting a `@object-ui/console`
// fallback the code no longer performs), and neither was detectable by this
// check on any run. A README IS documentation — the published front page of a
// package on npm — so this is a hole, not a tolerable exclusion.
//
// The run was already HONEST about it: the `limits` line below names the file
// it could not anchor, and that half is correct. But `limits` renders inside a
// <details> fold, which GitHub renders COLLAPSED, under a headline reading
// "**this run has no opinion** about the docs" — so a reviewer who does not
// expand the fold reads the whole comment as "nothing to check". That is #9192
// one surface up: the tool's silence taken for absence, this time by a reader
// of its own headline.
//
// So the blind spot is stated where it cannot be missed, and the fold keeps the
// detail. ⛔ NOTHING about anchor derivation changes here — this run finds
// exactly what it found before, and every verdict is byte-identical whenever
// `anchorless` is empty. The rendering is pinned both directions by
// `scripts/docs-audit/check-drift-comment.mjs`, which runs THIS script against
// fixture diffs; unconditional text would pass a one-direction pin.
const anchorlessNames = `\`${anchorless.slice(0, 3).join('`, `')}\`${anchorless.length > 3 ? ', …' : ''}`;
const notCovered = anchorless.length
? `⚠️ **${anchorless.length}** changed file(s) yielded no anchor (${anchorlessNames}), so the pages documenting them are **NOT COVERED by this run** — this is not a clean bill of health for those files.`
: '';
const limits = [];
if (anchorless.length) limits.push(`**${anchorless.length}** changed file(s) yielded no anchor (\`${anchorless.slice(0, 3).join('`, `')}\`${anchorless.length > 3 ? ', …' : ''}) — pages documenting those are invisible to this run`);
// Same names, one derivation: the headline and this line must never be able to
// disagree about which files went unseen.
if (anchorless.length) limits.push(`**${anchorless.length}** changed file(s) yielded no anchor (${anchorlessNames}) — pages documenting those are invisible to this run`);
if (crossCutting.length) limits.push(`**${crossCutting.length}** cross-cutting symbol(s) contributed no route anchor: \`${crossCutting.join('`, `')}\``);
if (overbroad.length) limits.push(`**${overbroad.length}** anchor(s) matched too much of the corpus to be a work list: \`${overbroad.join('`, `')}\``);
if (weak.length) limits.push(`**${weak.length}** name(s) were too generic to anchor anything (single lowercase words)`);
Expand DownExpand Up@@ -248,9 +296,18 @@ jobs:
if (docs.length === 0) {
// "Nothing found" and "I derived nothing to look for" are DIFFERENT results
// and must never render alike — that conflation is #9192's own subject.
// Since #11357 a THIRD result is separated from both: "part of this diff was
// never looked at". `anchorless.length` is what tells it apart, and it is
// already known here, at headline-composition time.
//
// ⛔ The ✅ is DROPPED whenever a file went unanchored, deliberately: a green
// tick is the clean-bill glyph, and "no page names the anchors I derived" is
// not a clean bill when some of the diff derived no anchor to name.
const headline = anchorList.length === 0
? `Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from **${pkgs.length}** changed package(s)), so **this run has no opinion** about the docs.`
: `**${anchorList.length}** anchor(s) derived from **${pkgs.length}** changed package(s); no hand-written page names any of them. ✅`;
? (anchorless.length
? `${notCovered} Nothing else in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from **${pkgs.length}** changed package(s)).`
: `Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from **${pkgs.length}** changed package(s)), so **this run has no opinion** about the docs.`)
: `**${anchorList.length}** anchor(s) derived from **${pkgs.length}** changed package(s); no hand-written page names any of them.${anchorless.length ? ` ${notCovered}` : ' ✅'}`;
// The tree identity is rendered here only when anchors WERE derived: that
// is the run with an opinion ("no page names any of them"), and a reader
// whose own tree does yield a row is owed the reason. With no anchors the
Expand All@@ -264,7 +321,11 @@ jobs:
body = [
marker,
'### 📓 Docs Drift Check',
`This PR changes **${pkgs.length}** package(s): ${pkgs.map(p => `\`${p}\``).join(', ')}, touching **${anchorList.length}** documentable anchor(s).`,
// The row list below is a work list, and a work list that silently omits
// part of the diff is read as the whole of it — so the same sentence rides
// this headline too (#11357). It is the same conditional: with nothing
// unanchored, this line is byte-identical to what it always was.
`This PR changes **${pkgs.length}** package(s): ${pkgs.map(p => `\`${p}\``).join(', ')}, touching **${anchorList.length}** documentable anchor(s).${anchorless.length ? ` ${notCovered}` : ''}`,
];
if (capped) {
// NO row list above the cap — folding rows behind a details tag would
Expand Down
38 changes: 35 additions & 3 deletions scripts/docs-audit/README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -203,9 +203,11 @@ means the list is incomplete **by a known amount**, and an empty `docs` beside i
never be read as "no page documents this change". The superseded coarse set is still
computed and emitted as `packageMentionDocs`, labelled — an audit that deliberately wants
the wide net can still ask for it, and keeping it visible is how a reader tells a *narrow*
list from a *blind* one. The PR comment renders all of this in a collapsed section, because
the failure #9192 records was never the tool lying — it was the tool never signalling its
own limits at the point of use.
list from a *blind* one. The PR comment renders all of this in a collapsed section — and,
since #11357, renders the anchorless count in the **headline** as well, because a limit a
reader has to expand a fold to find is one a reader does not find. The failure #9192
records was never the tool lying; it was the tool never signalling its own limits at the
point of use.

### Measured, before and after

Expand DownExpand Up@@ -423,6 +425,36 @@ the point-of-use half of #9192 — every one of the three derived-list failures
shift was caught only because a dev widened the probe past what the tool offered, never
because the tool signalled its own limits where it was read.

### The headline says what the run did not cover (#11357)

A README carries no `@docs-rule` block, exports no symbol, mounts no route and declares
no SDK method, so a README-only diff derives **nothing** — and the comment answered it
with *"this run has no opinion about the docs"*, which a reviewer reads as *"nothing to
check"*. The file it could not anchor **was** named, honestly, in the collapsed section
above; GitHub renders that section shut. Two real README defects landed inside that gap
on one day: #11180 (`packages/cli/README.md` advertising `os studio`, a command the CLI
does not ship) and #11262 (`packages/console/README.md` asserting a `@object-ui/console`
fallback the code no longer performs). Neither was detectable by this check on any run.

So whenever `anchorlessChanges` is non-empty the headline itself now names the count and
the files, says those pages are **not covered by this run**, and **withholds the ✅** —
a green tick is the clean-bill glyph and a partial look is not a clean bill. Anchor
derivation is untouched: with nothing unanchored, every headline is byte-identical to
what it was, and no run's verdict moves either way (this check never fails a build).

⚠️ This is #9282's **option 4**, not its option 3. Falling back to the coarse
package-mention set for an anchorless file was measured 0-for-3 on recall on its own
specimen (see [§1](#1-affected-docsmjs--change--docs-mapping-the-linchpin)), and would
regrow the ~106–112-row lists #6893 / #7009 measured and readers learned to skip.
Reporting the gap costs nothing and hides nothing; buying coverage with noise does both.

`check-drift-comment.mjs` pins it **in both directions** — a fixture diff touching only a
README must produce the not-covered wording, and one touching an anchorable source file
must not. It runs the workflow's own inline `github-script` block against real
`affected-docs.mjs` output from throwaway git repos, because a source grep passes just as
happily on text that never renders and on text that renders unconditionally, and neither
is a report. Zero dependencies, like the mapper: this job never runs `pnpm install`.

### The comment forks release-owned pages into a read-only section (#6893)

Same ruling as [1b](#release-owned-pages-are-in-scope-and-read-only-4920), one level
Expand Down
Loading
Loading