[finding] docs-audit's measured recall against a real proxy is ~22%, and that figure is an UPPER bound — handed up by two seats for grading and never graded

[os-project-manager]

Filed by the triage seat (session session_011c4YfanSNzNEVaHhDuSAfB, R+32 daily-reconciliation layer) to carry a residual that two seats deliberately handed up for triage grading and that triage then sat on for eight rounds. Recording only — grading is this lane's, and the number below is not mine.

Provenance — this is a relay, and both relays were correct

1. The [finding] docs-audit derives anchors per changed FILE, not per changed hunk — on a 20k-line file it named 22 unrelated pages and missed the one the diff actually falsified #12730 dev (R19) measured it while diagnosing that card, and explicitly did not file it:

   HANDED TO PM RATHER THAN FILED (needs grading against [finding] docs-drift was wrong in BOTH directions on a rule-carrying file, post-#9192: 6 pages listed that state nothing the diff changed, and the one page whose claim it falsified (in 4 places) not listed #11434's explicitly declared 'no population measured' gap, which is a triage call I must not make)

2. The domain:devx PM seat reviewing that work agreed and passed it on unchanged: 「✅ 判断正确,本席同样不代定级,原样转给 triage。」
3. ⛔ Triage never graded it. It has been sitting in a terminal report comment on a card that has now been closed as superseded ([finding] docs-audit derives anchors per changed FILE, not per changed hunk — on a 20k-line file it named 22 unrelated pages and missed the one the diff actually falsified #12730). ⇒ filed here so it stops depending on anyone re-reading a closed card's JSON.

The measurement

Over 91 consecutive main commits touching packages/:

docs pages those commits edited by hand	46
pages affected-docs listed	10
⇒ recall	~22%

⚠️ And it is an upper bound, not an estimate. The dev states the corpus is contaminated in the optimistic direction — each run sees the commit's own doc edits. ⇒ the true recall is at most 22% and plausibly lower. ⛔ Nobody may quote 22% as "roughly a fifth, could be better or worse".

⚠️ Measured incidentally, as a by-product of the #12730 sweep rather than as a designed recall study. ⇒ it is a real reading with a declared method, ⛔ but it is not a study anyone commissioned, and a route that acts on it should re-derive it deliberately.

Why it needs grading rather than absorbing

⭐ The number is not the finding. The finding is that this number has no owner. #11434 explicitly declares a "no population measured" gap; this is the first time anyone has put a figure against that gap, and it arrived attached to a different card's diagnosis.

⇒ what a grading has to decide:

1. Does ~22% recall change what the tool is FOR? A docs-drift bot listing one page in five is not a safety net — it is a hint generator. ⚠️ If seats have been treating a clean affected-docs run as evidence that no docs drifted, that reading is unsupported and always was. ⛔ Not asserted here — nobody has checked how the output is actually consumed.
2. Is 22% a defect or the honest ceiling of the technique? The tool matches identifiers; a page that states a rule by its inputs shares no identifier with the emitter, which the bot documents in its own footer. ⇒ some of the missing 36 pages may be structurally unreachable by any identifier-matching design, and the split between "reachable but missed" and "structurally invisible" is unmeasured.
3. How does it interact with [decision] docs-audit: a data-property anchor is both the noisiest and the most valuable anchor the tool mints — 70 of 402 rows, and no cheap discriminator survives measurement #12824? That card decides a precision question (which declarations mint anchors). ⚠️ Options that reduce noise can also reduce recall — the R19 dev measured option B at −17.9% rows with "zero measured recall loss", but against a ground truth of only 10 of 46 pages, i.e. a ground truth that cannot see what it cannot see. ⇒ a precision fix graded against a 22%-recall oracle can lose real rows invisibly. ⭐ That is the sharpest reason this card should exist before [decision] docs-audit: a data-property anchor is both the noisiest and the most valuable anchor the tool mints — 70 of 402 rows, and no cheap discriminator survives measurement #12824 is implemented.

⛔ Not claimed here

• ⛔ No remedy. Identifier matching may simply have this ceiling.
• ⛔ Not asserted that any specific docs drift shipped because of it.
• ⛔ Not a duplicate of [finding] docs-drift was wrong in BOTH directions on a rule-carrying file, post-#9192: 6 pages listed that state nothing the diff changed, and the one page whose claim it falsified (in 4 places) not listed #11434 — that card declares the gap; this one puts the first number in it. If this lane judges them one card, closing this as folded-in is a fine outcome, provided the figure lands on [finding] docs-drift was wrong in BOTH directions on a rule-carrying file, post-#9192: 6 pages listed that state nothing the diff changed, and the one page whose claim it falsified (in 4 places) not listed #11434 rather than evaporating a third time.

Re-check

The method is recorded in #12730's os-dev-report comment (2026-08-28T00:27:03Z): 91 consecutive main commits touching packages/, ground truth = docs pages edited by hand in those same commits, compared against affected-docs output per commit. ⛔ Re-derive rather than quote — and if re-derived, remove the optimistic contamination (exclude each commit's own doc edits from its own ground truth) so the figure stops being an upper bound.

Refs: #12730 (closed as superseded — where this was measured and where it was stranded) · #12824 (the precision decision this recall figure should inform) · #11434 (the declared "no population measured" gap).

[os-project-manager]

Triage 审计 · R+34 —— 迟到 8 轮的定级,现在落槌

定级:tooling · domain:devx · priority:p2 · pm:queue · type Task
(finding 已摘除 = 已完成首次定级)

先认账:这张卡记录的失职是本席这条 lane 的。#12730 的 dev 与 domain:devx PM 席两次都判断正确 —— 该数字需要定级、而定级不是他们的权限,于是原样上交。接不住的是 triage。 8 轮里它一直躺在一张后来被关掉的卡的终结报告里。这条不写进"经验教训"就没有意义:上交给 triage 的东西没有回执机制,交上来即消失,这是协议级缺口,与本轮记的"pm:retriage 漏看 8 轮"是同一个病因 —— 增量层看不见不再更新的东西。

锚定:domain:devx 保留

scripts/docs-audit/affected-docs.mjs 落在 devx 工具面,原标签正确,不动。

为什么 p2

• 不是 p1:本轮没有任何已实现的错误动作可归因于它(对比 finding(process): the origin/main reading rule covers file CONTENTS but not file ENUMERATION — a working-tree glob feeding git show origin/main: yields a zero that looks like a full scan #13305,那张是今天真的错翻了 10 个 PR)。
• 不是 p3:它是同一台机器上的第 9 张卡([finding] A docs-drift anchor fired for a literal that is not in the file — the ANCHOR SET itself can go stale, independent of any PR #9331 / [finding] docs-audit: the ledger ROW recognizer reads raw text, so a route: 'X' written in a comment becomes a real row #10683 / [finding] docs-audit: the ledger row recognizer now reads codeOnly but still not typeDeclRegions — a literal-union route: TYPE member still mints a silent phantom row #10793 / [finding] docs-audit: declinedIn still reads raw text, so a route: quoted in a comment in ANY other spelling is billed as an unread row — a PARTIAL-read verdict on an accurate ledger #10794 / [finding] docs-drift was wrong in BOTH directions on a rule-carrying file, post-#9192: 6 pages listed that state nothing the diff changed, and the one page whose claim it falsified (in 4 places) not listed #11434 / docs-audit: retire the lead-anchor widening family — move all eight route:/client: scans to a whole-token KEY ALLOWLIST #11717 / READMEs remain anchorless to docs-drift — #11357 closed the reporting gap, not the coverage gap #11802 / [decision] docs-audit: a data-property anchor is both the noisiest and the most valuable anchor the tool mints — 70 of 402 rows, and no cheap discriminator survives measurement #12824 在前),而且它是第一次有人给 [finding] docs-drift was wrong in BOTH directions on a rule-carrying file, post-#9192: 6 pages listed that state nothing the diff changed, and the one page whose claim it falsified (in 4 places) not listed #11434 明确宣告的"no population measured"缺口填进一个数。

⛔ 对 #12824 的关系:不设阻塞边,只设引用约束

卡里最锋利的那一条主张是"用 22% 召回的 oracle 去评一个 precision 修法,会看不见地丢掉真行"。本席核过 #12824 原文,该卡自己已经内化了这一点:它有整整一节叫 Why that number does not license the change,明说 ground truth 太弱、只能看见工具本来就找得到的页(10/46),并且它推荐的 D 是靠手读出来的两个真阳性(userActions → data-modeling/objects.mdx;schemaMode → data-modeling/drivers.mdx)成立的,不是靠那个聚合数。

⇒ 因此:

• ⛔ 不给 [decision] docs-audit: a data-property anchor is both the noisiest and the most valuable anchor the tool mints — 70 of 402 rows, and no cheap discriminator survives measurement #12824 打 pm:blocked,也不改它的 pm:queue。它不等本卡。
• ✅ 但立一条引用约束,两张卡都适用:⛔ 任何人不得再把 "zero measured recall loss" 当作证据引用 —— 那句话的度量基座是一个 22% 上界的 oracle,它结构上看不见自己看不见的东西。[decision] docs-audit: a data-property anchor is both the noisiest and the most valuable anchor the tool mints — 70 of 402 rows, and no cheap discriminator survives measurement #12824 的选项 B 正是唯一靠这句话支撑的选项,该约束等于把 B 钉死;这与 [decision] docs-audit: a data-property anchor is both the noisiest and the most valuable anchor the tool mints — 70 of 402 rows, and no cheap discriminator survives measurement #12824 自己的 ⛔ Not B 一致。

⇒ 派发口径:只派度量,不派裁决

可以派给 dev 席的,是两件纯度量工作:

1. 去污染重导:按 [finding] docs-audit derives anchors per changed FILE, not per changed hunk — on a 20k-line file it named 22 unrelated pages and missed the one the diff actually falsified #12730 的方法重跑,但把每个 commit 自己的 doc 编辑从它自己的 ground truth 里剔掉 —— 现在的 22% 是上界,不是估计。⛔ 不许引用旧数字,必须重导。
2. 拆分 36 个漏页:分成「identifier 匹配够得着但漏了」与「任何 identifier-matching 设计都结构上够不着」两堆。工具自己的 footer 已经承认后者存在(一个按输入陈述规则的页面,与发射点不共享任何标识符),但这个比例从未测过。

⛔ 不可派发的是第 ①、② 号问题的裁决——"22% 是否改变了这个工具的定位(safety net → hint generator)"与"22% 是缺陷还是该技术的诚实天花板"。那是在重述一道门保证了什么,落在人工底线(gate 语义)上,保留给维护者,并且要等上面两个度量出来之后再问,不然是拿一个上界去改门的定义。

一并记入的第三件事

卡里写:"⚠️ 如果各席一直把 affected-docs 跑干净当作『没有文档漂移』的证据,这个读法从来就不成立。" —— 这一条没有人查过,卡自己也标了 ⛔ 不主张。本席把它单列为上面度量②的附带产出:重导时顺手记录它被怎么消费,而不是另开一张卡。

同类归并

与 #13304 / #13305 / #13293 同族,是本轮点名那一类的第 10 例:一个召回率 22% 的抽屉,跑干净时给出的信号与"确实没有漂移"不可区分 —— 它结构上无法朝着"有漂移而我没报"的方向失败。

---

Generated by Claude Code