Skip to content

fix(spec): scan planned and experimental evidence, not live alone#13081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
os-trump merged 1 commit into from
Aug 29, 2026
Merged

fix(spec): scan planned and experimental evidence, not live alone #13081

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 95 additions & 20 deletions packages/spec/scripts/liveness/check-liveness.mts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -41,11 +41,15 @@
// (see proof-registry.mts), a `live` classification MUST carry a valid proof —
// the file must exist and declare the `@proof: <id>` tag. CI fails otherwise.
//
// EVIDENCE POINTERS (ADR-0087): a `live` verdict IS its evidence pointer — "this
// property has a runtime consumer, here it is". A cited path that is repo-rooted
// EVIDENCE POINTERS (ADR-0087): a verdict IS its evidence pointer — "this
// property has a runtime consumer, here it is", and for a refusal, "here is the
// code that rejects it". A cited path that is repo-rooted
// and attributed to THIS repo must resolve against this checkout, or CI fails:
// an unresolvable pointer makes the claim unfalsifiable, and a directory move or
// a rename is all it takes (see evidence.mts). Cross-repo attribution
// a rename is all it takes (see evidence.mts). Which STATUSES get scanned is a
// population decision with its own history — see EVIDENCE_SCANNED_STATUSES
// below, where the `planned`/`experimental` widening and the `dead` exclusion
// are argued from measurements. Cross-repo attribution
// (`objectui: …`, `cloud: …`, `packages/services/service-ai/…`) is counted, not
// resolved — those files are legitimately absent here. This was a ⚠ until #5623,
// for one historical reason: the pre-#3857 `evidence.split(':')[0]` parser
Expand DownExpand Up@@ -182,6 +186,7 @@ import {
STATE_COUNTS_FILE,
STATE_COUNTS_GUIDANCE,
STATE_COUNTS_PATH,
STATUS_COLUMNS,
foldStateCounts,
parseStateTable,
reconcileReadmeTable,
Expand DownExpand Up@@ -332,6 +337,69 @@ function markerStatus(d: string): string | null {
return null;
}

// ── WHICH STATUSES' `evidence` IS SCANNED (#13041) ──
//
// The four evidence checks — existence, line bound, symbol anchor, key mention —
// ran only when `status === 'live'`, on the reading that a `live` verdict IS its
// evidence pointer. `producer` was already scanned at ANY status, and that
// asymmetry is what made this visible: an entry whose whole content is a REFUSAL
// carried evidence the census COUNTED and no check READ.
//
// The measured instance. `api.json`'s `inputMapping.transform` and
// `outputMapping.transform` are `planned` — parsed, then loudly rejected — and
// #13039 migrated both citations to `path#symbol` anchors precisely because the
// refusal disappearing is what should go red. Under the `live`-only population
// those anchors were verified by NOTHING: renaming or deleting
// `mappingDeclarationRejection` changed no check's verdict, while the summary
// line still counted the pointers. "Counted" and "verified" had come apart,
// which is the whole failure this ledger exists to remove.
//
// So the scan reads every status whose evidence is a POINTER AT CODE, whatever
// verdict that pointer supports: `live` (a consumer reads the key), `planned`
// (a refuser rejects it), `experimental` (declared, not enforced).
const EVIDENCE_SCANNED_STATUSES = new Set<string>(['live', 'planned', 'experimental']);

// `dead` is OUT, and this is that boundary written into the code rather than
// left in a PR residual — the half of #13041 that is worth doing whichever way
// the population question went.
//
// A dead row's pointer does not live in `evidence`. Measured across every ledger
// on the commit that widened this scan: all 80 `dead` entries carry a `note`,
// and only 6 carry an `evidence` string at all — the retirement story (which
// sweep, which ADR, which tombstone rejects the key now) is prose in `note`,
// which no check scans, by design. Scanning `dead.evidence` would therefore hold
// 6 rows to a standard while reading nothing of the other 74 and publishing the
// result as coverage of the class: the same shape as the defect above, one
// status over. If dead rows ever cite code as a convention rather than as a
// remnant, move `dead` across this line — and give `note` a scan of its own,
// because that is the check this class actually needs.
const EVIDENCE_UNSCANNED_STATUSES = new Set<string>(['dead']);

// A status in NEITHER set is #13041 re-armed: its evidence would be counted by
// the census and read by nothing, silently, exactly as `planned`'s was. So the
// partition is held to the published vocabulary rather than to a reader's
// memory — a fifth status forces the decision instead of defaulting it to
// unread. (`STATUS_COLUMNS` is that vocabulary: the columns the generated count
// artifact publishes.)
for (const s of STATUS_COLUMNS) {
const scanned = EVIDENCE_SCANNED_STATUSES.has(s);
const unscanned = EVIDENCE_UNSCANNED_STATUSES.has(s);
if (scanned === unscanned) {
throw new Error(
`liveness status '${s}' is in ${scanned ? 'BOTH' : 'NEITHER'} evidence-scan set — every status in ` +
'STATUS_COLUMNS must be declared scanned or explicitly unscanned (#13041: a status in neither ' +
'has its evidence counted by the census and verified by nothing).',
);
}
}

/**
* The scanned population, rendered for the gate's own output. Derived from the
* set rather than written out again, so the numbers and the population they
* describe cannot drift apart — which is the class of bug this whole file is.
*/
const EVIDENCE_SCANNED_LABEL = [...EVIDENCE_SCANNED_STATUSES].map((s) => `'${s}'`).join(' / ');

// ---- Zod schema walking (version-tolerant: prefer _zod.def, fall back to _def) ----
function defOf(s: any): any {
return s && (s._zod?.def ?? s._def);
Expand DownExpand Up@@ -459,8 +527,10 @@ const report: any = {
producers: null as ProducerReport | null, // `producer` / `evidenceScope` — the #4837 / #4895 worklists
producerMissing: [] as string[], // a `producer` pointer into thin air — FAILS, like a rotted `evidence`
// The three evidence counters, and the distinction between the first two is
// the whole point: `evidenceLocal` is how many repo-rooted paths `live` entries
// DECLARE, `evidenceMissing` is how many of those do not exist here. The
// the whole point: `evidenceLocal` is how many repo-rooted paths the SCANNED
// statuses DECLARE (EVIDENCE_SCANNED_STATUSES — see there for why `planned`
// and `experimental` joined `live` and why `dead` did not), `evidenceMissing`
// is how many of those do not exist here. The
// summary line used to print `evidenceLocal` under the word "resolved", so
// breaking five pointers left the count at 330 and the run still said
// "330 resolved" (#5623). Count and word now agree.
Expand All@@ -478,7 +548,7 @@ const report: any = {
// moves WITHIN the cited file satisfies both while pointing at nothing. This
// asks the complementary question: does the cited file name the property at
// all, in any of its camelCase/snake_case spellings?
keyMentionsChecked: 0, // resolvable (live entry, cited local file) pairs asked
keyMentionsChecked: 0, // resolvable (scanned-status entry, cited local file) pairs asked
keyMentionExempt: 0, // ...of which this many are recorded in the shrink-only baseline
keyMentionUnanchored: [] as string[], // ...and this many are NOT — FAILS the gate
keyMentionStale: [] as string[], // a baseline row whose pair now anchors — also FAILS
Expand DownExpand Up@@ -585,7 +655,7 @@ function classify(type: string, path: string, status: string, led: any, cat: any
collectOutOfRange(pv, `${type}/${path} [producer]`);
collectAnchorFindings(pv, `${type}/${path} [producer]`);
}
if (status === 'live' && led?.evidence) {
if (EVIDENCE_SCANNED_STATUSES.has(status) && led?.evidence) {
// Extract every repo-rooted path the evidence claims and resolve the ones
// attributed to THIS repo. Cross-repo pointers (objectui / cloud) are
// counted, not resolved — see evidence.mts for why the old
Expand DownExpand Up@@ -865,7 +935,7 @@ const totalProofFailures = report.proofErrors.length + report.proofMissing.lengt
const failed =
totalUnclassified > 0 ||
totalProofFailures > 0 ||
// A `live` entry whose repo-local evidence path is gone. Red since #5623 — the
// A scanned-status entry whose repo-local evidence path is gone. Red since #5623 — the
// ⚠ it replaces was calibrated for the false-positive era, not for the parser
// that now resolves 330 paths and reports zero. Cross-repo attribution never
// reaches this list: checkEvidence only resolves the LOCAL bucket.
Expand DownExpand Up@@ -927,7 +997,7 @@ if (asJson) {
// "extracts nothing"); "resolved" is the verdict. They are equal on a green
// run, which is exactly why printing only the first read as a pass.
console.log(
`\nevidence paths: ${report.evidenceLocal} repo-local path(s) declared by 'live' entries, ` +
`\nevidence paths: ${report.evidenceLocal} repo-local path(s) declared by ${EVIDENCE_SCANNED_LABEL} entries, ` +
`${report.evidenceLocal - report.evidenceMissing} resolved against this checkout` +
(report.evidenceMissing ? `, ${report.evidenceMissing} MISSING` : '') +
`; ${report.evidenceForeign} attributed to another repo (objectui / cloud — not resolvable here).`,
Expand DownExpand Up@@ -1019,7 +1089,7 @@ if (asJson) {
(report.keyMentionUnanchored.length ? `, ${report.keyMentionUnanchored.length} UNANCHORED` : '') + '.',
);
if (report.keyMentionUnanchored.length) {
console.log(`\n✗ ${report.keyMentionUnanchored.length} 'live' citation(s) name a file that never names the property:`);
console.log(`\n✗ ${report.keyMentionUnanchored.length} ${EVIDENCE_SCANNED_LABEL} citation(s) name a file that never names the property:`);
report.keyMentionUnanchored.forEach((s: string) => console.log(` ${s}`));
console.log('\n' + KEY_MENTION_GUIDANCE.split('\n').map((l) => (l ? ` ${l}` : '')).join('\n'));
}
Expand All@@ -1034,21 +1104,26 @@ if (asJson) {
);
}
if (report.staleEvidence.length) {
console.log(`\n✗ ${report.staleEvidence.length} 'live' entr(ies) cite a file that is missing from THIS repo:`);
console.log(`\n✗ ${report.staleEvidence.length} ${EVIDENCE_SCANNED_LABEL} entr(ies) cite a file that is missing from THIS repo:`);
report.staleEvidence.forEach((s: string) => console.log(` ${s}`));
console.log(
'\n A `live` verdict IS its evidence pointer — "this property has a runtime consumer,\n' +
' here it is". When the cited file is gone from this checkout the claim is no longer\n' +
' falsifiable: declared, but nothing enforces that anything still reads the property,\n' +
'\n A verdict IS its evidence pointer — "this property has a runtime consumer,\n' +
' here it is", or for a refusal ("planned" / "experimental") "here is the code that\n' +
' rejects it". When the cited file is gone from this checkout the claim is no longer\n' +
' falsifiable: declared, but nothing enforces that anything still reads the property\n' +
' — or still refuses it —\n' +
' and a directory move or a rename is the whole cost of getting there.\n\n' +
' Three repairs, and picking the right one is the work:\n' +
' • the consumer MOVED inside this repo → repoint the path, and stamp `verifiedAt`\n' +
' while you have the call graph open;\n' +
' • the consumer moved to ANOTHER repo → say so with a realm marker\n' +
' • the consumer (or the refuser) MOVED inside this repo → repoint the path, and\n' +
' stamp `verifiedAt` while you have the call graph open;\n' +
' • it moved to ANOTHER repo → say so with a realm marker\n' +
' (`objectui: packages/app-shell/…`, `cloud: …`). Attributed paths are counted,\n' +
' never resolved, and never fail here — that boundary is deliberate;\n' +
' • the consumer is GONE → the verdict is not `live` any more. Re-classify under\n' +
' ADR-0049 enforce-or-remove instead of repointing at a plausible survivor.\n\n' +
' • it is GONE → the verdict no longer holds. A `live` entry whose consumer left\n' +
' re-classifies under ADR-0049 enforce-or-remove; a `planned` / `experimental`\n' +
' entry whose REFUSER left is the louder case — the key is now accepted in\n' +
' silence, and the entry is either wrong or the refusal needs restoring. Either\n' +
' way, do not repoint at a plausible survivor.\n\n' +
' This was a ⚠ until #5623 for one reason: the pre-#3857 parser took\n' +
' `evidence.split(":")[0]` as the filename, flagged 48 of 227 entries and every one\n' +
' was a false positive — so it could not fail the build, and the one real rot it was\n' +
Expand DownExpand Up@@ -1307,7 +1382,7 @@ if (asJson) {
console.log(
'\n✓ every governed-type property at the walk\'s one-level granularity is classified, every ' +
'registered type is governed or explicitly pending, no ledger row outlives its property, ' +
'every container inheritance is declared, every `live` entry\'s repo-local evidence path ' +
`every container inheritance is declared, every ${EVIDENCE_SCANNED_LABEL} entry's repo-local evidence path ` +
'resolves, every `path:NNN` citation names a line that file actually has, every ' +
'`path#symbol` anchor names a symbol its file contains, and every cited ' +
'file names the property it is evidence for (or is a recorded exemption), all bound ' +
Expand Down
Loading
Loading