diff --git a/scripts/ablation-dist-preflight.mjs b/scripts/ablation-dist-preflight.mjs index 510bb6e220..17eed42b4e 100644 --- a/scripts/ablation-dist-preflight.mjs +++ b/scripts/ablation-dist-preflight.mjs @@ -10,11 +10,29 @@ // // ## The failure this exists to stop // -// `packages/qa/dogfood` resolves the code under test from each package's built -// `dist/`, not `src/` -- deliberately, because that is what covers packaging and -// export-surface defects. Editing `src` therefore has no effect on the suite -// until that package is rebuilt, and the two directions of forgetting are NOT -// equally dangerous: +// The hazard is a property of RESOLUTION, not of any one suite. Its true +// condition is: **any test whose subject resolves through the dependency's +// `exports`** -- which point at that package's built `dist/`, not `src/` -- +// with no vitest alias redirecting the specifier back to source. Such a test's +// verdict is a function of BUILD state, so editing `src` has no effect on it +// until that package is rebuilt. +// +// That set is enumerable, and already enumerated: `KNOWN_UNALIASED_TEST_IMPORTS` +// in `scripts/check-test-source-alias.mjs` is the measured, shrink-only ledger +// of exactly those package-dependency pairs -- 61 packages, 305 pairs when this +// paragraph was written. Every one of them carries the failure below. +// +// `packages/qa/dogfood` is the most familiar instance -- it consumes `dist/` +// deliberately, because that is what covers packaging and export-surface +// defects -- but it is an INSTANCE, not the definition. Reading the hazard as +// dogfood-only is how an ablation gets trusted in a plain unit suite where it +// proves nothing: measured in `plugin-email` -> `platform-objects` (375 passes +// on an ablated field; 4 of them went red once `platform-objects` was rebuilt), +// and again in `plugin-auth` -> `core`. Neither package aliases the dep it +// mutated; both are ordinary ledger entries, and `plugin-email` has no +// `vitest.config.*` at all. +// +// The two directions of forgetting are NOT equally dangerous: // // forgot to rebuild a FIX -> false RED. Costs a lap, and gets noticed. // forgot to rebuild an ABLATION -> false GREEN. Silently certifies a vacuous @@ -30,6 +48,17 @@ // by hand-grepping `dist/` for the mutation marker. This script is that grep, // mechanized, with the traps the manual version cannot see. // +// ### Worse than uninformative -- it points confidently the WRONG WAY +// +// When the ablation's purpose was to prove a NEW GATE is capable of failing, +// the false green is not a null result. It is the observation "the gate did not +// fire", and in that context that reads as evidence the GATE is broken rather +// than evidence the harness is. A dev acting on it goes hunting for a fix that +// is not needed -- or, the expensive outcome, WEAKENS a working gate until it +// "fires", destroying the thing the ablation was written to certify. That is +// the `plugin-auth` -> `core` shape above: ablation legs whose entire purpose +// was to demonstrate a new gate can fail, every one of them dist-mediated. +// // ## The two ablation shapes, hence the two modes // // PLANT (default) the mutation ADDS something identifiable -- a changed @@ -39,10 +68,20 @@ // so the assertion inverts: a literal unique to the deleted // code must be GONE from dist. Same check, mirrored. // -// `--absent` is also the restore leg: after putting the fix back and rebuilding, -// it proves the marker really left the artifact. That leg matters more than it -// looks -- a marker left behind in `dist/` keeps mutated code live for every -// later suite run in that worktree, long after the ablation is "finished". +// The rule they mechanize has TWO halves, and only one of them is intuitive: +// +// mutate -> rebuild -> prove the marker is IN dist/ (default mode) +// restore -> rebuild -> prove the marker is GONE (`--absent`) +// +// So `--absent` is two things at once: the mode for a DELETE ablation, and the +// restore leg of a PLANT one. The restore half is the one that gets skipped -- +// rebuilding after mutating is obvious, while remembering that RESTORING also +// needs a rebuild before the NEXT measurement is trustworthy is not. It matters +// more than it looks: a marker left behind in `dist/` keeps mutated code live +// for every later suite run in that worktree, long after the ablation is +// "finished", so the runs that follow are measuring the wrong tree. Done by +// hand the leg reads `grep -c packages/core/dist/index.js` -> 0 after +// the `git checkout`; this script is that, plus the traps below. // // ## Sourcemap-only matches are RED, not green // @@ -51,14 +90,49 @@ // false green this script exists to prevent, so `.map` hits are reported and // excluded from the verdict. // +// ## The scan is WHOLE-PACKAGE, so the marker must be unique to the mutation +// +// Both modes grep the package's entire `dist/` tree, deliberately -- a mutation +// can land in any emitted chunk, and guessing which one is how the manual +// version missed things. The price is that the scan cannot tell YOUR literal +// from the same literal written elsewhere in the same package, and both modes +// assume it is unique. When it is not, both go wrong, in opposite directions: +// `--absent` reports surviving hits that were never yours (a false RED), and +// the default mode passes on a sibling's hit alone (a false GREEN -- the one +// this script exists to stop, reintroduced through the marker). +// +// Measured: ablating `internal: true` on ONE field of `sys_email`, then running +// +// node scripts/ablation-dist-preflight.mjs @objectstack/platform-objects 'internal: true' --absent +// +// reported 6 surviving hits, every one of them a legitimate `internal: true` on +// an identity object in `dist/identity/` and none of them the ablated field. +// +// There is no per-symbol mode. When the marker cannot be made unique -- a +// per-FIELD ablation of a flag the package also uses elsewhere is the standard +// case -- do NOT weaken the marker to make this script agree with you. Verify +// by PROPERTY READ against the same artifact instead, which is exact where a +// substring scan cannot be: +// +// node -e "const {SysEmail}=require('./packages/platform-objects/dist/audit/index.js'); +// console.log(SysEmail.fields.headers_json.internal)" +// # before rebuild: true (ablation NOT in the artifact -> the green was vacuous) +// # after rebuild: undefined +// +// Same question, same moment in the procedure, same two halves (mutate and +// restore); only the instrument changes. What is not negotiable is that SOME +// instrument reads the built artifact before the ablation's colour is believed. +// // ## Why this is not a `check:*` gate // // It judges a deliberately mutated working tree, so it can only be run by the // agent performing the ablation, at one specific moment between "mutate" and // "run the suite". CI has no ablation in flight and nothing to assert. It is // dev-side agent tooling, invoked from the ablation procedure in -// `.claude/agents/os-dev.md` and `.claude/skills/dogfood-verification/SKILL.md` -// -- keep those two and this file's usage line in step. +// `.claude/agents/os-dev.md`, `.claude/skills/dogfood-verification/SKILL.md` +// and `packages/qa/dogfood/README.md` step 4 -- keep those three and this +// file's usage line in step. (Those three state the procedure scoped to the +// dogfood suite; the true condition is the resolution one stated at the top.) // // Anything this script cannot see is RED, never a skip: a missing `dist/`, a // `dist/` with nothing readable in it, or a package name that resolves to