Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 85 additions & 11 deletions scripts/ablation-dist-preflight.mjs
Original file line numberDiff line numberDiff line change
Expand Up@@ -10,11 +10,29 @@
//
// ## The failure this exists to stop
//
// `packages/qa/dogfood` resolves the code under test from each package's built
// `dist/`, not `src/` -- deliberately, because that is what covers packaging and
// export-surface defects. Editing `src` therefore has no effect on the suite
// until that package is rebuilt, and the two directions of forgetting are NOT
// equally dangerous:
// The hazard is a property of RESOLUTION, not of any one suite. Its true
// condition is: **any test whose subject resolves through the dependency's
// `exports`** -- which point at that package's built `dist/`, not `src/` --
// with no vitest alias redirecting the specifier back to source. Such a test's
// verdict is a function of BUILD state, so editing `src` has no effect on it
// until that package is rebuilt.
//
// That set is enumerable, and already enumerated: `KNOWN_UNALIASED_TEST_IMPORTS`
// in `scripts/check-test-source-alias.mjs` is the measured, shrink-only ledger
// of exactly those package-dependency pairs -- 61 packages, 305 pairs when this
// paragraph was written. Every one of them carries the failure below.
//
// `packages/qa/dogfood` is the most familiar instance -- it consumes `dist/`
// deliberately, because that is what covers packaging and export-surface
// defects -- but it is an INSTANCE, not the definition. Reading the hazard as
// dogfood-only is how an ablation gets trusted in a plain unit suite where it
// proves nothing: measured in `plugin-email` -> `platform-objects` (375 passes
// on an ablated field; 4 of them went red once `platform-objects` was rebuilt),
// and again in `plugin-auth` -> `core`. Neither package aliases the dep it
// mutated; both are ordinary ledger entries, and `plugin-email` has no
// `vitest.config.*` at all.
//
// The two directions of forgetting are NOT equally dangerous:
//
// forgot to rebuild a FIX -> false RED. Costs a lap, and gets noticed.
// forgot to rebuild an ABLATION -> false GREEN. Silently certifies a vacuous
Expand All@@ -30,6 +48,17 @@
// by hand-grepping `dist/` for the mutation marker. This script is that grep,
// mechanized, with the traps the manual version cannot see.
//
// ### Worse than uninformative -- it points confidently the WRONG WAY
//
// When the ablation's purpose was to prove a NEW GATE is capable of failing,
// the false green is not a null result. It is the observation "the gate did not
// fire", and in that context that reads as evidence the GATE is broken rather
// than evidence the harness is. A dev acting on it goes hunting for a fix that
// is not needed -- or, the expensive outcome, WEAKENS a working gate until it
// "fires", destroying the thing the ablation was written to certify. That is
// the `plugin-auth` -> `core` shape above: ablation legs whose entire purpose
// was to demonstrate a new gate can fail, every one of them dist-mediated.
//
// ## The two ablation shapes, hence the two modes
//
// PLANT (default) the mutation ADDS something identifiable -- a changed
Expand All@@ -39,10 +68,20 @@
// so the assertion inverts: a literal unique to the deleted
// code must be GONE from dist. Same check, mirrored.
//
// `--absent` is also the restore leg: after putting the fix back and rebuilding,
// it proves the marker really left the artifact. That leg matters more than it
// looks -- a marker left behind in `dist/` keeps mutated code live for every
// later suite run in that worktree, long after the ablation is "finished".
// The rule they mechanize has TWO halves, and only one of them is intuitive:
//
// mutate -> rebuild -> prove the marker is IN dist/ (default mode)
// restore -> rebuild -> prove the marker is GONE (`--absent`)
//
// So `--absent` is two things at once: the mode for a DELETE ablation, and the
// restore leg of a PLANT one. The restore half is the one that gets skipped --
// rebuilding after mutating is obvious, while remembering that RESTORING also
// needs a rebuild before the NEXT measurement is trustworthy is not. It matters
// more than it looks: a marker left behind in `dist/` keeps mutated code live
// for every later suite run in that worktree, long after the ablation is
// "finished", so the runs that follow are measuring the wrong tree. Done by
// hand the leg reads `grep -c <marker> packages/core/dist/index.js` -> 0 after
// the `git checkout`; this script is that, plus the traps below.
//
// ## Sourcemap-only matches are RED, not green
//
Expand All@@ -51,14 +90,49 @@
// false green this script exists to prevent, so `.map` hits are reported and
// excluded from the verdict.
//
// ## The scan is WHOLE-PACKAGE, so the marker must be unique to the mutation
//
// Both modes grep the package's entire `dist/` tree, deliberately -- a mutation
// can land in any emitted chunk, and guessing which one is how the manual
// version missed things. The price is that the scan cannot tell YOUR literal
// from the same literal written elsewhere in the same package, and both modes
// assume it is unique. When it is not, both go wrong, in opposite directions:
// `--absent` reports surviving hits that were never yours (a false RED), and
// the default mode passes on a sibling's hit alone (a false GREEN -- the one
// this script exists to stop, reintroduced through the marker).
//
// Measured: ablating `internal: true` on ONE field of `sys_email`, then running
//
// node scripts/ablation-dist-preflight.mjs @objectstack/platform-objects 'internal: true' --absent
//
// reported 6 surviving hits, every one of them a legitimate `internal: true` on
// an identity object in `dist/identity/` and none of them the ablated field.
//
// There is no per-symbol mode. When the marker cannot be made unique -- a
// per-FIELD ablation of a flag the package also uses elsewhere is the standard
// case -- do NOT weaken the marker to make this script agree with you. Verify
// by PROPERTY READ against the same artifact instead, which is exact where a
// substring scan cannot be:
//
// node -e "const {SysEmail}=require('./packages/platform-objects/dist/audit/index.js');
// console.log(SysEmail.fields.headers_json.internal)"
// # before rebuild: true (ablation NOT in the artifact -> the green was vacuous)
// # after rebuild: undefined
//
// Same question, same moment in the procedure, same two halves (mutate and
// restore); only the instrument changes. What is not negotiable is that SOME
// instrument reads the built artifact before the ablation's colour is believed.
//
// ## Why this is not a `check:*` gate
//
// It judges a deliberately mutated working tree, so it can only be run by the
// agent performing the ablation, at one specific moment between "mutate" and
// "run the suite". CI has no ablation in flight and nothing to assert. It is
// dev-side agent tooling, invoked from the ablation procedure in
// `.claude/agents/os-dev.md` and `.claude/skills/dogfood-verification/SKILL.md`
// -- keep those two and this file's usage line in step.
// `.claude/agents/os-dev.md`, `.claude/skills/dogfood-verification/SKILL.md`
// and `packages/qa/dogfood/README.md` step 4 -- keep those three and this
// file's usage line in step. (Those three state the procedure scoped to the
// dogfood suite; the true condition is the resolution one stated at the top.)
//
// Anything this script cannot see is RED, never a skip: a missing `dist/`, a
// `dist/` with nothing readable in it, or a package name that resolves to
Expand Down
Loading