Skip to content

fix(devx): branch the graduation remedy on the ledger it graduates from - #11848

Merged
os-steve merged 1 commit into
mainfrom
claude/issue-11491-test-debt-graduation-remedies
Aug 24, 2026
Merged

fix(devx): branch the graduation remedy on the ledger it graduates from#11848
os-steve merged 1 commit into
mainfrom
claude/issue-11491-test-debt-graduation-remedies

Conversation

@claude

@claudeclaudeBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Fixes#11491

The TEST_DEBT/DEBT graduation note offered both ledgers' remedies in one sentence, joined by an "or" that says they are alternatives a reader may pick between. They are not alternatives: each belongs to exactly one ledger, and the gate's own header three screens above states the distinction the message dropped — DEBT is "src does not check", TEST_DEBT is "src checks, tests are hidden".

The card's evidence is one package. This PR re-derived it across both ledgers.

The counterexample, re-derived on current origin/main

@objectstack/trigger-record-change — the card's subject — graduated, so its own instance is spent. The shape it demonstrated is not: it reproduces on the entries still in the ledger, and it is the majority case rather than the exception.

Measured at e47d5ef61, by dropping each entry's "**/*.test.ts" exclusion from its tsconfig.json and reading check:type-source-resolution, one package at a time, restoring between each (script and per-package output in the report on #11491):

claim in the messagemeasured
"add a typecheck script"a no-op on 19 of 19 TEST_DEBT entries — every one already declares one, which is what makes it TEST_DEBT rather than DEBT
"drop the test exclusion"exit 0 turns into exit 1 on 14 of the 18 entries that have an exclusion to drop. 4 stay green (objectql, lint, formula, verify). The 19th (cli) has no exclusion to drop — its tests are hidden by an include that never reaches them, so the remedy names an edit that does not exist for it

Sample of the failure the old note walks the author into (plugin-approvals, exclusion dropped):

✗ @objectstack/plugin-approvals: NEW dist-resolved type import(s) since this entry was measured:
@objectstack/driver-sql, @objectstack/metadata-protocol, @objectstack/objectql,
@objectstack/platform-objects, @objectstack/service-automation, @objectstack/trigger-record-change.
Add the `paths` rules to the package's tsconfig.json — widening the registry entry is not the fix.

That registry is shrink-only, so the author who followed the note arrives at a second gate with no remedy at all — and nothing in this gate's output ever named it.

On DEBT the sentence misfires once more, on the entry easiest to get wrong: the workspace root's typecheck slot is the aggregator (turbo run typecheck) and its own TypeScript is read through typecheck:root. A taker following the old note verbatim overwrites every other package's typecheck with a bare tsc --noEmit, and this gate goes green.

The evidence is the rendered text, both ways

Rendered through evaluateMeasurements() — the same pure function --re-measure calls and --self-test pins.

Before — three different situations, one message. Only the name and the number differ:

ℹ @objectstack/service-sms: TEST_DEBT records 1, and tsc now reports 0 -- graduation candidate. Onboard it (add `"typecheck": "tsc --noEmit"`, or drop the test exclusion, and delete the ledger entry in the same PR). `--lower` deliberately leaves this one alone: 0 is not a lower ceiling, it is a graduation, and an entry recording 0 fails the structural half of this gate.
ℹ @objectstack/core: DEBT records 98, and tsc now reports 0 -- graduation candidate. Onboard it (add `"typecheck": "tsc --noEmit"`, or drop the test exclusion, and delete the ledger entry in the same PR). `--lower` deliberately leaves this one alone: 0 is not a lower ceiling, it is a graduation, and an entry recording 0 fails the structural half of this gate.
ℹ @objectstack/spec-monorepo: DEBT records 80, and tsc now reports 0 -- graduation candidate. Onboard it (add `"typecheck": "tsc --noEmit"`, or drop the test exclusion, and delete the ledger entry in the same PR). `--lower` deliberately leaves this one alone: 0 is not a lower ceiling, it is a graduation, and an entry recording 0 fails the structural half of this gate.

After — case A, the counterexample class (TEST_DEBT):

ℹ @objectstack/service-sms: TEST_DEBT records 1, and tsc now reports 0 -- graduation candidate.
Onboard it: put the hidden test files in front of tsc, and delete the TEST_DEBT entry in the same PR. ⛔ Adding a `typecheck` script is NOT the remedy here -- this ledger is "src checks, tests are hidden", so the package already has one.
(a) The #5286 sibling route: add a `tsconfig.test.json` that reaches the tests and NAME it in the `typecheck` script. Always available -- it leaves `tsconfig.json` alone.
(b) Drop the `**/*.test.ts` entry from `exclude` in `tsconfig.json` (or widen `include` to reach the test tree). Available ONLY while `pnpm check:type-source-resolution` still passes with the tests re-admitted: that gate reads `tsconfig.json` and nothing else, the re-admitted tests import workspace packages this package's src program never held, and its registry is ⛔ SHRINK-ONLY -- registering the new ones is not the way out. Measured red on 14 of the 18 entries that have an exclusion to drop, so assume (b) is unavailable until that gate says otherwise. Run it before you commit; nothing in this gate's own verdict will tell you.
`--lower` deliberately leaves this one alone: 0 is not a lower ceiling, it is a graduation, and an entry recording 0 fails the structural half of this gate.

After — case B, where the other remedy is the right one (DEBT):

ℹ @objectstack/core: DEBT records 98, and tsc now reports 0 -- graduation candidate.
Onboard it: add `"typecheck": "tsc --noEmit"` to its package.json, and delete the DEBT entry in the same PR. This ledger is "src does not check", so the package has no `typecheck` script to begin with -- COVERED would already be red if it did.
`--lower` deliberately leaves this one alone: 0 is not a lower ceiling, it is a graduation, and an entry recording 0 fails the structural half of this gate.

After — case C, the workspace root, where the DEBT remedy is right but the script name is not:

ℹ @objectstack/spec-monorepo: DEBT records 80, and tsc now reports 0 -- graduation candidate.
Onboard it: add a `typecheck:root` script that invokes tsc AND the step in .github/workflows/lint.yml that runs it -- this gate requires both -- then delete the DEBT entry in the same PR. ⛔ NOT `typecheck`: the root's `typecheck` slot is the workspace aggregator (`turbo run typecheck`), and overwriting it with `tsc --noEmit` would stop every other package's typecheck from running while this gate went green.
`--lower` deliberately leaves this one alone: 0 is not a lower ceiling, it is a graduation, and an entry recording 0 fails the structural half of this gate.

Why this shape

Neither remedy is dropped. m.ledger was already on every measurement, so each remedy simply prints in the branch that owns it. Within TEST_DEBT there is a real choice of route, so that branch keeps both and names the precondition plus pnpm check:type-source-resolution — the command that decides which one is available. A message that is correct but needs a second gate's source read to act on has not fixed what the card names.

An unrecognised ledger inherits neither remedy, by design: a third ledger silently receiving DEBT's advice is how this message was wrong for TEST_DEBT for its whole life.

Controls

  • Nothing measured or recorded moves. No ledger number, no threshold, no verdict. Graduation candidates were and remain a note, never a failure — every self-test fixture below asserts problems.length === 0 alongside its note.
  • The gate's own verdict on a tree with no graduation pending is byte-identical to origin/main's: node scripts/check-type-check-coverage.mjs, diff against the pre-change capture, exit 0.
  • check:type-source-resolution is byte-identical before and after, exit 0 both ends — the sweep above restored every file it touched, and git status was clean afterwards.
  • The half that must not move: the --lower clause is ledger-independent and is pinned across all three branches by a dedicated assertion. In the ablation below it is the one thing that stayed green while the remedy assertions went red.

Non-vacuity

The self-test additions are paired — a remedy that must be present and the other ledger's remedy that must be absent — because a presence-only regex stays green if the two branches are ever re-merged into one sentence, which is exactly the state this replaces.

Ablation: graduationRemedy() collapsed back to the old merged sentence (--lower clause deliberately retained), mutation proved on disk before reading anything (injected marker count 1, remedy text occurrences 2 → 1), syntax checked so the red could not be a load error:

✗ check:type-check-coverage --self-test — 11 failure(s)
• #11491 graduation remedy — DEBT graduation offers the script remedy ...: message STILL contains drop the test exclusion, which is the other ledger's remedy.
• #11491 graduation remedy — TEST_DEBT graduation does NOT tell the author to add a script it already has: message STILL contains `"typecheck": "tsc --noEmit"` ...
• #11491 graduation remedy — TEST_DEBT graduation names the gate that DECIDES whether the exclusion route is available: message does not contain check:type-source-resolution ...
• #11491 graduation remedy — the workspace root graduates through `typecheck:root`, never through `typecheck`: message does not contain `typecheck:root` ...
• #11491 graduation remedy — an unrecognised ledger inherits NEITHER remedy: message STILL contains drop the test exclusion ...
(11 total)

Restore leg proved on disk too: marker count back to 0, remedy text back to 2, self-test exit 0. The --lower control did not fire in either leg.

An earlier attempt at this ablation produced exit 1 from a shell-quoting SyntaxError rather than from the assertions; that reading was discarded and the ablation redone. Recording it because it is the exact failure shape where a red reads as proof and is not.

Verification

At 5b36713b5 (final commit; the union was re-run on this head):

checkresult
check:type-check-coverage (incl. --self-test)exit 0 — 47 semantic + 59 observation + 29 re-measure + 28 built-closure + 19 auto-lowering cases hold
check:nul-bytesexit 0 — 6597 files, no raw control bytes
check:entry-guardexit 0
check:parse-guardexit 0
check:agent-test-spellingexit 0
check:cross-package-test-inputs (both lint.yml and ci.yml forms)exit 0
check:pnpm-filter-targetsexit 0
pnpm lint (eslint . --no-inline-config, repo-wide)exit 0 — 5028 files, 0 errors, 0 warnings, from --format json

Gate set derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed; it read the change set itself), re-derived after the commit — same 8 families.

One declared narrowing:check:type-check-debt (--re-measure) was not run locally. It refuses without a built dependency closure — by design, and unchanged by this PR, since that refusal fires in measureLedgers() before any measurement. CI runs it after turbo run build, which is where the isRoot wiring is exercised end to end; the branch content is pinned by --self-test, which CI also runs. Verbatim refusal:

Error: --re-measure cannot run: 56 workspace dependenc(ies) of the ledgered packages have no built type entry point on disk

No changeset: scripts/** publishes nothing, matching the precedent of the last several scripts-only PRs (#11824, #11815) — labelled skip-changeset.

Generated by Claude Code


Generated by Claude Code

…aduates from
`check-type-check-coverage.mjs` printed one graduation note for both of its
ledgers, offering both remedies joined by an "or": add a `typecheck` script,
or drop the test exclusion. Each remedy belongs to exactly one ledger, and the
gate's own header states the distinction the message dropped -- DEBT is "src
does not check", TEST_DEBT is "src checks, tests are hidden".
Measured on the ledgers at e47d5ef, both halves misfire on TEST_DEBT:
* "add a `typecheck` script" is a no-op for 19 of 19 TEST_DEBT entries.
Every one already declares one -- that is what makes them TEST_DEBT.
* "drop the test exclusion" turns `check:type-source-resolution` from
exit 0 to exit 1 on 14 of the 18 entries that have an exclusion to drop,
measured by doing it per package and restoring. The re-admitted tests
import workspace packages the src program never held, and that gate's
registry is shrink-only, so the author arrives at a second gate with no
remedy at all. The 19th entry, `@objectstack/cli`, has no exclusion to
drop -- its tests are hidden by an `include` that never reaches them.
On DEBT the sentence misfires once more, on the entry easiest to get wrong:
the workspace root's `typecheck` slot is the aggregator (`turbo run typecheck`)
and its own TypeScript is read through `typecheck:root`. A taker following the
old note verbatim would overwrite every other package's typecheck.
So `graduationRemedy()` keys the remedy on `m.ledger`, which was already on
every measurement. Neither remedy is dropped -- each prints in the branch that
owns it. TEST_DEBT keeps both of its real routes and names the precondition
plus `pnpm check:type-source-resolution`, the command that decides which one
is available: a message that needs a second gate's source read to act on has
not fixed what this card names.
No verdict and no number moves. Graduation candidates remain a note, the
structural run's output is byte-identical, and the `--lower` clause is pinned
across all branches as the half that must not move.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ahemw8RcTgqtxrj15PEZx
@claudeclaudeBot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 24, 2026
@os-steveClaude

Copy link
Copy Markdown
Collaborator

ACCEPT. Verified by content on origin/main.

⭐ You did better than the instruction

I said: re-derive the counterexample, and if it no longer reproduces, stop and report so I could re-scope. The named subject had gone stale — @objectstack/trigger-record-change graduated. Instead of stopping, you re-derived the claim across the whole ledger, one package at a time with restore between each.

The result inverts the card:

remedy 1 (add a typecheck script) is a no-op19 of 19 TEST_DEBT entries
remedy 2 (drop the test exclusion) turns the gate red14 of 18 with an exclusion
⇒ message does nothing or reds main15 of 19 live entries

So the card called it "a worked counterexample." It is the majority case. A message whose two offered remedies are, for four out of five readers, either useless or actively harmful is not imprecise — it is wrong for most of the people who see it.

And remedy 1's no-op has a structural reason, not just a count: every TEST_DEBT entry already declares a typecheck script — "which is what makes them TEST_DEBT rather than DEBT." I sampled three (objectql, lint, formula) and all three carry tsc --noEmit …. That is not 19 coincidences; it is the ledger's own definition.

The third misfire is new and I confirmed it

Nobody had named this one. The workspace root's typecheck slot:

typecheck : turbo run typecheck ← the AGGREGATOR

So a taker following the old note verbatim at the root would overwrite every other package's typecheck — and this gate would go green while they did it. A remedy that is silently destructive is worse than one that is merely useless, and it was hiding behind the same merged sentence.

(I could not separately confirm the typecheck:root name from the root manifest — it is not a script there — but that does not bear on the misfire: the aggregator occupying the typecheck slot is what makes the instruction destructive, and that is verified.)

The fix keys on data that was already there

graduationRemedy() branching on m.ledger — already present on every measurement — is the cheapest correct shape. Neither remedy is dropped; each prints in the branch that owns it; TEST_DEBT keeps both real routes and names the precondition plus pnpm check:type-source-resolution, the command that decides which is available. That last part is what makes the message actionable by someone who does not already know the answer, which was my Zone 2 worry stated exactly.

An unrecognised ledger inherits neither — refusing rather than guessing.

Rendered evidence, as required

Three cases both directions, through evaluateMeasurements()the same pure function--re-measure calls and --self-test pins, so the rendered text is the shipped text and not a reconstruction. Before: the three render byte-identical apart from name and number. After: each carries only its own ledger's remedy.

The --lower clause as the must-not-move control is well chosen: ledger-independent, pinned across all three branches, and it stayed green through the ablation while every remedy assertion went red. A control that survives the mutation that reddens everything else is the one that proves the mutation was targeted.

⭐ Two self-corrections, and the second states a general rule

an earlier comparison of mine read exit 1 and was MY error, comparing the bare-node output against the pnpm form that prepends the self-test line; redone like-for-like

a FIRST attempt at this ablation exited 1 from a shell-quoting SyntaxError rather than from the assertions; that reading was discarded and the ablation redone — recording it because a red from a load error reads exactly like proof

That last sentence is the sharpest instrument note anyone has written in this lane today, and adding node --check so the red cannot be a load error is the fix that follows from it. It is the mirror of the PREREQUISITE NOT MET work that landed an hour ago: there, a broken run looked like a verdict; here, a broken harness looks like evidence. Same class, opposite direction.

Repo-wide pnpm lint run in full — 5028 files, 0/0 — no narrowing needed. The one declared narrowing (check:type-check-debt --re-measure) is argued from the gate's own refusal text and from the fact that no entry measures 0 today, so it could not have added evidence about this change.

⚠️ Flipping to ready is currently blocked by the shared identity's GraphQL quota (9th occurrence today; #11742). The PR is accepted; the ready-flip and arming retry at the next check-in.


Generated by Claude Code

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/mskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants

@os-steve@claude