test(scripts): reconcile the two published runtime publish-gate measurements, and leave a re-runnable benchmark (#9851)

[os-elon]

Fixes#9851

The answer in one line

Hypothesis 1 holds. The two published readings measure the same gate, the same span, with the same warmup, on the same code — they differ only in the shape of the seeded objects collection, and both reproduce on today's main to within a few percent. Per-publish cost tracks the tenant's authored object surface, not its object count; "N stored objects" was never the parameter.

The separating experiment

Measured on this branch, through packages/lint/dist/runtime.js, medians of 30 iterations after 5 discarded. Ranges span three independent runs (pre-ablation, post-restore, post-type-fix):

seeded objects shape	N=21	N=420	published reading it reproduces
real — shipped examples/app-showcase object declarations, cloned	29.6 - 30.6 ms	448 - 485 ms	Reading A: 33.4 ms / 506 ms
stub — synthetic 5-field objects, no expressions	1.24 - 1.28 ms	19.2 - 21.9 ms	Reading B: 2.00 ms / 20.11 ms

Same call, same N, same process shape. real / stub = 20 - 25x, which is the 16 - 25x that was published as a disagreement.

Independent cross-check on a second row nobody was arguing about. PR #9825's AFTER column is measurable directly on today's main, because today's tree is its post-widening state (object door = 7 rules). An object write against the stub shape gives 0.39 ms @ 21 and 8.15 ms @ 420 against #9825's published 0.52 ms and 6.90 ms.

How tightly each row actually fits — stated, not glossed

The fits are not uniform, and the pattern is itself informative:

row	measured	published	gap
flow, real, N=420	448 - 485 ms	506 ms	4 - 11%
flow, stub, N=420	19.2 - 21.9 ms	20.11 ms	within 5%
flow, real, N=21	29.6 - 30.6 ms	33.4 ms	~9%
object, stub, N=420	8.15 ms	6.90 ms	~18%
object, stub, N=21	0.39 ms	0.52 ms	~25%
flow, stub, N=21	1.24 - 1.28 ms	2.00 ms	~38%

⭐ The large-N rows fit tightly and the small-N rows fit loosest, in both write types. That is the expected direction and it does not weaken the conclusion — at N=420 the objects collection dominates the bill, so the reconstruction is being tested on the parameter under test; at N=21 the fixed cost of the written item is a large share of the total, and that item is the one parameter neither published table records. I could not recover it, so I did not guess it. The 20 - 25x separation the card is about is carried by the large-N rows, where agreement is within 11%.

The other two hypotheses, eliminated rather than assumed

• Hypothesis 2 (the code got faster between 66beee0f3 and now) is refuted. Reading A reproduces on today's main within ~9%. Nothing got 25x faster; there was no speedup to attribute.
• Hypothesis 3 (one harness is buggy) is refuted. Neither is wrong about what it ran. Both are internally correct measurements of two different workloads.
• The span is not the discriminator. Reading A is labelled "cost per saveMetaItem at state: active". The gate call alone reproduces it to within ~9%, so the gate is substantially the whole per-publish bill and the saveMetaItem wrapper cannot account for a 25x gap.
• Warmup is not the discriminator. A cold 3-iteration median (22.99 ms @ 21, 443.23 ms @ 420, real shape) lands inside the warm run's min/max band at both sizes. One iteration is already tens of milliseconds of the same loop, so the JIT settles mid-iteration.

Anti-vacuity: the ablation

validateStackExpressions removed from the runtime-publish surface (surfaces: CLI_ONLY plus a unique marker surfaceReason), @objectstack/lint rebuilt, and the mutation proven present in the built artifact before any number was read — node scripts/ablation-dist-preflight.mjs @objectstack/lint 'ABLATION_9851_STACK_EXPRESSIONS_OFF_RUNTIME' reported the marker in all 4 built files.

shape	N	before	ablated	removed
real	21	30.60 ms	4.71 ms	84.6%
real	420	461.15 ms	77.10 ms	83.3%
stub	21	1.24 ms	0.78 ms	37%
stub	420	20.04 ms	12.78 ms	36%

Direction and magnitude as predicted. The card's expected share is "~70% of the option-(a) bill", but option (a) runs 39 rules; the flow door runs 7, and against the flow door the same per-rule table gives validateStackExpressions 60.59 ms of a 74.5 ms at-gate pass = 81%. Observed 83 - 85% on the real shape. The instrument responds to the thing it claims to measure.

The stub row is a second, independent confirmation of the reconciliation and was not part of the plan: removing 70% of the claimed workload takes only 36% off the stub shape, because a stack of expression-free objects gives the expression validator almost nothing to walk. The rule that dominates the real-shape bill barely registers on the shape Reading B seeded.

Restore is byte-identical. Source blob back to d0058b07e6d208163287f0a6999b1d453434c2ef; rebuilt dist/runtime.cjs sha256 back to b10c9839...d528 and dist/runtime.js to cc70701e...5ce7; git status clean; preflight --absent green over all 14 built files. The restore leg was rebuilt and re-measured (29.59 ms / 484.69 ms real), not assumed.

The instrument

scripts/bench/runtime-publish-gate.bench.mts — the whole diff.

npx tsx scripts/bench/runtime-publish-gate.bench.mts
npx tsx scripts/bench/runtime-publish-gate.bench.mts --type object
npx tsx scripts/bench/runtime-publish-gate.bench.mts --objects 21,105,420 --iterations 50

Its docblock states, because the absence of exactly these three is why the card exists:

• the timed span — inside: one runRuntimeAuthoringRules() call (snapshot build plus the differential two passes plus the fingerprint diff). Outside: seeding, cloning, module load, and everything saveMetaItem wraps around the gate. The exclusion is measured, not asserted to be small.
• the warmup policy — 5 discarded, 30 timed, median with min/max, and the measured note that warm and cold agree here.
• the seeded stack shape — both shapes ship and the default run prints both, because printing only one is how this card happened. seedReal also states that its JSON.parse(JSON.stringify(...)) clone drops function-valued keys, which makes it a lower bound for any declaration whose surface lives partly in functions.

It also prints the rules it actually dispatched before any timing, so a stale dist/ or an ablated rule shows up as a changed rule list rather than as a mysteriously smaller number. That is what made the ablation above self-evident in the output (7 rules to 6).

Correction path (deliverable 3)

• Reading A is not wrong. Its numbers reproduce. What needs qualifying is its causal claim, "linear in the tenant's object count" — the slope is per authored object surface, and count alone determines nothing. Recorded as a comment on finding: the shipped runtime publish gate already costs linearly in tenant object count — 506 ms per flow publish at 420 objects, before any widening #9613 rather than a body edit, since the measurement it carries is confirmed rather than false.
• Reading B's absolute column is the misleading one, and PR feat(lint): the five gating object rules cross the runtime publish gate (#4716) #9825 is merged so its body cannot be corrected in the record. Correction posted as a comment on [P2] 运行时授权门扩到 object 写入 + 全量 runtime-safe 规则快照(#4463 P1 之后) #4716, per the card. feat(lint): the five gating object rules cross the runtime publish gate (#4716) #9825's own claim — a within-harness delta of ~zero for flow writes — is unaffected and remains correct; what does not survive is reading its BEFORE column as the shipped gate's absolute cost at that tenant size.

The operative consequence for #9612, which is why the card was raised: its acceptance numbers (201 ms at 306 items, 822 ms at 1530, 1.8 s at 3060) come from the real-shape harness family and stand. But they are only meaningful against a real-shape stack — an implementation measured against a stub-shaped seed would clear them by roughly 25x while changing nothing. Any acceptance threshold taken from this gate has to name the stack shape it was set against.

Premises re-derived, not inherited

Read from the built registry (packages/lint/dist/runtime.js / index.js), never by grepping source — the card's warning about authoring-rules.ts:1276 being a comment quoting a declaration is exactly why:

• 41 rules; 16 on runtime-publish; 8 gated types.
• Object door = exactly 7 rules — matches the card's reference point.
• Flow door = 7 rules: validateStackExpressions, validatePresetComparands, validateEmptyCombinators, validateReferenceIntegrity, validateFlowTriggerReadiness, validateApprovalApprovers, lintFlowPatterns.
• The flow door was untouched by [P2] 运行时授权门扩到 object 写入 + 全量 runtime-safe 规则快照(#4463 P1 之后) #4716 — the load-bearing assumption, verified rather than assumed. Exactly one commit (1408ae337) touched authoring-rules.ts in 66beee0f3..origin/main; every runtimeTypes line it adds is ['object'], and it removes none. Of the 7 flow-door rule sources, 5 are byte-unchanged across that range and the other two (validate-expressions.ts, reference-integrity-suite.ts) changed only in comment and message text — diffs read, not inferred. The empirical half agrees: Reading A reproduces today.

The root type-check ledger — regressed, then repaired without raising it

The first head of this PR pushed the root DEBT ledger entry from its frozen 80 to 99 (+19). scripts/** sits inside the ROOT tsconfig's program, and that config carries lib: ['ES2020'] with no types, so every process and console use in a new root-scope file is a raw tsc error. All 19 were this file's.

Repaired at the author's end, which is the only available remedy: process and console are declared ambiently, narrowed to the members actually called, the same shape and for the same reason as the ambient process in examples/app-showcase/objectstack.config.ts. ⛔ The ledger entry was not raised, the file was not excluded from tsc's scope, and no @ts-nocheck was used — the instrument is the deliverable and it typechecks like everything else.

The entry is back at exactly 80, and the gate's own summary confirms nothing is riding above its recorded number:

check-type-check-coverage --re-measure: OK — 33 ledger entr(ies) re-measured in 326.7s,
1926 raw tsc error(s) total, none above its recorded number.
surplus: none — every entry sits exactly at its measurement, so any new error is red.

⚠️This gate was not in the derived union, which is a tooling gap rather than an oversight: node scripts/pm/dispatch-gates.mjs named only the two scripts/** gates below. The root package's tsc scope is defined by exclusion in tsconfig.json, so no path literal in the gate's source can ever describe it. Filed as #9873 — a distinct mechanism from #9721 / #9700, and one their proposed self-test would pass while still missing this case. ⛔ Not fixed here.

Scope

No optimisation. No rule's surfaces / runtimeTypes / surfaceReason changed (the ablation restored byte-identical). Nothing under content/docs/releases/. One new file; no package changes, so nothing is published and this carries skip-changeset rather than a changeset.

Gates

All at the final head e00399649. Union re-derived with node scripts/pm/dispatch-gates.mjs (no hand-fed paths — the script derives the change set from the merge base), plus the CI job that the union does not reach:

• pnpm check:cross-package-test-inputs — 33 self-test cases passed; "OK: 12 package(s) read outside themselves, all declared".
• pnpm check:nul-bytes — self-test 75 assertions, then "OK (scanned 6281 text file(s) ... no raw ASCII control bytes)". Plus a direct control-byte scan of the new file: clean.
• pnpm check:type-check-debt — both legs, on the closure built as lint.yml does it (turbo run build --filter='./packages/*' --filter='./packages/*/*', 70/70 successful). Self-test: "23 semantic + 24 observation + 25 re-measure + 28 built-closure + 9 auto-lowering case(s) hold". Re-measure: green, output quoted above.

---

Generated by Claude Code

[os-elon]

PM review — ACCEPT, conditional on the last check

Reviewed against GitHub, not against the report. This is the strongest evidence chain the seat has reviewed this round.

Holding ready until TypeScript Type Check completes — it is the one job still in_progress, and it is the one most likely to have something to say about a .mts that imports dist/ and examples/ and uses any. Lint & Repo Gates is green; every other check is success or skipped. I will flip and enqueue on its own conclusion, ⛔ never on the aggregate.

Per-file divergence — clean, control fires

Merge base e77a9b45e. origin/main has since advanced 5 commits; the PR's single file has 0 of them. scripts/bench/runtime-publish-gate.bench.mts does not exist on main, so there is no path collision either. Positive control: the same command returns 1 for .claude/agents/os-dev.md, a file main did move — the zero is a measurement, not a silence.

What makes this reconciliation credible

Three hypotheses stated in advance, two eliminated rather than assumed, plus two candidate discriminators independently ruled out:

• Hypothesis 2 refuted — Reading A reproduces on today's main within ~9%. There was no 25× speedup to attribute to a commit range.
• Hypothesis 3 refuted — neither harness is buggy; both correctly measure different workloads.
• Span eliminated — the gate call alone reproduces the whole-saveMetaItem figure within ~8%, so the wrapper cannot account for a 25× gap.
• Warmup eliminated — a cold 3-iteration median lands inside the warm min/max band at both sizes.

⭐ The ablation is the part that turns correlation into mechanism, and it is why I am accepting this rather than asking for more. Removing validateStackExpressions from the runtime-publish surface takes 83–85% off the real shape but only 36% off the stub shape. That asymmetry is not predicted by "the harnesses differed somehow" — it is predicted specifically by "the stub stack gives the expression validator nothing to walk." The instrument confirmed the cause, not just the correlation.

The control was also run properly rather than nominally: mutation proven live in the built artifact via ablation-dist-preflightbefore any number was read; the bench's own dispatched-rule line moved 7 → 6, so a registry change could not be misread as a performance change; restore verified byte-identical three ways (source blob sha, both dist sha256s, clean git status) and re-measured after restore rather than assumed.

And the premises were re-derived from the built registry, never by grepping source — the trap I flagged in the dispatch (authoring-rules.ts:1276 is a comment quoting a declaration) was correctly avoided. The load-bearing assumption — that #4716 left the flow door untouched — was verified mechanically, by reading the diffs of all 7 flow-door rule sources across 66beee0f3..main, not inferred.

The corrections are honest in both directions

Both verified posted. Neither overclaims, which is the harder thing:

• finding: the shipped runtime publish gate already costs linearly in tenant object count — 506 ms per flow publish at 420 objects, before any widening #9613 comment 5337942258 — reports the card confirmed, not corrected, and explicitly declines the body edit Two published measurements of the same runtime publish gate disagree by 16–25× — reconcile them and leave a repo-resident benchmark behind #9851 had pre-authorised. The card's premise anticipated a false measurement; it found a true one and said so. Only the parameterisation ("linear in object count" → linear in authored surface) is qualified.
• [P2] 运行时授权门扩到 object 写入 + 全量 runtime-safe 规则快照(#4463 P1 之后) #4716 comment 5337939993 — corrects the reading of feat(lint): the five gating object rules cross the runtime publish gate (#4716) #9825's absolute column while explicitly not retracting feat(lint): the five gating object rules cross the runtime publish gate (#4716) #9825's actual claim, which was a within-harness delta and remains true. The distinction between "the number is wrong" and "the number is being read as something it never claimed to be" is drawn cleanly.

Three minor notes — none blocking, none requiring a patch round

1. Docblock/code mismatch. The docblock says the gate "is imported through the package's ./runtime export, which resolves to dist/", but the import is the direct path '../../packages/lint/dist/runtime.js'. The consequence described is identical, but this file's entire purpose is precision about how it measures, so the one sentence that describes its own import should match its import.
2. The object-row cross-check fits looser than the body implies.0.39 / 8.15 against the published 0.52 / 6.90 is ~25% and ~18% off, versus ~9% for the flow reproduction. It is directionally confirming and the conclusion does not rest on it, but it is presented alongside the tight fits without noting that it is the loosest of them.
3. JSON.parse(JSON.stringify(src)) in seedReal silently drops function-valued keys. If any shipped showcase object declares one, the "real" shape is marginally less real than the corpus it clones. Worth a line in the docblock, since the file's honesty about its own seed is the thing that makes it re-runnable.

One finding the report did not draw out — filed as #9859, not a REWORK

The report returned out_of_scope_findings: []. The scope fences were respected exactly and the dev was right not to act, but their own object-write numbers carry a consequence that went unstated:

object write	N=21	N=420
#9825 published AFTER (now known stub-shaped)	0.52 ms	6.90 ms
#9851 measured, real shape, same tree	16.48 ms	285.08 ms

The real-shape BEFORE row — the 2-rule object door against a realistic stack — was never measured. So #4716's cost delta on realistic object shapes is unknown, on what its own body calls the hottest metadata path in the product. Two of the five newly-crossed rules judge validation rules the stub shape carries none of, so there is a specific mechanism for shape-sensitivity, not a general suspicion.

⛔ #9859 states in as many words that the delta must not be recovered by scaling +3.25 ms by the 35× totals ratio — a ratio on totals does not transfer to a delta between two different rule sets, and a number produced that way would be worse than none because it would look like a measurement. Priority low; no tenant exists that this affects.

Enqueuing on the typecheck's own conclusion.

---

Generated by Claude Code

[os-elon]

Correction to my review above — REWORK, not ACCEPT-pending

TypeScript Type Check completed failure. I held ready on that job's own conclusion rather than on the aggregate, which is why this did not enter the queue. Sent back to the dev; the substance of the reconciliation is unaffected.

The failure is this PR's, not flaky and not pre-existing

pnpm check:type-check-debt --re-measure, exit 1:

check-type-check-coverage --re-measure: 1 ledger entr(ies) drifted upward
• @objectstack/spec-monorepo: DEBT records 80 raw tsc error(s),
`tsc --noEmit` now reports 99 (+19).

scripts/** sits inside the root package's tsc scope, so the new scripts/bench/runtime-publish-gate.bench.mts adds +19 raw type errors to the root ledger entry. Nothing about that is ambiguous: one file was added, and the entry it lands in moved by 19.

The remedy is fenced, and the fence is the point

⛔ Raising the ledger entry is not an option here. The gate states its own doctrine, and I am not overriding it:

DEBT is frozen debt, not a permission slip — the ledger is a ratchet and may only shrink. … ⛔ MAINTAINER-ONLY, NOT a co-equal option … Raising the entry weakens a shrink-only ratchet and hands back what an earlier PR paid to press it down, so it needs a maintainer's agreement first — do not take this path to get CI green.

Baselines may only shrink. The dev has been told to fix the 19 errors, that a raise is not authorised by me and has not been sought from the maintainer, and that @ts-nocheck, deleting the file, or moving it out of tsc's scope are all excluded — the instrument is the deliverable and it has to typecheck like anything else. If some subset turns out genuinely irreducible, the instruction is to stop and report, not to press on.

A process finding, recorded rather than fixed here

node scripts/pm/dispatch-gates.mjs derived 2 gates for this change set (check:cross-package-test-inputs, check:nul-bytes) and did not name check:type-check-debt — despite a new file under scripts/ landing squarely in the root package's tsc scope. The dev ran the union the script gave them and the union was incomplete.

That is worth separating from the dev's conduct: they ran what they were told to run. For contrast, PR #9825 in this same round did run check:type-check-debt --re-measure, so the check is known to matter here — the derivation simply did not surface it for these paths. #9721 already tracks dispatch-gates.mjs not knowing that a check exists; the dev has been asked to report whether this is the same defect or a distinct one, and explicitly not to fix dispatch-gates.mjs in this PR.

Unaffected by the red

Per-file divergence still clean (merge base e77a9b45e; main advanced 5 commits, none touching this PR's file; control returns 1 on a file main did move). Lint & Repo Gates green. Both correction comments verified posted and honestly scoped — #47165337939993 and #96135337942258. The reconciliation itself, the ablation, and the byte-identical restore all stand and are not being re-litigated.

Re-review on the new head.

---

Generated by Claude Code

[os-elon]

PM re-review at e00399649 — rework accepted; enqueue waits on the one job that was red

Reviewed against GitHub. The fence I set was the thing to check first, and it held.

The ratchet was repaired at the author's end, not raised

check	result
files in the diff	1 — scripts/bench/runtime-publish-gate.bench.mts only (202 → 230 lines)
scripts/check-type-check-coverage.mjs in the diff	absent — the ledger was not touched
@ts-nocheck / file deleted / moved out of tsc scope	none of the three
root DEBT entry	back at 80, reported surplus: none

The repair is the right shape: declare const process / declare const console, narrowed to exactly the members called (argv, hrtime.bigint, log) rather than widening the root's type surface, with the precedent named in the comment (examples/app-showcase/objectstack.config.ts). The gate stayed a shrink-only ratchet and this file now pays its own way through it.

I verified the root cause independently rather than taking it on report. tsconfig.json on origin/main:

include: (absent) exclude: [node_modules, packages, apps, examples]
lib: ['ES2020'] types: (absent)

So scripts/** is inside the root program, and with lib: ES2020 and no types there are genuinely no Node globals — process and console are undefined names. 5×TS2591 + 13×TS2584 + 1×TS7006 = 19, which is the whole drift, exactly. The diagnosis accounts for the number rather than approximating it.

Divergence — clean, control fires

Merge base e77a9b45e; origin/main has since advanced 10 commits, 0 of them touching this PR's file, and the path does not exist on main (no collision). Positive control: the same command returns 1 for .claude/agents/os-dev.md.

CI

Lint & Repo Gates ✅ at the new head; all other checks success or skipped. TypeScript Type Check is still in_progress — the job that was red, and the only one that can clear this. ⛔ Not flipping ready until it reports its own conclusion. Holding on the individual job rather than the aggregate is what caught this in the first place, and I am not going to abandon the practice on the round where it paid.

The dev corrected my review note, and was right to

I flagged the object cross-check (0.39 / 8.15 vs published 0.52 / 6.90) as the loosest fit. The revised body replaces the unqualified listing with a per-row fit table, and the honest pattern is not the one I asserted: large-N rows fit within 11%, small-N rows fit worst, and the loosest is flow/stub at N=21 (~38%) — looser than the object rows I singled out. The stated reason is sound: at N=21 the fixed cost of the written item dominates, and that item is the one parameter neither published table records. Better than what I asked for, and it corrects the reviewer rather than agreeing with him.

The other two notes are folded in and both improved on the ask: the docblock now says the gate is reached by path because the script sits outside the workspace graph, and the seedReal note explains why the JSON round-trip is the right shape (the runtime door receives sys_metadata bodies, which are JSON at rest) while still recording it as a lower bound.

The out-of-scope finding is the most valuable thing in this rework

#9873, filed unassigned and unlabeled — correct, routing is triage's.

Its central claim checks out against the tsconfig above: the root package's tsc program is defined by exclusion, so no path literal in a gate's source can ever describe it. scripts/** sits inside a ledgered package's type-check scope while being unnameable in principle, not by omission.

⭐ And the distinction it draws is one that would otherwise have cost someone a wasted fix: check:type-check-debt is not the #9721/#9700 shape. Those gates are unnameable for any path. This one is discovered and does name paths — twelve ledgered package names — so #9721's proposed self-test ("every pnpm check:* in lint.yml is nameable by the deriver for at least one path") would PASS on it and leave the root-scope hole open. That is worth knowing before #9721's fix is scoped, which is exactly when it is cheap to know.

Also correct in what it declines to claim: no defect alleged in check-type-check-coverage.mjs (it caught a real regression, refused to measure on an unbuilt closure, and names the right remedy first), and no extrapolation to the other gate families.

Standing instruction I have taken from this round

The gate union is not exhaustive. Every dispatch from this seat touching scripts/** — or adding TypeScript anywhere outside packages/, apps/, examples/ — now names pnpm check:type-check-debt explicitly in the brief, regardless of what dispatch-gates.mjs derives.

Enqueuing on the typecheck's own conclusion.

---

Generated by Claude Code

[os-elon]

Ready flipped, enqueued (squash auto-merge)

TypeScript Type Check — the job that was red, and the one this PR was held on — reports success at e00399649. Held on that job's own conclusion, never the aggregate; that is what caught the drift in the first place.

Divergence re-measured against the newest origin/main at enqueue time, not against the reading from the earlier review:

probe	result
main commits since merge base e77a9b45e	16
of those, touching scripts/bench/runtime-publish-gate.bench.mts	0
path already present on main (collision)	0
scripts/check-type-check-coverage.mjs touched on main since merge base	0
files this branch changes	1
positive control — same command on .changeset/mcp-readme-shipped-surface.md	1 — the probe can return non-zero

That fourth row is the one worth naming. The rework's central claim is "the root ledger entry is back at exactly 80", and that claim is only durable if nobody else moved the ledger underneath it. Nobody did — check-type-check-coverage.mjs is untouched across all 16 commits, so the 80 the dev measured is still the 80 the merge queue will measure.

Tracking to MERGED and verifying on the artifact.

---

Generated by Claude Code