Two published measurements of the same runtime publish gate disagree by 16–25× — reconcile them and leave a repo-resident benchmark behind

[os-elon]

Maintainer adjudication 2026-08-19: measurement reconciliation runs before either #9612 or #9613 is scoped. Both are blocked on this card.

The disagreement

Two published measurements describe the same operation — a flow publish through the shipped runtime publish gate, before any #4716 widening — and do not agree.

Reading A — #9613 body, sourced from #4716 comment 5309105113, measured on origin/main @ 66beee0f3 by a throwaway harness:

Today's shipped gate — before anything from the P2 widening lands — costs 33.4 ms at 21 objects → 506 ms at 420 objects per flow publish, linear in the tenant's object count.

Reading B — PR #9825 body, the BEFORE column, measured on dist/runtime.cjs, medians of 30 iterations after warmup, origin/main registry rebuilt and preflighted absent:

write	N stored objects	BEFORE
flow publish	21	2.00 ms
flow publish	420	20.11 ms

N	Reading A	Reading B	ratio
21	33.4 ms	2.00 ms	16.7×
420	506 ms	20.11 ms	25.2×

The ratio is not constant, so this is not a unit error or a constant-factor harness offset. Something structural differs.

PR #9825 treats the two as compatible — "the flow rows above show this PR adds nothing to it" — and that claim is fine on its own terms (it is a within-harness delta, and it is genuinely ~zero). But "my change adds nothing" does not establish which absolute baseline is true, and #9825 never claimed it did. Nobody has reconciled them.

Why this blocks both cards

#9612's acceptance criterion is quoted from its own body:

option (a) per-save p50 = 201 ms at 306 items, 822 ms at 1530, 1.8 s at 3060 … Any implementation must beat or bound this, or gate it.

Those numbers come from the same harness family as Reading A. If that harness is measuring something other than what its label says, then #9612's acceptance target is unreliable and a dev dispatched against it would be optimising toward a possibly-wrong number — and, worse, could "pass" by beating a target that was never real.

The product conclusion also inverts between the two readings. At 420 objects, Reading A says a Studio designer waits half a second on an interactive publish (perceptible, and it grows with exactly the number that grows as a tenant succeeds); Reading B says 20 ms (imperceptible). One of these is a priority and the other is a non-issue.

The three hypotheses — and the measurement that separates them

1. The harnesses measure different spans. One may time the whole saveMetaItem round-trip (schema parse, persistence, hooks) while the other times only the gate call; or one is cold and the other post-warmup; or they seed structurally different stacks (a 420-object stack with no flows/expressions is a different workload from one with them).
2. The code genuinely got faster between 66beee0f3 and today's 1408ae337.
3. One measurement is simply wrong — a harness bug, a mis-set N, a stale build.

Only (2) would be real improvement; (1) and (3) are instrument faults. Measure current main with BOTH methodologies — that single comparison separates them. If both methodologies agree on today's main, hypothesis 2 is live and the delta is attributable to a commit range. If they still disagree on the same tree, it is (1) or (3), and the job is to say which and why.

Deliverable

1. A reconciled number for the shipped gate's per-publish cost as a function of stored-object count, with the disagreement explained — not averaged, not quietly dropped. Name which of the three hypotheses holds and show the evidence for it. "One of them was wrong" is an acceptable answer if you can say how.
2. A repo-resident benchmark, not a third throwaway harness. Both existing readings came from one-off scripts that no longer exist in a runnable form, which is exactly why this card exists. Whatever you build must be re-runnable by the next person from a documented command, and must state its own span (what is inside the timed region and what is outside), its warmup policy, and the shape of the stack it seeds.
3. A written correction path for whichever published number turns out to be wrong. If Reading A is wrong, finding: the shipped runtime publish gate already costs linearly in tenant object count — 506 ms per flow publish at 420 objects, before any widening #9613's body carries a false measurement and must be corrected on the card. If Reading B is wrong, PR feat(lint): the five gating object rules cross the runtime publish gate (#4716) #9825's merged body carries one — that cannot be edited into the merged record, so it needs a comment on [P2] 运行时授权门扩到 object 写入 + 全量 runtime-safe 规则快照(#4463 P1 之后) #4716 recording the correction, the same way this seat handled the [finding] node-metadata-manager's watcher has the same one-shot delivery structure #9339 just repaired, with no reconciliation behind it #9701 branch-deletion correction.

Anti-vacuity requirement

A benchmark that cannot fail is not a benchmark. Demonstrate yours responds to the thing it claims to measure: ablate one hot rule (validateStackExpressions is ~70% of the option-(a) bill by both cards' accounting) from the runtime-publish surface, show the number moves in the expected direction and by roughly the expected share, then restore and show the tree is byte-identical. A benchmark whose reading does not move when you remove 70% of its claimed workload is measuring something else, and that fact is itself the answer to this card.

Scope fences

• ⛔ No optimisation in this PR. Do not scope what the hot rules are handed, do not add caching, do not change any rule's surfaces / runtimeTypes / surfaceReason. This card produces a number and an instrument. The fix is runtime gate: full runtime-safe rule snapshot for the publish door — the expensive half split out of the object-gating card #9612/finding: the shipped runtime publish gate already costs linearly in tenant object count — 506 ms per flow publish at 420 objects, before any widening #9613's, and how to scope it is a maintainer decision that is explicitly still open (per-rule scoping vs caching+invalidation vs size threshold — and the size-threshold option is a deliberate fail-open at scale, so it is not a dev's pick).
• ⛔ Ablation is for the anti-vacuity control only, and must be restored byte-identical.
• Reference points, to be re-derived rather than inherited: the object door is 7 rules at 1408ae337; the flow door was untouched by [P2] 运行时授权门扩到 object 写入 + 全量 runtime-safe 规则快照(#4463 P1 之后) #4716 (feat(lint): the five gating object rules cross the runtime publish gate (#4716) #9825's premise re-derivation), which is what makes the flow-publish number comparable across the two refs in the first place — verify that rather than assuming it, because it is the load-bearing assumption of this whole comparison.

Refs: #9612 · #9613 · #4716 (comments 5309105113, 5309093362) · PR #9825 (merged 1408ae337) · origin/main @ 66beee0f3 (Reading A's ref).

[os-elon]

Claimed by the metadata / engine-core / drivers PM seat — session session_019yDEhPBC3tcGkW9bkce1HM, branch claude/issue-9851-publish-gate-benchmark-reconcile.

Maintainer scoping input — read this before sizing the work

Adjudicated 2026-08-19 by the maintainer, who chose measurement reconciliation over three alternatives (fix the hot path directly / implement #9612 as specced / park both). Two qualifiers came with it, and the second is the one that sets this card's size.

Verbatim, on whom the latency affects today: 「目前也没有相关的真实用户」 — there are no relevant real users. This sits alongside the standing instruction from the #4716 adjudication, 「objectstack cloud 还没有正式上线,简化开发」.

So be clear about what this card is and is not. It is not performance firefighting. Nobody is waiting on a slow publish today; both readings describe a synthetic seeded stack, and the disagreement between them currently costs zero user-visible latency. What it is: an instrument-correctness card. Its entire value is stopping a possibly-wrong number from propagating into #9612's acceptance criterion, where it would silently define what "fast enough" means for work that has not started yet. A dev optimising against a phantom target, and passing, is the failure this prevents.

Two consequences for how you work it:

• Bound the effort. Enough benchmark to answer the question and be re-run by the next person — not a performance-engineering apparatus. If you find yourself building fixture generators for stack shapes nobody has, stop; that is past the line. Report the reconciliation you can defend and say what you did not measure.
• Do not let "no real users" become "so the number doesn't matter." It matters because it is about to be used as an acceptance gate. A wrong baseline is cheap to fix now and expensive to discover after someone has been held to it.

If the reconciliation turns out to be trivial — say, the two harnesses obviously time different spans and one line of each script says so — then the honest deliverable is small: state it, correct the wrong published number per the card's item 3, leave the re-runnable benchmark, and stop. A short PR is a good outcome here, not an under-delivery. Do not pad it to look proportionate to the discussion that produced it.

The rollback clause applies with unusual force here

This card's premise is that two published numbers disagree. If you measure and find they do not actually disagree — the readings are reconcilable at a glance, or I have misread one of the two source tables — then the premise is falsified. STOP and report back. ⛔ Do not silently re-adjudicate into "well, I'll optimise something while I'm here," and do not quietly widen into #9612's territory. The scope fences in the body are hard: no optimisation, no surfaces / runtimeTypes / surfaceReason edits, ablation restored byte-identical.

Same discipline in the other direction: if you find a third disagreeing number, or the flow door turns out to have been touched by #4716 after all (the body flags that as this comparison's load-bearing assumption, to be re-derived rather than inherited), that changes the card — report it rather than absorbing it.

Blocked on this card

#9612 and #9613 are both parked pending this reconciliation, by the same adjudication. Neither is dispatched.

---

Generated by Claude Code

[os-elon]

{
 "issue": 9851,
 "status": "done",
 "branch": "claude/issue-9851-publish-gate-benchmark-reconcile",
 "pr": "https://github.com/objectstack-ai/objectstack/pull/9853",
 "premise_still_valid": true,
 "summary": "Hypothesis 1 holds: the two published readings measure the same gate, the same span, with the same warmup, on the same code, and differ ONLY in the shape of the seeded objects collection. Both reproduce on today's main at 9acd8fd6a (base e77a9b45e): real shipped showcase objects give 29.6-30.6 ms @21 and 461-485 ms @420 (Reading A: 33.4 / 506); synthetic 5-field expression-free objects give 1.24-1.28 ms @21 and 19.2-20.0 ms @420 (Reading B: 2.00 / 20.11). Hypothesis 2 refuted (Reading A reproduces today, so nothing got 25x faster); hypothesis 3 refuted (neither harness is buggy, both correctly measure different workloads); span eliminated (the gate call alone accounts for the whole-saveMetaItem figure within ~8%); warmup eliminated (cold 22.99/443.23 sits inside the warm min/max band). The reconciled statement: per-publish cost tracks the tenant's authored object SURFACE, not object COUNT, so any number quoted as 'X ms at N objects' without its stack shape is not reproducible. Deliverable: scripts/bench/runtime-publish-gate.bench.mts, one new file, no package changes, skip-changeset label applied and read back. Corrections filed per deliverable 3: #4716 comment 5337939993 (Reading B's absolute column, PR #9825 being merged) and #9613 comment 5337942258 (Reading A confirmed, not corrected; its 'linear in object count' parameterisation qualified). Independent cross-check: today's tree IS #9825's post-widening state, so its AFTER object rows are directly measurable, and the stub shape reproduces them at 0.39 ms @21 / 8.15 ms @420 against the published 0.52 / 6.90.",
 "tests": "All at HEAD 9acd8fd6a (base origin/main e77a9b45e), after the final commit. GATE UNION re-derived with `node scripts/pm/dispatch-gates.mjs` (no hand-fed paths; script derived 1 path vs merge base e77a9b45e), naming 2 gates, both green: `pnpm check:cross-package-test-inputs` -> 'All 33 self-test cases passed. OK: 12 package(s) read outside themselves, all declared, and turbo.json hashes every declared glob.'; `pnpm check:nul-bytes` -> 'check-nul-bytes --self-test: 75 assertions' then 'OK (scanned 6281 text file(s) -- 6281 tracked, 0 untracked-not-ignored; skipped 5 binary; no raw ASCII control bytes)'. Plus a direct control-byte scan of the new file: clean (grep exit 1). No package test suite is in scope: the diff adds one scripts/bench file and changes no package. BENCHMARK RUNS (median of 30 after 5 warmup, through packages/lint/dist/runtime.js, built closure `pnpm --filter '@objectstack/lint^...' build` then `pnpm --filter @objectstack/lint build`, both under flock /tmp/os-heavy-verify.lock): flow write real 30.60 ms @21, 461.15 ms @420; flow write stub 1.24 ms @21, 20.04 ms @420; object write real 16.48 @21, 285.08 @420; object write stub 0.39 @21, 8.15 @420. ABLATION (anti-vacuity control) — validateStackExpressions taken off the runtime-publish surface via `surfaces: CLI_ONLY` plus marker surfaceReason ABLATION_9851_STACK_EXPRESSIONS_OFF_RUNTIME. BOTH LEGS REBUILT AND PREFLIGHTED. Mutation leg: `pnpm --filter @objectstack/lint build` then `node scripts/ablation-dist-preflight.mjs @objectstack/lint 'ABLATION_9851_STACK_EXPRESSIONS_OFF_RUNTIME'` -> 'marker present in 4 built files ... the ablation is live in the artifact the suite consumes' (exit 0); measured real 30.60 -> 4.71 ms @21 (-84.6%) and 461.15 -> 77.10 ms @420 (-83.3%), stub 1.24 -> 0.78 (-37%) and 20.04 -> 12.78 (-36%); the bench's own 'rules dispatched' line went 7 -> 6, so the registry change is visible in the instrument's output rather than only in the timing. Predicted direction and magnitude: the card's ~70% is the option-(a) share across 39 rules, but the flow door runs 7 and the same per-rule table gives validateStackExpressions 60.59 of 74.5 ms at-gate = 81%; observed 83-85%, as predicted. Restore leg: `git checkout -- packages/lint/src/authoring-rules.ts`, rebuilt, `ablation-dist-preflight ... --absent` -> 'marker absent from all 14 built files' (exit 0); byte identity proven three ways — src blob back to d0058b07e6d208163287f0a6999b1d453434c2ef, dist/runtime.cjs sha256 back to b10c983938572c649c65600129ae975bf63451639ba179f5785b5779b0b9d528, dist/runtime.js back to cc70701ef46859bd101762262888c7976ddb505ca161ef9bfeddd34295025ce7, `git status` empty; re-measured after restore at 29.59 ms @21 / 484.69 ms @420, matching pre-ablation. PREMISES re-derived from the BUILT registry (packages/lint/dist/runtime.js and index.js), never by grepping source: 41 rules, 16 on runtime-publish, 8 gated types, object door = exactly 7 rules (matches the card), flow door = 7 rules. Flow-door-untouched-by-#4716 verified mechanically: exactly one commit (1408ae337) touched authoring-rules.ts in 66beee0f3..origin/main, every runtimeTypes line it adds is ['object'] and it removes none; 5 of the 7 flow-door rule sources are byte-unchanged across that range and the two that changed (validate-expressions.ts, reference-integrity-suite.ts) changed only in comment and message text, diffs read line by line.",
 "open_questions": [],
 "out_of_scope_findings": []
}

---

Generated by Claude Code