Uh oh!
There was an error while loading. Please reload this page.
fix(metadata-protocol): write the merged autonumber high-water mark before retiring the __global__ counter - #12554
Conversation
…efore retiring the __global__ counter The #8686 seed/API tenancy handoff ran an UPDATE of the organization-scoped _objectstack_sequences row followed by an unconditional DELETE of the '__global__' one. On a fresh install there is no organization-scoped row yet, so the UPDATE matched nothing (a success on every dialect), the DELETE ran anyway, and the counter table was left empty — sending SqlDriver.getNextSequenceValue back into its one-time MAX(data) bootstrap and re-issuing an already-allocated business identifier. The handoff is now one ordered decision per scope: write the merged mark (INSERT when the destination row is absent, UPDATE when it is not), read it back, and only then retire the '__global__' row by its own stored key_hash. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6HFzyH98W1YaQXhJUJt6o
📓 Docs Drift CheckThis PR changes 1 package(s): 12 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 4 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 7 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 1f71a49193c7b4de17cffd7daa138c07b649bc7b && git checkout 1f71a49193c7b4de17cffd7daa138c07b649bc7b
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin b307bfd2aef1067317900f73dafe15478ad8fe22 7ec7d8d4b43e9d6e34b7fa8331876317f994ea46 && git checkout -B drift-repro b307bfd2aef1067317900f73dafe15478ad8fe22 && git merge --no-ff 7ec7d8d4b43e9d6e34b7fa8331876317f994ea46
node scripts/docs-audit/affected-docs.mjs --json b307bfd2aef1067317900f73dafe15478ad8fe22
|
os-warren
commented
Aug 26, 2026
PM review — accepted, flipped ready, auto-merge armedI re-measured the three claims a PM should never take on the implementer's word, on
The part that earned the acceptTwo of the four ablation reds were pre-existing assertions this PR reverses —
The suite was pinning the symptom as the contract. Reversing it is the fix, and a reader six months from now will find out why without leaving the file. The card's explicit trap is cleared too: it warned that a "no duplicates after seed + sign-up + create" pin is green with and without the defect, so any pin must burn a number first. Both new pins delete the top-numbered row before exercising the handoff. Read the diff to confirm rather than the prose. And the Honest non-measurements, recorded rather than smoothed overBoth are declared in the report and neither is being counted as a pass:
Open question carried to the maintainer, not answered hereThe card's third item — a That reasoning is sound on its own terms and I am not overruling it. But the alternative the implementer names — a tombstone the backfill writes, so "this counter was retired here" becomes a fact in the schema rather than an inference — is the only option that would really detect it, and it is a stored-shape design decision, not a hunk. That is the maintainer's call, not the seat's. Flagged here; if it is wanted it gets its own card rather than an append to this one. CI is the remaining gate. Auto-merge armed; #12395 unblocks the moment this lands. Generated by Claude Code |
os-warren
commented
Aug 26, 2026
|
| check | conclusion | reading |
|---|---|---|
Temporal Conformance (live PG + MySQL) | failure | very likely ours — see below |
Test Core (1/6) | failure | 16-minute run; failing step not yet located |
Test Core (rollup) | failure | derived, not independent |
The rollup is not a third failure
"test-1-of-6 MISSING … 1 of 6 declared shard(s) published no positive attestation — a shard that never ran cannot be counted as passing (#6082)." It is shard 1's failure restated. Worth saying because three red checks read as three problems and it is two.
The one that points at this diff
The Postgres service log carries this twice:
ERROR: duplicate key value violates unique constraint "_objectstack_sequences_pkey"
DETAIL: Key (key_hash)=(b8c4320…6cca) already exists.
STATEMENT: insert into "_objectstack_sequences" ("field","key_hash","last_value","object","scope","tenant_id") values ($1,$2,$3,$4,$5,$6)
That INSERT is this PR's, and nothing else in the diff writes that table.
Working hypothesis, stated as a hypothesis because it is not yet confirmed from the failing assertion: the "is the org-scoped row absent?" probe and the table's primary key are not the same key. The probe asks by (object, field, tenant_id, scope); the PK is key_hash. A row that already exists under that hash but does not match the probe's predicate reads as absent, the INSERT fires, and the PK rejects it.
If that holds, it is the sharp edge this PR's own body identified and believed it had covered — the duplicated key_hash spelling, "controlled by a test that can only pass if the two agree". The {field}-scope pin does drive a real SqlDriver, and I verified that when I reviewed it. But it drives SQLite, and this is Postgres. The control was real and its coverage was narrower than the claim it was supporting. That is the honest read, and it is also exactly the gap the report flagged when it declared the live-MySQL suite not measured rather than passing — the declaration was correct and the risk it named landed.
os8269_poison_probe_pkey, os11161_no_such_relation), so a PG ERROR line is not by itself a failure. The failing assertion still has to be read from the job step output, and the dispatch says so.
What was NOT concluded
Test Core (1/6)'s own check-test-completeness step printed OK (6 of 6 scheduled package(s) reported … 3572 test(s) declared and all accounted for) — so the shard's tests accounted for themselves and the failure is in some other step, which I could not reach from the log tail. Not attributed to this diff yet, and not attributed away from it either.
Disposition
Dispatched to repair, with instructions to rule out "not this PR's" against main's own runs for both jobs before changing anything, and to add a pin that closes the dialect gap rather than one more SQLite case — a pin that only runs on SQLite is the same blind spot one layer along.
Auto-merge stays armed: it fires only on green, so a red head cannot slip through, and a validated fix merges without another round trip. ⛔ Nothing gets pushed to this branch that has not been validated first.
The serial hold on #12395 stands until this lands.
Generated by Claude Code
… the platform actually stores CI caught the #12394 handoff writing a SECOND counter row for one logical sequence, on live MySQL and on SQLite alike. Root cause is the fixtures, not the repair: both hand-seeded `key_hash` as an invented string (`'h1'`/`'h2'` and `'h_global'`/`'h_org'`), which was inert for as long as the repair addressed counter rows by `(object, field, tenant_id)`. #12394 addresses the destination row by `key_hash` — the table's only key — so an invented hash describes a table no install can hold: the org row reads ABSENT and a second row is inserted beside it. Measured: the driver stores `key_hash = sha256(object US tenant US field US scope)` for every row it writes, and `ensureSequencesKeyHashShape` recomputes the same hash for every legacy row it migrates. - cli: take the hash from the driver's own `sequenceKeyHash`, so the fixture is the same bytes the only production writer would have written and cannot drift. - metadata-protocol live-MySQL: spell the derivation independently (this package does not depend on driver-sql), making it a third spelling and therefore a pin on it; give `key_hash` its real PRIMARY KEY. - metadata-protocol unit: new #12394 suite over a KEYED store that answers the probe by its parameter and enforces the primary key. The INSERT-vs-UPDATE decision had no unit coverage keyed by a real hash — every existing fake matched on statement shape and handed back its one row for any key. `sequenceKeyHash` is exported from the module for that suite; it is NOT re-exported from the package index, so the published surface is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6HFzyH98W1YaQXhJUJt6o
os-warren
commented
Aug 26, 2026
CI red on |
os-warren
commented
Aug 26, 2026
PM review of the repair — accepted. My diagnosis was wrong on all three counts; here is the correction.I posted a hypothesis with this dispatch and it does not survive. Recording that plainly, because the dispatch brief told the dev to "confirm or refute rather than assume" and refuting it was the right outcome. 1. My hypothesis is refuted. I said the "is the org-scoped row absent?" probe and the primary key were different keys. They are not — probe, INSERT and UPDATE already address the same key. Nothing was wrong with the repair's key handling. 2. My dialect framing was wrong. I wrote that the 3. My smoking gun was a deliberate probe — and I had warned about exactly that trap in the same comment. The The real root cause is better than mine, and it is not in the repairTwo integration fixtures hand-seeded The production invariant was measured, not read: a real The finding under the findingEvery fake seam in the unit layer dispatches on statement shape — The new pin replaces that with a real keyed store: rows live under their own Also right: the two integration fixtures got no synthetic ablation, because their red was the real defect — observed before the fix, green after. Manufacturing one would have been theatre. Base branch ruled out first, as the discipline requiresRecent completed No assertion was changed, weakened, skipped or quarantined — the two failing assertions were already correct and now pass against fixtures that are realistic. One judgement call surfaced for the maintainer, not decidedShould the migration address the destination row by its stored Declared, still owed to CI
Auto-merge stays armed. #12395 unblocks when this lands. Generated by Claude Code |
Fixes#12394
The defect, reproduced before it was repaired
The #8686 seed/API tenancy handoff destroyed the counter it existed to move. It ran two
independent statements —
buildCounterMergeSql, anUPDATEof the organization-scoped_objectstack_sequencesrow, thenbuildGlobalCounterDeleteSql, an unconditionalDELETEof the'__global__'one. On a fresh install there is no organization-scoped rowyet, because no API create has happened: the
UPDATEmatched nothing (a success on everydialect), the
DELETEran regardless, and the counter table was left empty.That is the normal first-boot shape, not an edge case — it is precisely the shape
buildSplitProbeSql'sLEFT JOINwas widened to catch, so the repair fired on exactly theinstalls where its merge loop body never executed once.
Measured here on
origin/main,better-sqlite3, seed + burn + sign-up + create, with thebuilt artifact proved to carry the un-fixed module (
ablation-dist-preflight, markerabsent from all 24 built files):
{ "seed": { "seededMax": "ACC-000009", "counters": ["__global__=9"] }, "afterBurn": { "rows": 8, "dataMax": "ACC-000008", "counters": ["__global__=9"] }, "afterHandoff": { "status": "applied", "counters": [] }, "firstApiCreate": "ACC-000009", "reIssuedAnAllocatedNumber": true }ACC-000009handed out a second time, to a different record — the card's measurement,reproduced. The same harness on this branch:
{ "afterHandoff": { "status": "applied", "counters": ["org_mssymr19xzd645gv=9"] }, "firstApiCreate": "ACC-000010", "reIssuedAnAllocatedNumber": false }The burn is what makes the two distinguishable. With no number burned, the driver's
MAX(data)rescan lands on exactly the value the counter held, and both trees mintACC-000010.The fix
mergeSplitCounterreplaces the two hopeful statements with one ordered decision, perscope:
INSERTwhen the organization-scoped row is absent,UPDATEwhen it exists. The absent case is first boot.and an
UPDATEmatching zero rows is exactly this defect. The destination row isre-read and must hold at least the merged value.
'__global__'row, addressed by its own storedkey_hash, soa retirement can only ever hit the row whose mark was just merged.
A throw at any step leaves the
'__global__'row in place, which is the state the nextboot's split probe detects and retries; a failed repair now loses nothing. Partial progress
across scopes is safe for the same reason.
Per scope, because a
{YYYYMMDD}/{field}/ per-parent format runs one counter row perrendered prefix. The old merge was scope-blind in both directions: it could raise every
scope's counter to one merged value, and it deleted every scope's
'__global__'row.The merge rule itself is unchanged and is the 2026-08-15 ruling's: the greater of the two
counters, never the data max.
The
key_hashobjection, and why it is discharged rather than dodgedThe deleted comment argued the delete was the design: the driver keys counters by a
key_hashcomputed in app code, "and a hash spelled two ways is a counter the drivercannot find". That is right about the hazard and wrong about the remedy — deleting the row
hands the merged mark to nobody, and the driver's one-time
MAX(data)bootstrap thenre-derives a number it has already issued, which its own docstring forbids ("after the
one-time bootstrap the data table is never consulted again").
So the spelling is duplicated deliberately — the same way
GLOBAL_TENANTand the seedloader's platform-namespace regex already are here, because
metadata-protocolmust notdepend on a driver — and it is controlled by a test that can only pass if the two agree:
the new
{field}-scoped pin drives a realSqlDriveracross a burned number, so a hashdiffering by one byte leaves a row the driver never reads, the driver re-enters its
bootstrap, and the burned number comes back. Divergence is a red test, not a silent counter.
Table shape is asked of the database, once, in the
WHERE 1 = 0idiom this module alreadyuses: a table without
key_hashrefuses the probe, and that refusal is the answer. Thoseinstalls are keyed by
(object, tenant_id, field)— which is whatSqlDriveritself fallsback to on them, not a dialect invented here.
The second signal the card raised — declined, with reasons
The card asks whether
driver-sqlre-entering itsMAX(data)bootstrap after the countertable has been initialised deserves its own guard. Declined, deliberately, and the
reasoning is recorded in the module docstring rather than left in this PR:
if (!existing)is not evidence of lost state. That branch is the normalpath for every new tenant, every new day, and every new
{field}group — on a matureinstall, constantly. A guard keyed on "the table is not empty" would fire on the hot
allocation path.
"this key never existed" from "this key existed and was deleted". The guard could not
detect this defect even where it fired.
repaired by making the allocator smarter. The state was destroyed upstream; that is where
it is repaired.
If the maintainer wants a signal there anyway, it wants a tombstone the backfill writes —
a design decision, not a hunk to append here.
Public surface
No clause ② widening.
packages/metadata-protocol/src/index.tsis untouched(
git diff --stat origin/main...HEAD -- packages/metadata-protocol/src/index.tsis empty).The new SQL builders are module-scoped so the module's own unit tests can import them —
matching the index's own recorded rule that an export added so a cross-package test can
import a value is the shape to catch before it ships.
needs:contract-reviewcan come offon that basis. One module-private helper,
buildOrgCounterProbeSql, is deleted; it had noconsumer anywhere in the tree (zero-hit scan run with
buildCounterMergeSqlas the positivecontrol, which returned 4 files).
Verification
Ablation, one leg, prediction written before the mutation ran (direction and exact
count), mutation proved on disk with anchored
grep -cFcounts before any result was read,restored under
trap ... EXIT INT TERMand the restore verified by an emptygit status:stay green.
Tests 4 failed | 7 passed (11)— the same four, no others.mergeSplitCounter3 to 0,buildCounterInsertSql2 to0, and the two-way anchor
buildOrgCounterProbeSql0 to 2.ablation-dist-preflight @objectstack/metadata-protocol mergeSplitCounter --absenton the mutated leg, present on the restored one. The package resolves through its
exportstodist/, so an unbuilt mutation reads green.Two of those four failures are pre-existing assertions this PR changes on purpose:
[ruling: option 1]and[wiring]assertedreadSequences(driver)was[]after thehandoff. An empty counter table is the symptom, so the suite was pinning the defect as the
contract. Both now assert the mark survives in the organization's own row.
Green on
f7b251cbae, the final commit:pnpm lint(eslint . --no-inline-config, whole repo)lint-rc=0pnpm --filter @objectstack/metadata-protocol exec vitest run src/migrations/Test Files 9 passed / 2 skipped,Tests 191 passed / 10 skippedpnpm --filter @objectstack/runtime exec vitest run(both tenancy-split integration files)Test Files 2 passed (2),Tests 15 passed (15)pnpm --filter @objectstack/runtime typecheck(tsc --noEmit)pnpm check:type-check-debt(--re-measure, over a full workspace build)OK — 32 ledger entr(ies) re-measured, 1843 raw tsc error(s) total, none above its recorded numberGate union derived, not recalled —
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstackover the real changeset (4 paths, three-dot from the merge base). All 25 matched families
run locally, each exit code captured before any pipe, all
rc=0:nul-bytes·cross-package-test-inputs(both spellings) ·durability-log-level·objectql-double-limit·page-declaration-shape·published-files·slot-lookup·test-source-alias·type-source-resolution·ci-filter-parity·comment-mask-adoption·plugin-teardown-shape·docs-audit/check-affected-docs·docs-audit/check-drift-comment·query-options-erasure·engine-double-contract·where-matcher·type-check-coverage·type-check-debt·changeset-gate-self-tests·objectui-changeset·check-adr-0087-registration·check-changeset-no-major·check-empty-changeset·release-rehearsal-clone --self-test.Changeset
patchon@objectstack/metadata-protocol, graded deliberately: a defect repair inside anexisting migration, no export added to the package index, no API shape changed. The one
argument for
minorwould be new published surface, and there is none.@objectstack/runtimetakes no bump — its only change is test code.
Serial constraint
#12395's band (the
organizationIds.length !== 1guard and its warning payload, lines1016–1025 on
origin/main) is untouched. The earliest line this diff changes in thatfunction is 1083.
Generated by Claude Code