fix(service-job): leader-elect interval schedules on multi-replica deployments

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[os-steve]

Closes#13686

DbJobAdapter — the adapter a production assembly upgrades to — routed type: 'cron' to CronJobAdapter, which takes a per-fire cluster lock, and type: 'interval' to IntervalJobAdapter, which has no lock anywhere in the file. So on a multi-replica deployment every replica armed its own setInterval and every tick executed N times. #2219 declared the capability as leader-electing scheduled cron/interval jobs across the cluster; only the cron half enforced it.

The reporter's premise, re-verified against the tree first

The card proposed delegating interval to this.cron because CronJobAdapter "already supports interval and holds the lock". That is the load-bearing claim, and a one-line delegation onto an adapter that only locks its cron branch would have fixed nothing silently. Measured on origin/main before writing any code:

claim	reading on the tree
CronJobAdapter schedules an interval registration itself	cron-job-adapter.tsschedule(): else if (schedule.type === 'interval' && schedule.intervalMs) arms setInterval(() => { void this.runScheduled(name); }, …)
that branch reaches the locked fire path	it calls the samerunScheduled() the cron branch's croner callback calls — one method, lock.acquire('job:' + name, { ttlMs, waitMs: 0 }), peers return early
DbJobAdapter sends interval to the unlocked adapter	schedule(): if (schedule.type === 'cron') to this.cron, every other type to this.inner, which is an IntervalJobAdapter
that adapter has no election	zero lock / leader / cluster / fence hits in interval-job-adapter.ts; control: the same grep on cron-job-adapter.ts resolves, so the query works

Premise holds. cron-job-adapter.leader.test.ts also already pins the lock semantics using { type: 'interval' } registrations, so the locked interval limb was covered — just unreachable from the adapter production assembles.

Their live evidence (3-replica cluster, traefik → 3 app replicas, shared postgres + redis, OS_CLUSTER_DRIVER=redis)

1. The fence counter is frozen. With all 8 jobs configured as 60 s intervals, os:fence:job:ts:* did not move for 100 s; under cron it increments by N (the replica count) per tick.
2. A caught race. One SLA escalation's action was de-duplicated by app-level business logic and fired once, but its notifications landed 2× for each of 3 recipients — 6 inserts inside a 54 ms window. A second "no eligible target" notification also doubled: both replicas sent before the dedup marker was persisted.
3. Business effects are not doubled only by luck. Per-handler business de-duplication plus staggered container start times mask the duplicate execution; the writes de-duplication does not cover (notifications) double.

Single-replica is clean for both schedule types — the defect is specific to multi-replica interval.

What changed

• db-job-adapter.ts — the routing.interval now goes to this.cron when one is assembled, inheriting the existing job: lock rather than growing a second locking implementation.
• interval-job-adapter.ts — a new register(). Delegated types are still registered on inner, but through a seam that stores a registration without arming a timer. This is the part the one-line version of the fix gets wrong: inner.schedule() on an interval registration arms a second, unelectedsetInterval beside the elected one, so one process would run the job twice per tick — strictly worse than the across-replicas duplication being fixed. cron was safe to hand down here only because IntervalJobAdapter happens not to arm a schedule type it cannot run; that inference stops holding the moment the delegated type is one it can run, so it is now said out loud. The cron branch was moved onto the same seam in the same edit — no behaviour change (the inner adapter is constructed without a logger, so its cron warning was already unreachable), but the routing now expresses its intent instead of relying on a coincidence.
• A warning on the cron-less limb. With no cron adapter assembled (enableCron: false, or its construction threw) an interval job still fires on inner's timer exactly as before — unelected. Silently dropping a job an assembly can run would not be an improvement, so it runs and says what is missing. Functional degradation, warn per the AGENTS.md level rule.

⛔ No new configuration key, per the card: #2219 declared this as the behaviour, and a switch would re-open the same declared-≠-enforced gap on the switch.

Public surface note for review: this adds one method, IntervalJobAdapter.register(), to an exported class. Additive, no accept-set widening, no spec/authorable surface. The path-derived tier limb does not fire (node scripts/pm/dispatch-gates.mjs --tier on the actual diff: "no path-derived mandate"), and the change alters no contract accept/reject behaviour — but the added method is declared here rather than left for a reviewer to find.

What is pinned — packages/services/service-job/src/db-job-adapter.interval-leader.test.ts, 10 cases

⚠️This is a concurrency defect across OS processes and the harness is single-process. Nothing below is a cluster test. What is pinned deterministically:

• Routing — asserted at the adapter seam, not against the wall clock: with a recording cron adapter that owns no clock, the interval registration arrives at it, and ten ticks produce zero runs (a second unelected timer in inner would show as ten). The registration is still visible through listJobs().
• One timer per process — with a real CronJobAdapter and a granting lock, one tick runs the handler once and acquires job:sla_escalation with { ttlMs: 60000, waitMs: 0 } exactly once; two ticks, twice.
• Lock semantics, two simulated replicas — one fake engine and one shared lock, two adapter stacks, one advanceTimersByTime. The winner holds its lease on a gate so the loser's acquire lands while the lock is held (making the count a fact about the lock, not about how many microtasks the timer flush ran). Result: one execution, one sys_job_run row, run_count +1 for one tick.
• The loser skips — driven at the fire seam so both promises are observable: both resolve, neither throws, the handler ran once, and the winner released its lease. waitMs: 0 is the skip spelled structurally — a waiting acquire would let the loser run the same tick a moment later.
• Single-replica / no cluster driver still fires — the pin that stops this repair from being worse than the defect.
• No cron adapter assembled — the job still fires on the inner timer, and the warning says it is unelected.
• Consumer surface unchanged — manual trigger() still runs on this node while a peer holds the lock (and would throw Job not found if the delegated registration had left inner); replay() + getExecutions() still work; the sys_job row is still upserted with schedule_type: 'interval'.
• Cron unchanged — a declared control case, plus the package's pre-existing cron suites.

Ablation

Direction predicted first: reverting only the interval routing branch should redden exactly the four pins that are about the elected path and leave the other six — and all 84 pre-existing package tests — green.

Committed the repair, then mutated the working tree and confirmed the mutation on disk before measuring: anchor marker count 1 → 0; blob hash f5e5c48d… (HEAD) → 40bce718… (worktree); git diff HEAD --stat non-empty (4 deletions). Restore leg proven by state, not by an exit code: git diff HEAD empty, worktree blob hash back to f5e5c48d… byte-identical to the HEAD blob, marker count back to 1, git status clean. The mutation script carried a trap restoring an absolute path, and no rebuild is involved — the suite imports the adapter by a relative specifier inside its own package, so it reads src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green).

Measured, 4 failed / 90 passed of 94:

× routes an interval registration to the cron (leader-electing) adapter…
→ expected [] to deeply equal [ { name: 'heartbeat', …(1) } ]
× one process holds exactly ONE timer for a delegated interval job…
→ expected "vi.fn()" to be called 2 times, but got 0 times (the lock is never acquired)
× two simulated replicas, ONE tick: exactly one execution and ONE run row
→ one tick must execute the job once across the cluster, not once per replica:
expected "vi.fn()" to be called 1 times, but got 2 times ← the defect itself
× the replica that loses the lock SKIPS…
→ expected "vi.fn()" to be called 1 times, but got 0 times

Prediction matched, including the second case's shape: reverting the routing leaves the handler firing once per tick (on inner's timer), so that pin reddens at the lock assertion rather than at the count — which is precisely why a count-only test would not have caught this defect.

⚠️Declared controls, green in both directions and therefore NOT ablation evidence: the six remaining new cases (single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control) plus all 84 pre-existing tests in the package. That last one is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped.

Measured / NOT MEASURED

• Measured here: the routing, the lock semantics at the adapter seam, the single-replica and no-cron fallbacks, and that the consumer surface is unchanged — all in one process, with a fake lock.
• NOT MEASURED, and only the reporter's deployment can measure it: that a real redis fence behaves this way across three real replicas — i.e. that os:fence:job:ts:* now increments once per tick under interval schedules as it already does under cron, and that the duplicate notification inserts stop. A single-process simulation is not a cluster verification and is not presented as one. @baozhoutao — the confirmation this needs is one run of your existing 60 s-interval configuration against a build carrying this branch, watching that counter and the notification table.
• NOT MEASURED locally, deferred to CI (prerequisite bound, exit 3 — not a pass and not a red):check:dual-build-cjs-loads and check:type-check-debt both require the full workspace build; check:test-completeness requires a saved turbo run test log and instructs that a local run record it as NOT MEASURED. service-job appears in neither the DEBT nor the TEST_DEBT ledger, and the structural half check:type-check-coverage is green, so nothing in this diff can move a ledgered count.

Verification (all at 7e3aee5a, the head of this branch, tree clean)

• pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2 → 9 files, 93 tests, 0 failed (84 pre-existing + 9 new at that point; 10 new after the contention pin was split, 94 total).
• pnpm --filter @objectstack/service-job run typecheck → exit 0. --listFiles confirms the program really contains 9src/*.test.ts files including the new one, so "typecheck clean" is a statement about the new test file and not a vacuous one.
• pnpm --filter '@objectstack/service-job^...' build → exit 0 (dependency closure, built before any judgement).
• Gate families re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands (39 runnable): all green except the three prerequisite-bound ones above. Three fired on the new test double and were repaired rather than baselined — check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (the caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch, and the new pinned coverage was recorded with --write, a ledger growth).
• pnpm lint (repo-wide eslint . --no-inline-config) → exit 0. Not narrowed.
• node scripts/check-nul-bytes.mjs → exit 0; the changed files also self-scanned clean for raw control bytes.

Every exit code above was captured before any pipe.

Out of scope

type: 'once' schedules take the same unlocked limb and duplicate the same way on a multi-replica cluster. Filed as #13918 rather than fixed here — it is not what #2219 declared, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease is a decision rather than a mechanical extension. out of scope: #13918.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

2 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• 3 name(s) were too generic to anchor anything (single lowercase words)
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 080b2e35ae916e37191b113cb3b606d556f18472 — the merge of head 7e3aee5a03fd18e94850988529add1d96df74157 into base 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 080b2e35ae916e37191b113cb3b606d556f18472 && git checkout 080b2e35ae916e37191b113cb3b606d556f18472
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 7e3aee5a03fd18e94850988529add1d96df74157 && git checkout -B drift-repro 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3 && git merge --no-ff 7e3aee5a03fd18e94850988529add1d96df74157
node scripts/docs-audit/affected-docs.mjs --json 4642f4c64c002f94f1d737bbb2a5fc94ae43ddf3

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.