fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618

Merged
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election
Sep 2, 2026
Merged

fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter#14618
os-sales merged 5 commits into
mainfrom
claude/issue-13918-once-schedule-leader-election

Conversation

@os-sales

Copy link
Copy Markdown
Collaborator

Fixes#13918

DbJobAdapter.schedule() decides which adapter owns a scheduled fire, and only that
choice decides whether the fire is leader-elected: CronJobAdapter takes the cluster
lock in runScheduled(), IntervalJobAdapter holds no lock at all. cron was routed
to the electing adapter from the start and interval since #13686once was the
limb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.

Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated

Recorded by the director seat in 13918#issuecomment-5494528879:

  1. 修法照卡:DbJobAdapter.schedule()type:'once' 委托给 this.cron(CronJobAdapter.schedule() 已有 once 分支,走同一条 runScheduled() + lock.acquire 发射路径 —— service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686 对 interval 的同款,一枝之隔);inner 侧走 register() 存不armed,保住 trigger()/replay()/getExecutions()/listJobs();
  2. 崩溃语义裁定:at-most-once per cluster 即可 —— 今天单节点 setTimeout 本就不落盘、崩溃同样丢,收敛到 leader-elected 不使任何场景变差(今天的现实是「每副本各跑一遍」的重复灾);「锁成功才释放」的重投机制 ⛔ 不建(无实测消费者,不为设想场景造持久化);语义写进 docblock 一句;
  3. 消费者普查随实施顺手做(grep type: 'once' 注册点),结果记 PR 正文 —— 零消费者也照修(小、关洞、保险);
  4. 复现钉照卡的 repro sketch(两个 DbJobAdapter 栈共享一把锁,今天两行 sys_job_run,修后一行);
  5. Clause-②:预期 no(services 实现面),实施者按实际 diff 复declare。

The change

One condition, in DbJobAdapter.schedule():

-}elseif(schedule.type==='interval'&&this.cron){+}elseif((schedule.type==='interval'||schedule.type==='once')&&this.cron){// The leader-elected path — same one cron takes, for the same reason.awaitthis.cron.schedule(name,schedule,wrapped,downstream);awaitthis.inner.register(name,schedule,wrapped,downstream);

inner.register() stores without arming, so trigger(), replay(), getExecutions()
and listJobs() keep answering from one place, and one process never holds an elected
timer beside an unelected one. Everything else in the diff is the docblock and the pins.

Crash semantics, per ruling point 2, stated in the schedule() docblock:once is
at-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected setTimeout was not
persisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.

One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (enableCron: false, or cron construction threw) is unchanged for
once: it still fires on inner's own timer. interval emits a warn in that case;
once deliberately does not, and the reason is frequency, not importance. Interval
registrations are per-plugin-startup and countable; once registrations are
per-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a warn on a hot path is how everyone
learns to skim warn. The docblock says this in place. No new log site of any level is
added by this PR.

Consumer census (ruling point 3)

Method: git grep -n "type: 'once'" and git grep -n 'type: "once"' over packages,
examples, apps, skills, content, scripts, then each hit classified by reading
its call site. Positive control: the same method run for type: 'interval' returns
the registrations #13686 was about (plugin-approvals escalation sweep
approvals-plugin.ts:329, plugin-reportsreports-plugin.ts:155), so a zero here
would have been a real zero. It is not a zero.

SiteKindReaches IJobService.schedule?
packages/services/service-automation/src/builtin/wait-node.ts:259registration — a wait node arms its timer resumeyes, one per suspended flow run
packages/services/service-automation/src/builtin/wait-node.ts:440registrationrearmSuspendedWaitTimers on cold bootyes, one per suspended run, on every replica's boot
packages/triggers/trigger-schedule/src/schedule-trigger.ts:140registrationnormalizeSchedule for a schedule-triggered flow declaring atyes, via ScheduleTrigger.start()
packages/runtime/src/job-schedule.ts:62registrationtoBoundaryJobSchedule for an app-declared once jobyes, via app-plugin.ts:1017
examples/app-showcase/src/automation/flows/index.ts:576prose commentno
packages/runtime/src/job-schedule.test.ts, packages/spec/src/system/job.test.ts, packages/triggers/trigger-schedule/src/schedule-trigger.test.ts, packages/services/service-automation/src/builtin/wait-node.test.tstestsno
content/docs/automation/jobs.mdx:88, content/docs/references/system/job.mdx:58, packages/services/service-job/README.md:75, packages/triggers/trigger-schedule/README.md:44docs / type proseno
packages/services/service-automation/CHANGELOG.md (3 hits)changelog proseno

Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.

Premise checks (verified on origin/main before the first edit)

PremiseVerdict
P1schedule() still routes once to this.inner.schedule(...) while cron/interval go to this.cron + inner.register()holdsonce fell through to the else limb
P2CronJobAdapter.schedule() still has its own type === 'once' && schedule.at branch arming setTimeout(() => { void this.runScheduled(name); }, delay), and runScheduled takes lock.acquire('job:' + name, { waitMs: 0 })holds — both, unchanged
P3The card's repro executes twice todayholds, reproduced before any source edit — see below
P4packages/spec/** and content/docs/releases/** untouchedholds — the diff is four files, none of them under either path

Hypotheses the dispatch declared

HypothesisVerdict
H1The whole fix is the once limb mirroring the interval limb; the no-cron fallback stays inner.scheduleholds — one condition changed, nothing else
H2trigger() / replay() / getExecutions() / listJobs() still answer for a delegated once jobholds — pinned in three tests
H3cancel() still cancels a delegated once job on both adaptersholds — pinned: cancel before at gives zero executions, zero acquire calls, listJobs() empty and sys_job.active === false
H4At-most-once by construction; no re-arm exists and none is added; one docblock sentenceholds — the lease is released by the existing runScheduledfinally; the diff adds no timer, no retry and no store

Tests

Head: 51fb2de6f. All runs under scripts/pm/os-verify-lock.sh (shared container),
exit codes captured before any pipe.

P3 reproduction, before the source editvitest run src/db-job-adapter.once-leader.test.ts:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times

After the fixpnpm --filter @objectstack/service-job test (whole package):

 Test Files 10 passed (10)
Tests 106 passed (106)

Type checkpnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,
exit 0. Coverage measured rather than assumed: the --listFiles output names both edited
files (db-job-adapter.ts and db-job-adapter.once-leader.test.ts), 407 files total, so
"typecheck clean" really does cover the new test file. The package's tsconfig.json
includes src and excludes only node_modules/dist.

The twelve new pins in db-job-adapter.once-leader.test.ts, all deterministic and all in
one process (like their interval sibling, this is not and cannot be a cluster test — it
pins ROUTING and LOCK SEMANTICS at the adapter seam):

  • routing: a once registration reaches the cron adapter, and ten deadlines' worth of
    fake time produces zero runs from any timer DbJobAdapter armed itself;
  • the card's pin (ruling point 4): two DbJobAdapter stacks, one fake engine, one
    shared lock, one { type: 'once', at } each, one advanceTimersByTimeAsync ⇒ one
    execution, two acquire calls with waitMs: 0, onesys_job_run row,
    run_count: 1;
  • the losing replica skips: resolves, does not throw, does not retry, and the winner
    releases its lease;
  • one process holds exactly one timer, and a one-shot stays a one-shot (five further
    deadlines add no executions);
  • single-replica with a cron adapter but no cluster driver ⇒ still fires once;
  • no cron adapter assembled ⇒ still fires once on the inner timer, exactly as before;
  • a deadline already in the past arms nothing, with or without a cron adapter, and stays
    registered for manual triggering;
  • H2: trigger() while a peer holds the lock, replay() + getExecutions(), listJobs();
  • H3: cancel() before the deadline, on both adapters;
  • the sys_job upsert for a once schedule (schedule_type: 'once', expression = at);
  • a declared control: cron and interval routing unchanged.

Ablation (on the committed tree)

Mutation: restore the pre-fix routing ((schedule.type === 'interval' || schedule.type === 'once') && this.cron
back to schedule.type === 'interval' && this.cron). No rebuild is involved on either
leg and none is owed: the pins import the subject relatively
(import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to the
package's TypeScript source, never through the package exports to dist/. That is not
an assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no service-job build in between.

Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:

HEAD blob for packages/services/service-job/src/db-job-adapter.ts: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
PRE-MUTATION fixed-form lines: 1 broken-form lines: 0 hash: 4d58eb1fa64a5b342643ab5eea7c2ab46354705b
POST-MUTATION fixed-form lines: 0 broken-form lines: 1 hash: 6e2c49ae1803e7fc275e5d346725b3680e9d00b4
MUTATION CONFIRMED ON DISK (hash moved off the HEAD blob).

Mutated leg — vitest run src/db-job-adapter.once-leader.test.ts, exit 1:

 Test Files 1 failed (1)
Tests 4 failed | 8 passed (12)
AssertionError: one deadline must execute the job once across the cluster, not once per replica: expected "vi.fn()" to be called 1 times, but got 2 times
AssertionError: each replica must ASK the fence — an unrouted fire never consults it at all: expected "vi.fn()" to be called 2 times, but got 0 times
AssertionError: expected [ { …(10) }, { …(10) } ] to have a length of 1 but got 2

That third line is the card's sys_job_run count, and it is why the three pre-row
assertions in that one test are expect.soft: unrouted, the two replicas write two
rows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (acquire called zero
times).

Mutated-leg control — vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:

 Test Files 1 passed (1)
Tests 10 passed (10)

The mutation touches only the once limb, so #13686's pins must stay green — they do,
which is what makes the red above attributable to this change rather than to the harness.

Restore leg, proved the same way rather than by an exit code (git checkout HEAD -- ...
with an absolute path, plus a trap ... EXIT INT TERM so a container cap kill cannot
leave a mutated tree behind):

RESTORE CONFIRMED: hash equals the HEAD blob and `git diff HEAD` is empty for packages/services/service-job/src/db-job-adapter.ts.
restored-form lines: 1 (expect 1)

Restored leg — the same file, exit 0:

 Test Files 1 passed (1)
Tests 12 passed (12)

Gates

Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. The
family was re-derived after the ledger row below entered the diff — that added seven
gates (check:agent-test-spelling, check:bash32-floor, check:cli-command-ids,
check:entry-guard, check:parse-guard, check:pnpm-filter-targets,
check:watch-hint-literal) which were then run too.

All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (PREREQUISITE NOT MET — NOT MEASURED), 0 red.

check-adr-0087-registration OK check-changeset-no-major OK check-ci-filter-parity OK
check-comment-mask-adoption OK check-cross-package-test-inputs OK check-empty-changeset OK
check-keyed-text-bounds OK check-plugin-teardown-shape OK check-shard-attestation OK
check-system-context-census OK check-tenant-audit-census OK check-test-completeness NOT MEASURED (exit 3)
check-undeclared-dep-imports OK docs-audit/check-affected-docs OK docs-audit/check-drift-comment OK
pm/check-half-states OK pm/release-rehearsal-clone --self-test OK
check:agent-test-spelling OK check:bash32-floor OK check:changeset-gate-self-tests OK
check:cli-command-ids OK check:cross-package-test-inputs OK check:doc-authoring OK
check:dual-build-cjs-loads NOT MEASURED (exit 3) check:engine-double-contract OK
check:entry-guard OK check:logger-receiver-detach OK check:objectql-double-limit OK
check:objectui-changeset OK check:page-declaration-shape OK check:parse-guard OK
check:pm-half-states OK check:pnpm-filter-targets OK check:published-files OK
check:query-options-erasure OK check:slot-lookup OK check:swallow-census-controls OK
check:test-source-alias OK check:type-check-coverage OK check:type-check-debt NOT MEASURED (exit 3)
check:type-source-resolution OK check:watch-hint-literal OK check:where-matcher OK
check:nul-bytes OK

check:engine-double-contract is green after the ledger row below; its verdict line on
this head:

update doubles: 347 in 313 test file(s) — 247 pinned to ObjectQL.update's dispatch predicate, 100 in the shrink-only baseline (3 admitted by a DECLARED IDataEngine).

Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:

  • check-test-completeness — exit 3, PREREQUISITE NOT MET: it grades a saved
    turbo run test log and none was named. Its own text: "running the family locally,
    record this gate as NOT MEASURED. ⛔ It is not a red".
  • check:dual-build-cjs-loads — exit 3, PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repo pnpm build.
  • check:type-check-debt — exit 3, PREREQUISITE NOT MET: --re-measure refuses
    without the built workspace closure, because a number taken without it "would silently
    measure a DIFFERENT WORLD". (check:type-check-coverage itself is green.)

Repo-wide pnpm lint was not run locally; CI owns it. This is the "not run" case, not a
proven narrowing — no eslint file-count measurement is claimed here.

Adjacent mechanical change, declared

scripts/engine-double-contract.pinned.json is outside the claimed file surface and is in
the diff for exactly one reason: the new test file carries an engine double whose update
already routes through assertEngineUpdateDispatch, and the gate refuses until its
COVERAGE ledger records it. Its own verdict line:

x RETAINED [update]: packages/services/service-job/src/db-job-adapter.once-leader.test.ts pins 1 engine double(s) that the pinned ledger does not record. New pinned coverage is GOOD and nothing is wrong with your change — the ledger just has to learn about it, or it never protects this file. Run `node scripts/check-engine-double-contract.mjs --write` and commit.

Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost — coverage growth only, in the
grow-only direction this ledger is defined to move. git merge-tree --write-tree --name-only origin/main HEAD reports a clean merge with no file listed, and origin/main
is merged into this branch as of the head above.

Clause-②: no

Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEAD filtered to added/removed lines containing export
returns nothing (the only export occurrences are hunk-header context for the unchanged
export class DbJobAdapter). No export is added, removed or renamed; no accept set moves;
packages/spec/** is untouched. This is a services implementation face, exactly as the
ruling's point 5 expected.

Not touched

packages/spec/**, content/docs/releases/**, skills/**, the lock implementation, and
any re-arm / retry / persistence mechanism (ruled out: at-most-once).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8


Generated by Claude Code

`DbJobAdapter.schedule()` routed `once` registrations to `inner`
(`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in
that file, so a one-shot job ran once per replica instead of once per cluster
— the last limb left after #13686 did the same for `interval`, and the
worst-shaped of the three: a one-shot has no later tick during which a
business-level de-duplication marker could win.
Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already
fires through the leader-electing `runScheduled()`) when a cron adapter is
assembled, and keep the registration in `inner` via `register()` so
`trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No
cron adapter assembled => unchanged: `inner.schedule()`, as before.
Crash semantics are at-most-once per cluster, per the maintainer ruling of
2026-09-01, and stated in the docblock: no re-arm and no persistence is added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger
`node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost:
the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is
already routed through `assertEngineUpdateDispatch`. Coverage growth only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence
The pin exists to state the card's repro: two replicas, one deadline, two
`sys_job_run` rows today and one after. A hard throw on the execution count
stops the run before the fence count and the row count are ever reported, so
the ablation that proves the pin can fail printed only the first of the three.
The three pre-row assertions are now soft; the row assertions stay hard.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to listnot a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422packageMentionDocs.

Which tree this was computed on

This run read content/docs from 4f11efe84d9451a602c151875418254027b6ad1a — the merge of head 51fb2de6fe2aa829e299fded5c192deceb27b3fd into base 2514d49f388e898e666ae04f19ba376d04db5422, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

@os-salesClaude

Copy link
Copy Markdown
CollaboratorAuthor

Landing provenance — ready + auto-merge at head 51fb2de6f

  • Card: service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918 (Fixes, closes on merge). Ruling of record: maintainer 2026-09-01 「同意」 on the five points, recorded by the director seat in 13918#issuecomment-5494528879 and quoted verbatim in the PR body.
  • Review path: Clause-② no, so the seat's own ACCEPT rather than an isolated contract review — 13918#issuecomment-5511971555. The declaration was measured on the tree, not read from the report: git diff -U0 origin/main...HEAD | grep -E '^[+-].*\bexport\b' returns nothing; no export added, removed or renamed; no accept-set move; packages/spec/** and content/docs/releases/** untouched.
  • Surface: 4 files — packages/services/service-job/src/db-job-adapter.ts (one condition: once joins interval on the leader-elected path), the new db-job-adapter.once-leader.test.ts (12 pins), the patch changeset for @objectstack/service-job, and the declared adjacent scripts/engine-double-contract.pinned.json row regenerated by the gate's own --write (coverage growth only, 694 rows, 1 added or grown, 0 lost).
  • Ruling conformance: point 1 is the whole code change; point 2's at-most-once-per-cluster semantics are in the schedule() docblock with no re-arm, retry or persistence added; point 3's consumer census is in the PR body and is not a zero (four live registration paths, the sharpest being the wait-node's cold-boot re-arm that every replica ran); point 4's repro is the pin, red before the source edit (handler twice, two sys_job_run rows) and green after; point 5 re-declared from the diff.
  • Serialisation: git merge-tree --write-tree --name-only origin/main HEAD lists no file; the engine-double ledger is the one shared path in the diff and no other open PR in this lane touches it (measured by diffing the open branches against origin/main); CI's No other open PR may claim the same single-writer path is green.
  • Log levels: no new log site at any level. The deliberate asymmetry — interval warns on the cron-less fallback, once does not — is argued in place from frequency (once registrations are per-occurrence, so the same line would be a per-run flood).
  • CI on 51fb2de6f: every check green — Lint & Repo Gates success 15:35:42Z, Type Check · workspace success 15:31:06Z, Test Core (1/6) success 15:43:05Z, aggregate Test Core success 15:43:19Z, Build Core / Dogfood gates / Temporal Conformance / Check Changeset all success.
  • Action: draft → ready and auto-merge enabled at 15:44:13Z.
  • On MERGED: card auto-closes; strip pm:dispatched from service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope #13918; packages/services/service-job/** is released. [finding] service-job scheduler leader election excludes for the DURATION OF THE FIRE, not for the deadline — the lease is released in finally, so replica clock skew larger than the handler's runtime defeats it #14619 (the lease is released in runScheduled's finally, so the exclusion window equals the handler's runtime — pre-existing, shared by all three schedule types) stays with triage for first-touch grading.

Generated by Claude Code

@os-sales
os-sales added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit ca48cf3Sep 2, 2026
35 checks passed
@os-sales
os-sales deleted the claude/issue-13918-once-schedule-leader-election branch September 2, 2026 16:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

service-job: type: 'once' schedules on DbJobAdapter get no leader election either — same routing limb as #13686, deliberately left out of its scope

2 participants

@os-sales@claude