Uh oh!
There was an error while loading. Please reload this page.
fix(service-job): leader-elect type: 'once' schedules on DbJobAdapter - #14618
Conversation
`DbJobAdapter.schedule()` routed `once` registrations to `inner` (`IntervalJobAdapter`), a bare `setTimeout` with no cluster lock anywhere in that file, so a one-shot job ran once per replica instead of once per cluster — the last limb left after #13686 did the same for `interval`, and the worst-shaped of the three: a one-shot has no later tick during which a business-level de-duplication marker could win. Route `once` to `this.cron` (`CronJobAdapter`, whose own `once` branch already fires through the leader-electing `runScheduled()`) when a cron adapter is assembled, and keep the registration in `inner` via `register()` so `trigger()`, `replay()`, `getExecutions()` and `listJobs()` are unaffected. No cron adapter assembled => unchanged: `inner.schedule()`, as before. Crash semantics are at-most-once per cluster, per the maintainer ruling of 2026-09-01, and stated in the docblock: no re-arm and no persistence is added. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…d ledger `node scripts/check-engine-double-contract.mjs --write` — 1 row added, 0 lost: the `update` double in the new `db-job-adapter.once-leader.test.ts`, which is already routed through `assertEngineUpdateDispatch`. Coverage growth only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
…ence The pin exists to state the card's repro: two replicas, one deadline, two `sys_job_run` rows today and one after. A hard throw on the execution count stops the run before the fence count and the row count are ever reported, so the ablation that proves the pin can fail printed only the first of the three. The three pre-row assertions are now soft; the row assertions stay hard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
📓 Docs Drift Check1 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 3 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 4f11efe84d9451a602c151875418254027b6ad1a && git checkout 4f11efe84d9451a602c151875418254027b6ad1a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2514d49f388e898e666ae04f19ba376d04db5422 51fb2de6fe2aa829e299fded5c192deceb27b3fd && git checkout -B drift-repro 2514d49f388e898e666ae04f19ba376d04db5422 && git merge --no-ff 51fb2de6fe2aa829e299fded5c192deceb27b3fd
node scripts/docs-audit/affected-docs.mjs --json 2514d49f388e898e666ae04f19ba376d04db5422 |
os-sales
commented
Sep 2, 2026
Landing provenance — ready + auto-merge at head |
Uh oh!
There was an error while loading. Please reload this page.
Fixes#13918
DbJobAdapter.schedule()decides which adapter owns a scheduled fire, and only thatchoice decides whether the fire is leader-elected:
CronJobAdaptertakes the clusterlock in
runScheduled(),IntervalJobAdapterholds no lock at all.cronwas routedto the electing adapter from the start and
intervalsince #13686 —oncewas thelimb still left over, so a one-shot job ran once per replica instead of once per
cluster. This routes it the same way, one branch over.
Maintainer ruling of record (2026-09-01, 「同意」) — quoted verbatim, untranslated
Recorded by the director seat in 13918#issuecomment-5494528879:
The change
One condition, in
DbJobAdapter.schedule():inner.register()stores without arming, sotrigger(),replay(),getExecutions()and
listJobs()keep answering from one place, and one process never holds an electedtimer beside an unelected one. Everything else in the diff is the docblock and the pins.
Crash semantics, per ruling point 2, stated in the
schedule()docblock:onceisat-most-once per cluster. Election decides who fires, never that the fire
survives — there is no second deadline, so a leader that dies mid-fire loses it and
nothing re-arms it. That takes nothing away: the previous unelected
setTimeoutwas notpersisted either and the same crash lost it on every replica at once. No re-arm, no
"release the lease only on success", no persistence.
One asymmetry worth naming, because it is a deliberate omission and not an oversight.
The cron-less fallback (
enableCron: false, or cron construction threw) is unchanged foronce: it still fires oninner's own timer.intervalemits awarnin that case;oncedeliberately does not, and the reason is frequency, not importance. Intervalregistrations are per-plugin-startup and countable;
onceregistrations areper-occurrence — the automation wait-node arms one per suspended flow run — so the same
line there is a per-run log flood, and writing a
warnon a hot path is how everyonelearns to skim
warn. The docblock says this in place. No new log site of any level isadded by this PR.
Consumer census (ruling point 3)
Method:
git grep -n "type: 'once'"andgit grep -n 'type: "once"'overpackages,examples,apps,skills,content,scripts, then each hit classified by readingits call site. Positive control: the same method run for
type: 'interval'returnsthe registrations #13686 was about (
plugin-approvalsescalation sweepapprovals-plugin.ts:329,plugin-reportsreports-plugin.ts:155), so a zero herewould have been a real zero. It is not a zero.
IJobService.schedule?packages/services/service-automation/src/builtin/wait-node.ts:259packages/services/service-automation/src/builtin/wait-node.ts:440rearmSuspendedWaitTimerson cold bootpackages/triggers/trigger-schedule/src/schedule-trigger.ts:140normalizeSchedulefor a schedule-triggered flow declaringatScheduleTrigger.start()packages/runtime/src/job-schedule.ts:62toBoundaryJobSchedulefor an app-declaredoncejobapp-plugin.ts:1017examples/app-showcase/src/automation/flows/index.ts:576packages/runtime/src/job-schedule.test.ts,packages/spec/src/system/job.test.ts,packages/triggers/trigger-schedule/src/schedule-trigger.test.ts,packages/services/service-automation/src/builtin/wait-node.test.tscontent/docs/automation/jobs.mdx:88,content/docs/references/system/job.mdx:58,packages/services/service-job/README.md:75,packages/triggers/trigger-schedule/README.md:44packages/services/service-automation/CHANGELOG.md(3 hits)Four live production registration paths, not zero. The re-arm one is the sharpest:
every replica of a multi-replica deployment re-arms the same suspended run's wake timer
at boot, and before this change every one of them fired it.
Premise checks (verified on
origin/mainbefore the first edit)schedule()still routesoncetothis.inner.schedule(...)whilecron/intervalgo tothis.cron+inner.register()oncefell through to theelselimbCronJobAdapter.schedule()still has its owntype === 'once' && schedule.atbranch armingsetTimeout(() => { void this.runScheduled(name); }, delay), andrunScheduledtakeslock.acquire('job:' + name, { waitMs: 0 })packages/spec/**andcontent/docs/releases/**untouchedHypotheses the dispatch declared
oncelimb mirroring theintervallimb; the no-cron fallback staysinner.scheduletrigger()/replay()/getExecutions()/listJobs()still answer for a delegatedoncejobcancel()still cancels a delegatedoncejob on both adaptersatgives zero executions, zeroacquirecalls,listJobs()empty andsys_job.active === falserunScheduledfinally; the diff adds no timer, no retry and no storeTests
Head:
51fb2de6f. All runs underscripts/pm/os-verify-lock.sh(shared container),exit codes captured before any pipe.
P3 reproduction, before the source edit —
vitest run src/db-job-adapter.once-leader.test.ts:After the fix —
pnpm --filter @objectstack/service-job test(whole package):Type check —
pnpm --filter @objectstack/service-job exec tsc --noEmit --listFiles,exit 0. Coverage measured rather than assumed: the
--listFilesoutput names both editedfiles (
db-job-adapter.tsanddb-job-adapter.once-leader.test.ts), 407 files total, so"typecheck clean" really does cover the new test file. The package's
tsconfig.jsonincludes
srcand excludes onlynode_modules/dist.The twelve new pins in
db-job-adapter.once-leader.test.ts, all deterministic and all inone process (like their
intervalsibling, this is not and cannot be a cluster test — itpins ROUTING and LOCK SEMANTICS at the adapter seam):
onceregistration reaches the cron adapter, and ten deadlines' worth offake time produces zero runs from any timer
DbJobAdapterarmed itself;DbJobAdapterstacks, one fake engine, oneshared lock, one
{ type: 'once', at }each, oneadvanceTimersByTimeAsync⇒ oneexecution, two
acquirecalls withwaitMs: 0, onesys_job_runrow,run_count: 1;releases its lease;
deadlines add no executions);
registered for manual triggering;
trigger()while a peer holds the lock,replay()+getExecutions(),listJobs();cancel()before the deadline, on both adapters;sys_jobupsert for aonceschedule (schedule_type: 'once', expression =at);cronandintervalrouting unchanged.Ablation (on the committed tree)
Mutation: restore the pre-fix routing (
(schedule.type === 'interval' || schedule.type === 'once') && this.cronback to
schedule.type === 'interval' && this.cron). No rebuild is involved on eitherleg and none is owed: the pins import the subject relatively
(
import { DbJobAdapter } from './db-job-adapter.js'), so vitest resolves it to thepackage's TypeScript source, never through the package
exportstodist/. That is notan assumption — it was demonstrated in this run: the suite went red then green across a
source edit with no
service-jobbuild in between.Both legs proved on disk before anything was measured, by occurrence counts anchored on
the exact text being changed plus the blob hash:
Mutated leg —
vitest run src/db-job-adapter.once-leader.test.ts, exit 1:That third line is the card's
sys_job_runcount, and it is why the three pre-rowassertions in that one test are
expect.soft: unrouted, the two replicas write tworows for one deadline, and routed they write one. The middle line is the defect at its
root — the fence is not merely lost, it is never consulted at all (
acquirecalled zerotimes).
Mutated-leg control —
vitest run src/db-job-adapter.interval-leader.test.ts, exit 0:The mutation touches only the
oncelimb, so #13686's pins must stay green — they do,which is what makes the red above attributable to this change rather than to the harness.
Restore leg, proved the same way rather than by an exit code (
git checkout HEAD -- ...with an absolute path, plus a
trap ... EXIT INT TERMso a container cap kill cannotleave a mutated tree behind):
Restored leg — the same file, exit 0:
Gates
Derived on the head being pushed, from the tool rather than a hand-written list:
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands. Thefamily was re-derived after the ledger row below entered the diff — that added seven
gates (
check:agent-test-spelling,check:bash32-floor,check:cli-command-ids,check:entry-guard,check:parse-guard,check:pnpm-filter-targets,check:watch-hint-literal) which were then run too.All 44 run on that head, each exit captured before any pipe:
41 exit 0, 3 exit 3 (
PREREQUISITE NOT MET— NOT MEASURED), 0 red.check:engine-double-contractis green after the ledger row below; its verdict line onthis head:
Three gates are NOT MEASURED, each by its own printed verdict, and none of them is a
red:
check-test-completeness— exit 3,PREREQUISITE NOT MET: it grades a savedturbo run testlog and none was named. Its own text: "running the family locally,record this gate as NOT MEASURED. ⛔ It is not a red".
check:dual-build-cjs-loads— exit 3,PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/. Needs a whole-repopnpm build.check:type-check-debt— exit 3,PREREQUISITE NOT MET:--re-measurerefuseswithout the built workspace closure, because a number taken without it "would silently
measure a DIFFERENT WORLD". (
check:type-check-coverageitself is green.)Repo-wide
pnpm lintwas not run locally; CI owns it. This is the "not run" case, not aproven narrowing — no eslint file-count measurement is claimed here.
Adjacent mechanical change, declared
scripts/engine-double-contract.pinned.jsonis outside the claimed file surface and is inthe diff for exactly one reason: the new test file carries an engine double whose
updatealready routes through
assertEngineUpdateDispatch, and the gate refuses until itsCOVERAGE ledger records it. Its own verdict line:
Regenerated with that exact command, never hand-edited:
694 (file, verb) row(s), 1 added or grown, 0 lost— coverage growth only, in thegrow-only direction this ledger is defined to move.
git merge-tree --write-tree --name-only origin/main HEADreports a clean merge with no file listed, andorigin/mainis merged into this branch as of the head above.
Clause-②: no
Declared from the actual diff, not from the expectation.
git diff -U0 origin/main...HEADfiltered to added/removed lines containingexportreturns nothing (the only
exportoccurrences are hunk-header context for the unchangedexport class DbJobAdapter). No export is added, removed or renamed; no accept set moves;packages/spec/**is untouched. This is a services implementation face, exactly as theruling's point 5 expected.
Not touched
packages/spec/**,content/docs/releases/**,skills/**, the lock implementation, andany re-arm / retry / persistence mechanism (ruled out: at-most-once).
🤖 Generated with Claude Code
https://claude.ai/code/session_01AUF1NoViznQK32gqpK8wS8
Generated by Claude Code