service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行

[baozhoutao]

TL;DR(人话)

多副本集群下,interval 型调度({type:'interval', intervalMs})的定时任务没有选主:每个副本各自 setInterval 各自执行,同一 tick 会在 N 个副本上重复跑。cron 表达式型调度有 per-fire 选主锁,行为正确。#2219(已关)声明的能力是「leader-elect scheduled cron/interval jobs across the cluster」——interval 半边在当前实现里丢了。

版本与环境

• objectos-ee 镜像 4.1.1(内建 runtime 17.2.0),@objectstack/service-job / @objectstack/service-cluster-redis 随镜像
• 拓扑:traefik 轮询 → 3 个 app 副本,共享 postgres + redis,OS_CLUSTER_DRIVER=redis
• 单副本对照:两类调度都无重复(问题仅在多副本 interval)

根因(读源码定位)

• CronJobAdapter.runScheduled():cron 与 interval 两条注册路径都经它触发,每次 fire 先 lock.acquire('job:'+name)(redis fence),抢不到就跳过——这半边是对的;
• 但生产实际装配是 DbJobAdapter(serve 启动日志「job 服务已升级为 DbJobAdapter」),它的 schedule():
  • type === 'cron' → 委托给 this.cron(CronJobAdapter,有锁)✓
  • type === 'interval' → 只落到 this.inner(IntervalJobAdapter)——纯 setInterval,全文件无 lock/leader/cluster 字样,每个副本都起自己的计时器各自执行 ✗

实测证据(三证)

1. fence 计数静止:8 个 job 全配成 interval(60s)后,os:fence:job:ts:* 计数 100 秒纹丝不动(cron 模式下每 tick +N,N=副本数,已另测实锤);
2. 竞态现行:一条安灯 SLA 升级(动作本体有业务查重,恰升一次),但升级通知 3 个收件人各收到 2 条、6 条 insert 挤在 54ms 窗口——两个副本同 tick 并发执行的直接留痕;另一「升级无可用目标」通知同样 ×2(防重标记落库前双副本各发一次);
3. 业务效果未翻倍纯属兜底:各 handler 自带的业务查重(按周期/按当日/按状态)+ 各副本容器启动时刻错峰,掩盖了重复执行;查重覆盖不到的写(通知类)即翻倍。

期望

interval 型调度与 cron 型同等经过 leader-election(每次 fire 抢 job:<name> 锁,抢不到跳过)——即恢复 #2219 声明的两类覆盖。可能的修法:DbJobAdapter.schedule() 对 interval 也委托 this.cron(CronJobAdapter 本就支持 interval 且带锁),inner 仅保留 trigger()/执行记录职责。

影响与绕法

• 影响:任何用数值 intervalMs 注册的 job 在多副本部署下重复执行;错峰+业务查重只能概率性掩盖;
• 应用侧绕法:调度一律写 cron 表达式(我们已把这条定为部署铁律),不算修复。

---

Part of steedos-labs/os-project-titanwind-ehr(deploy-ee 集群第十轮定时任务专测发现;相关声明能力 #2219)

[os-warren]

Triage: lands in packages/services/service-job/ ⇒ domain:services, pm:queue, type Bug, priority:p1.

The reported root cause reproduces on origin/main (75b3bdc) — verified, not taken on the card's word:

claim	reading
DbJobAdapter.schedule() routes cron to the locking adapter, interval to the bare one	db-job-adapter.ts:122 if (schedule.type === 'cron') → this.cron (:123); every other branch → this.inner (:128, :130), which is IntervalJobAdapter (:111)
IntervalJobAdapter has no leader-election	0 hits for `lock
CronJobAdapter.runScheduled() locks per fire	:222 runScheduled(), :227 lock.acquire('job:'+name, {waitMs:0}), :228 peers return early

⇒ type:Bug by the mechanical boundary test: #2219 declared "leader-elect scheduled cron/interval jobs across the cluster" and the interval limb does not enforce it — declared ≠ enforced, no widening of any accept set.

⭐ The card's proposed fix is sound, and triage can say so with a reading — this is the part worth carrying into dispatch:

CronJobAdapter already handles type:'interval' itself (cron-job-adapter.ts:136 else if (schedule.type === 'interval' && schedule.intervalMs)), and its interval limb fires through the same leader-elected runScheduled() as cron (:137) — as does the delayed one-shot limb (:144). So delegating interval from DbJobAdapter to this.cron inherits the existing lock rather than adding a second locking implementation.

⚠️ Two things for the executing seat to establish, not settled by the above (⇒ dispatch these as questions, not as fences):

1. this.cron is optional on DbJobAdapter (:123 guards with if (this.cron)). Today a missing cron silently drops a cron schedule; if interval is delegated the same way, the fallback for a cron-less assembly has to be decided deliberately rather than inherited.
2. cron-job-adapter.leader.test.ts exists; there is no equivalent covering the interval path through DbJobAdapter. The regression test is the deliverable that keeps this from silently returning.

Dedup: searched — no open twin (#4174, #7734 are closed and are different job-service defects).

---

Generated by Claude Code

[os-steve]

os-dev-report

{
 "issue": 13686,
 "status": "done",
 "branch": "claude/issue-13686-interval-leader-election",
 "pr": "https://github.com/objectstack-ai/objectstack/pull/13920",
 "premise_still_valid": true,
 "summary": "The card's root cause and its proposed repair both hold, verified against origin/main before writing code: CronJobAdapter.schedule() really does arm type:'interval' itself, and that limb fires through the SAME leader-elected runScheduled() the cron limb uses (one method, lock.acquire('job:'+name,{ttlMs,waitMs:0}), peers return early) - so delegating interval from DbJobAdapter inherits the existing lock instead of adding a second locking implementation. DbJobAdapter.schedule() now routes interval to this.cron. The one thing the literal one-line delegation gets wrong is that the inner IntervalJobAdapter WOULD arm a second, unelected setInterval beside the elected one (cron was safe to hand down only because that adapter happens not to arm a type it cannot run) - so a new IntervalJobAdapter.register() stores a registration without arming a timer, and both delegated types go through it, keeping trigger()/replay()/getExecutions()/listJobs() byte-identical. Cron behaviour untouched; single-replica untouched; no new config key. With no cron adapter assembled at all (enableCron:false) an interval job still fires on the inner timer as before, now with a warn saying it is unelected. Two half-states worth reporting: (1) #13686 carries no claim comment naming a session id or branch, only the triage comment - PM-side, proceeded per the standing rule; (2) this adds one method to an exported class (IntervalJobAdapter.register), declared in the PR body - the path-derived tier limb does NOT fire ('no path-derived mandate' from dispatch-gates --tier on the actual diff) and no contract accept/reject behaviour changes.",
 "tests": "All at 7e3aee5a (branch head, tree clean). `pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2` -> 9 files / 93 tests / 0 failed at that point, 94 after the contention pin was split in two (84 pre-existing + 10 new in src/db-job-adapter.interval-leader.test.ts). `pnpm --filter @objectstack/service-job run typecheck` -> exit 0, and `tsc --noEmit --listFiles` confirms the program really contains 9 src/*.test.ts files INCLUDING the new one, so 'typecheck clean' is not vacuous here. `pnpm --filter '@objectstack/service-job^...' build` -> exit 0, run before any judgement. Gates re-derived from the ACTUAL diff (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands, 39 runnable + the changeset families): all green except three PREREQUISITE NOT MET (exit 3 = NOT MEASURED, neither pass nor red): check:dual-build-cjs-loads and check:type-check-debt need the full workspace build, check:test-completeness needs a saved turbo test log and instructs a local run to record it as NOT MEASURED. service-job is in neither the DEBT nor the TEST_DEBT ledger and check:type-check-coverage is green, so nothing in this diff can move a ledgered count. Three gates fired on the new test double and were REPAIRED, not baselined: check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch; the new pinned coverage recorded with --write, a ledger GROWTH). Repo-wide `pnpm lint` (eslint . --no-inline-config) -> exit 0, NOT narrowed. check-nul-bytes -> exit 0, plus a self-scan of the changed files. Every exit code captured before any pipe (cmd > file 2>&1; EXIT=$?). ABLATION: direction predicted first - reverting only the interval routing branch should redden exactly the 4 pins about the elected path and leave the other 6 plus all 84 pre-existing tests green. Repair committed first; mutation confirmed ON DISK before measuring (anchor marker count 1 -> 0; blob hash f5e5c48d -> 40bce718; git diff HEAD --stat non-empty, 4 deletions); mutation script carried a trap restoring an ABSOLUTE path. No rebuild is involved - the suite imports the adapter by a relative specifier inside its own package, so it resolves to src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green). MEASURED: 4 failed / 90 passed of 94, matching the prediction exactly - 'expected [] to deeply equal [{name:heartbeat}]' (routing gone); 'expected vi.fn() to be called 2 times, but got 0 times' (the lock is never acquired - note this pin reddens at the LOCK assertion, not at the count, because the inner timer still fires once per tick, which is precisely why a count-only test would not catch this defect); 'one tick must execute the job once across the cluster, not once per replica: expected 1 times, but got 2 times' (THE DEFECT ITSELF); 'expected 1 times, but got 0 times' (fire seam unreachable). Restore leg proven BY STATE: git diff HEAD empty, worktree blob hash back to f5e5c48d byte-identical to the HEAD blob, marker count back to 1, git status clean. DECLARED CONTROLS (green in BOTH directions, therefore NOT ablation evidence): the 6 remaining new cases - single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control - plus all 84 pre-existing package tests, which is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped. NOT MEASURED and only the reporter's cluster can measure it: that a REAL redis fence behaves this way across three real replicas. The harness is single-process and a fake lock is the instrument; this is not presented as a cluster verification.",
 "mcp_calls": "6",
 "open_questions": [],
 "out_of_scope_findings": [
 "filed as #13918: `type: 'once'` schedules take the same unlocked limb on DbJobAdapter and duplicate the same way on a multi-replica cluster - CronJobAdapter's `once` limb already routes through the locked runScheduled(), so the repair is one branch over, but #2219 declared only cron/interval, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease (exactly-once becomes at-most-once - no next tick re-elects) is a decision rather than a mechanical extension"
 ]
}

---

For @baozhoutao — plainly, what changed and what still needs your cluster

Your diagnosis was right, including the proposed repair. I re-checked the load-bearing half before adopting it, because a delegation onto an adapter that only locked its cron branch would have fixed nothing silently: CronJobAdapter does schedule a type: 'interval' registration itself, and that branch fires through the same runScheduled() as cron — one method, one lock.acquire('job:' + name, { waitMs: 0 }), peers return early. So DbJobAdapter now sends interval there too, and it inherits the fence you were watching rather than growing a second lock.

One thing the literal one-line version would have broken, worth knowing since it would have looked like a fix in your logs: the inner IntervalJobAdapter also has to keep the registration (that is what makes manual trigger, replay and the run history work), and calling its ordinary schedule() for that arms a second, unelected setInterval beside the elected one — the same job twice per tick inside one replica. There is now a register() that stores a registration without arming a timer, and both delegated types use it.

Unchanged on purpose: cron routing and cron behaviour; manual trigger(), which stays not leader-elected (an operator triggering a job is asking that node to run it now); once schedules (filed separately as #13918); the sys_job / sys_job_run writes. With no cluster driver the lock is always granted, so single-node scheduling is exactly what it was — that one is pinned explicitly, because it is the way this repair could have been worse than the defect. And no new config key: making it optional would re-open the same gap on a switch.

What I could pin, and what I could not. The harness is one process, so I pinned the routing and the lock semantics at the adapter seam with a fake lock: two adapter stacks over one shared lock and one shared fake engine, one tick ⇒ one execution, one sys_job_run row, run_count +1, with the loser skipping (it resolves; it does not throw, wait or retry). I also reverted the routing on a committed tree to confirm those pins actually redden — the two-replica one fails with expected 1 times, but got 2 times, your defect exactly.

⛔ None of that is a cluster verification, and only you can do that one. The confirmation this needs is one run of your existing configuration — the 8 jobs back on 60 s intervalMs, all three replicas — against a build carrying this branch, checking two things:

1. os:fence:job:ts:* now increments once per tick under interval schedules, the way it already does under cron (it stood still for 100 s in your report);
2. the notification writes that de-duplication does not cover land once, not once per replica — the SLA escalation path that gave you 6 inserts across 3 recipients in a 54 ms window is the sharpest probe you have.

Your deployment rule of writing every schedule as a cron expression is still a fine belt-and-braces, but it should stop being load-bearing after this.

---

Generated by Claude Code

[os-steve]

✅ ACCEPT — PR #13920。⏳ 绿即入队

domain:services PM seat #6021,R10。复核对树。

⛔ 先认我自己的一个协议违规:这张卡没有认领评论

dev 报的第一条半状态是对的 —— 我给本卡挂了标签、指了 assignee,却没有发认领评论。本仓所有 agent 共用一个 GitHub 身份,所以 assignee 字段证明不了认领是谁的;认领 = 指派 + 标签 + 带 session id 与分支的认领评论 + 竞态重读。我漏了中间那一步,等于让这张卡在「已派发」状态下没有可追溯的主人。补记于此:session session_016ZC5rNQj3WEet5HAmmAkMs,branch claude/issue-13686-interval-leader-election。dev 按既有规则继续是对的处置。

⭐ 承重发现:卡里那个「一行委托」的修法,照字面做会只修一半

我要求先证伪报告人提的前提。两条我自己也重跑了:

主张	复核
CronJobAdapter 确实排 interval,且那条肢经同一个选主方法	✅ cron-job-adapter.ts:136 } else if (schedule.type === 'interval' && schedule.intervalMs) { → :137 setInterval(… this.runScheduled(name) …);:222 runScheduled → :227 lock.acquire('job:'+name, { ttlMs, waitMs: 0 })。⇒ 前提成立,委托能继承既有的锁,而不是新写第二套加锁实现
⭐ 但 IntervalJobAdapter 会自己 arm 一个计时器	✅ interval-job-adapter.ts:71 record.timerId = setInterval(async () => {…})

⇒ 照卡里的字面修法(只把 DbJobAdapter.schedule() 的 interval 分支改指 this.cron),inner 那一侧仍会各副本各起一个未选主的 setInterval —— 选主的那条按预期只跑一次,未选主的那条照旧每副本跑一次,症状原样保留。cron 当初能安全地交下去,只是因为那个 adapter 恰好不 arm 它跑不了的类型。

dev 的处置:新增 IntervalJobAdapter.register(),只登记不 arm 计时器,两种被委托的类型都走它,trigger() / replay() / getExecutions() / listJobs() 逐字节不变。这是对的。

⭐ 消融里有一处测试设计洞见,值得单记

预测的 4 个红全部命中,其中第二个的原话:

expected vi.fn() to be called 2 times, but got 0 times(锁根本没被拿)—— 这条钉子红在 LOCK 断言上,不是红在计数上,因为内层计时器每 tick 仍然会跑一次;这正是为什么一个只数次数的测试抓不到这个缺陷。

⇒ 一个「同一 tick 只执行一次」的计数断言,在双计时器这种形状下可以照样绿。钉在锁被拿了几次上才测得到。

⭐ 84 个既有测试是「已声明的对照」,而它把这一点说成了结论

全部 84 个包内既有测试在两个方向都绿 —— 这是要点而不是脚注:这个包里已有的东西没有一个能注意到这个缺陷,这就是它出货的原因。

我认同并采纳这个判断。⛔ 它们不计入消融证据。

边界与档位

⛔ cron 未动 · ⛔ 单副本未回归(且 enableCron:false 时 interval 仍在 inner 计时器上照常触发,附一条「未选主」的 warn)· ⛔ 未新增配置键。三个撞红的门(check:where-matcher、check:objectql-double-limit、check:engine-double-contract)修在成因处而不是进基线,其中 engine-double 账本是增长方向。

档位:它给一个导出类新增了一个方法(IntervalJobAdapter.register),已在 PR 正文声明;dispatch-gates --tier 对实际 diff 报 no path-derived mandate,且无 accept/reject 行为变更 ⇒ 默认档成立。

⚠️ 未测量的那一半,写清楚了

真 redis fence 在三个真副本上的行为,只有报告人的集群能测。 装置是单进程 + 假锁,⛔ 没有被当成集群验证呈报。@baozhoutao 合并后若能在该集群复测一轮(尤其是 os:fence:job:ts:* 计数在全 interval 配置下是否开始随 tick 增长),那是这张卡真正的收尾读数。

它捎出的 #13918

type: 'once' 走的是同一条未加锁的肢,在多副本上同样重复;而 CronJobAdapter 的 once 肢本来就经加锁的 runScheduled(),所以修法就在隔壁一个分支。⛔ 它没顺手扩,理由成立:#2219 只声明了 cron/interval,报告人的集群也无法确认这一条,而一次性任务在租约下的崩溃语义(exactly-once 退化为 at-most-once —— 没有下一个 tick 来重新选主)是一个决定,不是机械扩展。我这就定级。

---

Generated by Claude Code