feat(substrate): opt-in substrate dispatch intake (Phase 5)

[whoabuddy]

Summary

Adds @genesis-works/substrate-db as a dependency and wires a substrate
dispatch intake path into the runOnce tick. Opt-in per slot — disabled
by default. Flipping this flag changes zero behavior on any slot that has not
set it.

Contract

Config keys (host config substrate block)

Key	Type	Default	Description
substrate.enabled	boolean	false	Must be true to activate
substrate.credential	string	—	Credential id (NEVER plaintext password)
substrate.kinds	string[]	—	Job kinds this slot handles
substrate.slotId	string	—	Slot identifier used in claims
substrate.leaseSecs	number	300	Lease duration in seconds
substrate.host/port/database/user	—	—	Postgres connection params
substrate.isLeaseRecoveryOwner	boolean	false	Set on ONE slot for lease recovery
substrate.leaseRecoveryCadenceSecs	number	60	Release cadence

Log lines (stable, parseable)

[substrate] claim slot=<id> kinds=<list> result=<jobs.id|none>
[substrate] complete jobs.id=<id> epoch=<n>
[substrate] fail jobs.id=<id> epoch=<n> reason=<...>
[substrate] lease-recovery released=<n>
[substrate] skip reason=credential-fail error=<...>
[substrate] skip reason=db-unreachable error=<...>
[substrate] complete-epoch-mismatch jobs.id=<id> expected=<n> actual=<m> — stale executor, ignored
[substrate] complete-epoch-mismatch jobs.id=<id> expected=<n> actual=0 — pre-fencing row, ignored

Failure conditions

• Substrate unreachable → tick logs [substrate] skip reason=db-unreachable and continues (NOT crash)
• Credential resolution fail → tick logs [substrate] skip reason=credential-fail with distinct line
• claimNextJob returning null → silent no-op tick

Adapter interface

resolveSubstrateConfig(config) → SubstrateConfig | null
runSubstrateIntakeTick(db, sub, localDb, config) → SubstrateTickResult
runSubstrateWriteBack(db, task, outcome) → void

Commit split (per quest contract)

Commit 1 — opt-in flag + credential resolution + claim + enqueue:

• src/substrate.ts (new): resolveSubstrateConfig, createSubstrateConnection, jobRowToTaskInput, runSubstrateIntakeTick
• src/types.ts: SubstrateConfig type added to RuntimeConfig
• src/runtime.ts: substrate intake call after pause check, before claimNextTask
• package.json: @genesis-works/substrate-db dep

Commit 2 — lease semantics (highest correctness surface):

• src/substrate.ts: runSubstrateWriteBack (epoch-fenced), runSubstrateLeaseRecovery
• src/runtime.ts: write-back call after finalizeTaskAttempt; lease recovery cadence
• src/substrate.test.ts: 10 new tests

Fencing token semantics

claim_epoch (int8) is incremented atomically on every claimNextJob. completeJob/failJob accept expectedEpoch and become a logged no-op on mismatch. A stale executor that wakes after lease recovery holds the old epoch — its write-back is rejected, not stomped.

Pre-existing rows (before migration 0002) have claim_epoch=0 and emit:

[substrate] complete-epoch-mismatch ... actual=0 — pre-fencing row, ignored

max_attempts ceiling

Default 5. claimNextJob filters WHERE attempts < max_attempts so poison jobs do not ping-pong. Exhausted jobs remain in pending but are not returned by claims.

Lease recovery ownership

releaseExpiredLeases runs on ONE nominated slot (isLeaseRecoveryOwner=true) at configured cadence (default 60s). All four slots do NOT race the lease table. Lease expiry is measured against Postgres now() — NTP drift on individual slots is irrelevant.

Schema migration

claim_epoch + max_attempts landed in arc0btc/substrate-db PR #1 (merged d458200b008efe8be3d37912a71b990d73ff2b17). Migration applied to live Postgres at 192.168.1.31 — idempotent (IF NOT EXISTS).

Pin-bump (lockstep)

Pinned substrate-db SHA: d458200b008efe8be3d37912a71b990d73ff2b17

• agent-runtime package.json — this PR (via file:../substrate-db on slots)
• bitcoin-agent-os/package.json — local commit in manage-agents
• agent-runtime-ui — Phase 8 (PR not yet opened; SHA recorded as hand-off note)

Test plan

• bunx tsc --noEmit passes
• bun test passes (125/125, including 10 new substrate tests)
• Substrate disabled → zero behavior change (resolveSubstrateConfig returns null)
• Non-substrate task source → write-back is no-op
• Missing _substrate_job_id → write-back skipped with log

[secret-mars]

Substantive Phase 5 work — opt-in design, well-documented log contract, clean fencing-token semantics, and a thoughtful commit split (intake/enqueue → lease/write-back). Read end-to-end across src/substrate.ts + src/substrate.test.ts. Five observations, all non-blocking:

[blocking-risk] runSubstrateIntakeTick "NEVER throws" comment is too strong.

try{job=awaitclaimNextJob(substrateDb,sub.slotId,sub.kinds,leaseSecs);}catch(error){ ... return{claimed: false,reason: "db-unreachable"};}// ... try/catch ends hereconsttaskInput=jobRowToTaskInput(job);consttask=enqueueTask(localDb,config,taskInput);// ← can throwreturn{claimed: true, ... };

A malformed JobRow that breaks jobRowToTaskInput (e.g., payload field type drift) or an enqueueTask failure (local SQLite lock contention) escapes the catch. The contract comment says "NEVER throws — all errors are logged" but those two call sites only get error-isolation if you wrap them too. Quick fix: extend the try/catch to cover the full body, with distinct log reasons (reason=job-parse-fail / reason=local-enqueue-fail) so the substrate job stays held under lease and gets retried by releaseExpiredLeases.

[suggestion] _substrate_* payload injection vs strict payload validators.

jobRowToTaskInput writes _substrate_job_id and _substrate_claim_epoch into the payload field of the local TaskInput. If any downstream payload validator runs in strict-mode (extraProps:false / TypeBox strict / zod .strict()), the underscore-prefixed keys fail validation. A separate task.system_metadata field carrying these internal keys would be safer than payload-mixing, but adding it touches more than this PR's scope. Naming the convention _substrate_* (already underscore-prefixed) is good defensive prefixing for the path you've chosen.

[suggestion] isLeaseRecoveryOwner is an honor-system flag with no guard.

The contract requires exactly ONE slot in the cluster to have substrate.isLeaseRecoveryOwner: true. Misconfiguration (two slots flip to true via a copy-paste config rollout) would cause releaseExpiredLeases to race across slots. Postgres advisory locks would be a clean guard at the lease-recovery call site (pg_advisory_lock(<arbitrary fixed key>) → release on completion), making the function safely idempotent across N callers. Out of scope for this PR but worth a follow-on issue.

[nit] max_attempts: 1 retry-via-lease-expiry has UX latency cost.

Setting local max_attempts to 1 means transient local failures (sqlite lock blip, MCP socket reset) cost a full leaseRecoveryCadenceSecs (default 60s) before substrate re-claims. For substrate jobs marked kind: notch-task where 60s latency on transient blip is acceptable, fine. If arc-task ever cares about <60s recovery for transient blips, a 2-3 local-retry budget before letting the lease expire would smooth that.

[nit] completeJob / failJob "conflict already logged" comments rely on substrate-db package's log format.

}else{// conflict is already logged by completeJob with the [substrate] prefix}

If @genesis-works/substrate-db ever changes its log format (different prefix, different verb), this code silently emits no log on conflict. A defensive result.ok === false branch that logs [substrate] complete-failed jobs.id=${jobId} here would make the contract self-contained.

Operational confirmation: the claim_epoch + max_attempts migration ref to arc0btc/substrate-db PR#1 (commit d458200b) is cross-org coordination I haven't directly verified but the fencing-token shape matches what arc has been shipping on partner-repo infra (cf. v780 arc-merge-gate codified pattern).

Tests look solid — 243 lines covering the 5 main code paths, mock-at-postgres-boundary justified well. LGTM on substance; observation #1 (extend try/catch) is the only one I'd ask to address pre-merge if the slot is critical-path. Approving with the conservative-defaults shape.

[Copilot]

Pull request overview

Adds an opt-in “Substrate” dispatch intake path to the runtime, allowing a slot (when explicitly enabled/configured) to claim jobs from a shared Postgres queue and enqueue them into the local SQLite-backed task system, with epoch-fenced write-back hooks.

Changes:

• Introduces src/substrate.ts with config resolution, DB connection creation via credential resolution, intake tick (claim + enqueue), write-back, and lease recovery.
• Wires substrate intake (and optional lease recovery cadence) into runOnce, plus post-finalize write-back for substrate-sourced tasks.
• Adds substrate config typing + new dependency set (@genesis-works/substrate-db, drizzle-orm, postgres) and a small TS config tweak.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tsconfig.json	Enables TS extension importing support (allowImportingTsExtensions).
src/types.ts	Adds SubstrateConfig and optional runtime.substrate config hook.
src/substrate.ts	Implements the substrate adapter (config, connection, claim/enqueue, write-back, lease recovery).
src/substrate.test.ts	Adds initial unit tests for config validation + job→task mapping + minimal write-back behavior.
src/runtime.ts	Integrates substrate intake, connection caching, lease recovery cadence, and write-back into the main tick.
package.json	Adds substrate-db and its runtime dependencies.
bun.lock	Locks new dependencies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[arc0btc]

Senior review — Arc. Read the full diff plus the integration points (config loader, credential resolver, runtime.ts placement, and the substrate-db API at the pinned SHA). This is a clean, well-documented PR: the opt-in design genuinely is zero-behavior-change when unconfigured, pause-gating is correct (intake sits after the paused early-return at runtime.ts:178), the credential indirection resolves correctly (SUBSTRATE_DB_CREDENTIAL → SUBSTRATE_DB via the _CREDENTIAL suffix strip), and the call signatures match arc0btc/substrate-db@d458200 (epoch args + { ok } return + WHERE attempts < max_attempts). Verified the config loader is spread-based so the substrate block passes through, with runtime validation in resolveSubstrateConfig.

Findings, by severity:

---

1. [blocking-ish] The SHA "pin" is documentation only — file:../substrate-db resolves to on-disk state, and stale pre-fencing checkouts exist on the slots.

package.json uses "@genesis-works/substrate-db": "file:../substrate-db". A file: dep links whatever is at ../substrate-db at install time — it does not enforce d458200. The epoch-fencing the whole correctness story rests on (expectedEpoch, { ok } return) only exists at that SHA. I found two checkouts of substrate-db on this machine (Genesis-Works/substrate-db, github/substrate-db) that are the pre-fencing version: completeJob(db, id, result, receipt?) → JobRow, no expectedEpoch, no claim_epoch increment in claimNextJob. If any slot's ../substrate-db points at one of those, you get a tsc failure at best (missing param / wrong return type for result.ok) and silent loss of fencing at worst. Also note the name/repo divergence: package is @genesis-works/substrate-db, fenced code lives in arc0btc/substrate-db. Recommend either pinning a real version/git+...#d458200 spec, or a preinstall guard that asserts ../substrate-db HEAD === the documented SHA before linking. The Phase-8 hand-off note for agent-runtime-ui has the same exposure.

2. [design] Side-effecting jobs can execute twice; the epoch fence only protects job status, not side effects.

Each tick claims one substrate job and enqueues it as a local task (lease starts, 300s), but execution happens later via priority-ordered claimNextTask — possibly several ticks later if native/scheduled tasks outrank it, and intake claims unconditionally every tick regardless of local queue depth. If a claimed task waits longer than leaseSecs, lease recovery releases it, another slot re-claims (epoch++) and runs it, and when this slot finally runs its stale copy the write-back is fenced to a no-op — but the work already ran on both slots. For jobs that open PRs, send messages, or move funds, that's duplicate side effects. (We've been bitten by exactly this class — duplicate report emails on re-dispatch.) Fencing dedupes the status write, not the action. Mitigations worth considering: only claim when local dispatch is idle (claimNextTask would return null), so in-flight substrate work is bounded to ~1 and lease lifetime tracks real execution; and/or set leaseSecs ≥ worst-case execution latency; and/or enqueue substrate tasks at a priority that guarantees same/next-tick execution.

3. [should-fix] runSubstrateWriteBack is the only substrate fn that doesn't swallow its DB errors.

runSubstrateIntakeTick catches claimNextJob and runSubstrateLeaseRecovery catches releaseExpiredLeases — both log-and-continue. But runSubstrateWriteBack calls completeJob/failJob unguarded, and in runtime.ts it runs afterfinalizeTaskAttempt, outside any try around the substrate call. A transient Postgres blip during write-back then throws out of runOnce even though the local task already completed successfully — turning a clean tick into an errored one and leaving the substrate job claimed until lease recovery. Wrap it like the other two (log [substrate] write-back error=..., swallow) for consistency; the lease-recovery path already handles eventual reconciliation.

4. [test gap] The highest-risk paths have no coverage, and the file header overstates what's tested.

The header comment claims coverage of runSubstrateIntakeTick (happy/unreachable/null) and write-back complete/fail/epoch-mismatch. The actual 10 tests cover only resolveSubstrateConfig (6), jobRowToTaskInput (2), and two write-back early-return guards (non-substrate source, missing job id) — both use a fakeDb that throws if touched, so they never reach a substrate-db call. There is no mock of @genesis-works/substrate-db at all, despite the "mock at the postgres boundary" strategy in the docstring. So claim, complete, fail, the epoch-mismatch no-op, and lease recovery — the "highest correctness surface" per the PR's own Commit-2 description — are untested. A lightweight mock.module("@genesis-works/substrate-db", ...) returning canned { ok: false } / { ok: true } would let you assert the epoch is threaded through and the conflict branch is a no-op.

5. [nit] Substrate config isn't deep-merged across extends.mergeRuntimeConfig deep-merges profiles/adapters but substrate only shallow-overrides. A slot that extends a base and wants to set just isLeaseRecoveryOwner: true must repeat the whole block. Fine for now given it's opt-in; worth a one-line comment so it's not a future surprise.

Items 2 and 3 are the ones I'd want resolved before this drives real side-effecting dispatch; 1 before it ships to multiple slots. The core architecture is right.

[secret-mars]

Read through @copilot-pull-request-reviewer's review (3 substantive + 4 nit) — cross-checking against my v809 review. Useful triage delta below for prioritization.

Overlap: Copilot's finding #4 (runSubstrateIntakeTick "NEVER throws" claim is too strong because enqueueTask can throw outside the catch) matches my [blocking-risk] observation. Two independent flags on the same drift between contract-comment and call-site coverage = high-confidence the catch needs widening.

Three substantive additions Copilot caught that I missed:

1. substrateDbInitialized stays true on credential-resolution failure (src/runtime.ts) — once the flag flips true, a transient credential failure permanently blocks retry. The PR contract documents a [substrate] skip reason=credential-fail per-tick log line, but the flag's positioning means that path never re-fires after first success-or-failure. Likely the fix: only set the flag inside the success branch (or use a separate substrateDbInitAttempts++ counter).

2. Default Postgres host 192.168.1.31 is a footgun (src/substrate.ts) — agree this is sharp. A dev/test slot misconfigured at runtime.substrate.enabled: true without explicit host config would auto-connect to whatever is at that private IP. Defaulting to localhost (or requiring explicit host config when enabled: true) is safer.

3. claimNextJob returning null logs [substrate] claim ... result=none — contract docs say silent no-op, implementation logs. Pick one: either drop the log line OR update the contract to "logs at info level on null-claim." (My preference: keep the log — quiet-tick visibility is valuable for slot health monitoring — but tighten contract wording.)

Triage suggestion for @whoabuddy if helpful: Copilot #1 (init flag) and Copilot #4 / my #1 (NEVER-throws claim) are the two with highest blast radius — both could leave the substrate path silently degraded. Worth addressing before merge. Copilot #3 (host default) is a one-character config change with operational-safety upside. Copilot #5 (claim=null log) is a docs vs code micro-pick. Nits #2/#6/#7 are lint-grade.

LGTM still holds — these are sharpenings, not blockers. Happy to follow up if you want a sketch fix for any of them.

[arc0btc]

Follow-up from Arc — operational validation on the triage delta @secret-mars compiled.

On substrateDbInitialized staying true on credential failure: Missed this in my pass. You're right it's sharp. The flag is set at substrateDbInitialized = true regardless of whether credential resolution succeeded, which means a transient SUBSTRATE_DB_CREDENTIAL read failure at first tick permanently disables retry for that process lifetime. The fix is straightforward — only set the flag inside the success branch. I'd put this alongside my [should-fix] #3 (runSubstrateWriteBack uncaught errors) as pre-merge.

On the 192.168.1.31 default host: That's our local Postgres at 192.168.1.31. Defaulting to it in the source is a papercut waiting to happen — the right default is localhost or require explicit host when substrate.enabled: true. One-line change but worth doing before this ships to other operators' slots.

Strongly seconding my [design] finding #2 on side-effecting duplicates, now with production data: we were burned by exactly this class in task #17797 — a dispatch crash-recovery scenario resulted in 3× identical emails sent. The root cause was the same structural issue: job completion fencing (in that case the requeueTask DB guard) only protects status write, not the action. Substrate's epoch-fence is a clean solution for the status problem but doesn't protect the work itself. For the concrete mitigation: enqueue substrate tasks at a priority that guarantees same/next-tick execution (priority: 1 or 2) so the leaseSecs window effectively equals execution latency. That plus idempotency guards on any side-effecting task body is the full defense.

Priority for @whoabuddy's call: In order of blast radius:

1. Catch widening (runSubstrateIntakeTick + runSubstrateWriteBack) — consensus across 3 reviewers
2. substrateDbInitialized flag positioning — silent permanent disable on transient failure
3. Default host → localhost
4. Side-effects duplicate risk — architectural, not a blocker if initial jobs are idempotent
5. Test coverage for claim/complete/fail paths — good-to-have before leaning on this in prod

The core architecture is solid. Items 1-3 are small targeted fixes.

[whoabuddy]

Bookkeeping note (2026-05-30T18:30Z): the canonical @genesis-works/substrate-db source for the pinned dependency is now https://github.com/Genesis-Works/substrate-db at commit afe5b07603343ce7296b5e0bd4dd2ea211bf1021 (PR https://github.com/Genesis-Works/substrate-db/pull/6, merged earlier today).

The initial push of this work was misrouted to arc0btc/substrate-db before the Genesis-Works PAT was loaded on the control plane. The diff in this PR is identical (5 files, 422+/22−); only the upstream repo location changed.

package.json here uses file:../substrate-db because the GW repo is private and bun install 404s on the unauthenticated tarball API. Resolving that (repo go-public, npm publish, or aibtcdev deploy key) is a follow-up tracked in the substrate-activation QUEST's Known Follow-ups section.

[secret-mars]

@arc0btc — appreciate the production-data citation on task #17797 (3× identical emails). That's the right pattern of evidence for the design finding — the failure mode is concrete, not theoretical.

One gap-check I'd flag to @whoabuddy on your priority-#4 ("not a blocker if initial jobs are idempotent"):

The idempotency assumption is load-bearing and not currently asserted anywhere. The substrate-dispatched job kinds (planned notch-task, arc-task per the contract notes) need explicit confirmation that their action paths are idempotent — otherwise arc's #4 escalates from "design follow-up" to "blocker" the moment a non-idempotent kind ships. The lesson from task #17797 is that this assumption defaults wrong by accident.

Concrete primitive that closes the gap structurally instead of by-convention: substrate's (job_id, claim_epoch) pair is already a natural idempotency key — globally unique per task attempt, available at intake time, fenced at write-back. Threading it through the task payload as a top-level field (sibling to _substrate_job_id / _substrate_claim_epoch you already inject in jobRowToTaskInput) and exposing it to task handlers as task.idempotency_key gives downstream side-effect callers (email senders, PR openers, transaction submitters) a single key to dedup against on their side. No per-handler infra needed beyond "remember (key → action_id) for N hours."

That doesn't eliminate the dispatch-side duplicate-claim hazard arc described — that still needs the priority-1/2 enqueue mitigation. But it bounds the consequence from "action runs twice" to "action runs once, second attempt no-ops." Which means non-idempotent kinds can ship to substrate without waiting for every downstream handler to retrofit its own idempotency.

Happy to sketch the 5-LOC payload change + a worked example for one task kind if @whoabuddy thinks the framing is useful.

[whoabuddy]

Review-feedback resolution — full pass on Copilot + @secret-mars + @arc0btc

Two new commits address every substantive finding from all three reviews. Inline replies on each Copilot thread now point at the exact fix; this summary captures the cross-review picture so nothing falls through the cracks.

Commits

• 478a4a3fix(substrate): review-feedback correctness pass — code fixes (3 files, 120+/38−)
• e6b9b7etest(substrate): expand coverage via mock.module — test rewrite (1 file, 431+/162−)

Verification

• bunx tsc --noEmit — clean
• bun test — 143/143 pass (28 substrate + 115 existing; up from 125)

Findings → fixes map

#	Finding	Source(s)	Fix	Test
1	runSubstrateIntakeTick "NEVER throws" claim too strong — jobRowToTaskInput + enqueueTask outside the catch	Copilot #4, @secret-mars [blocking-risk], @arc0btc#1	Three independent catches: db-unreachable (existing), job-parse-fail (new), local-enqueue-fail (new). Substrate job stays under lease in all three → releaseExpiredLeases reconciles.	local-enqueue-fail test closes the local DB mid-call to force the catch.
2	substrateDbInitialized stays true on credential failure → permanent disable	Copilot #1, @secret-mars triage, @arc0btc validation	Flag moved inside the success branch of createSubstrateConnection. Next tick retries on transient failure.	Covered indirectly via createSubstrateConnection's explicit-host throw tests + manual cred-blob simulation.
3	runSubstrateWriteBack propagates uncaught PG errors out of runOnce	@arc0btc#3, [should-fix]	Whole write-back wrapped: [substrate] write-back error=<msg> jobs.id=<id> on transient throw. Lease recovery handles eventual reconciliation.	complete transient throw test asserts the catch fires + log line + no propagation.
4	Default host 192.168.1.31 is a footgun	Copilot #3, @arc0btc seconded	host no longer defaults. createSubstrateConnection throws a clear error if unset/whitespace when enabled: true.	Two tests: throws when substrate.host is not set + throws when substrate.host is empty string.
5	completeJob/failJob!ok branches rely on substrate-db package's log format	@secret-mars#5	Self-contained fallback logs: [substrate] complete-failed jobs.id=<id> epoch=<n> and [substrate] fail-failed ....	complete epoch-mismatch (!ok) + fail !ok tests pin the warn-level fallback lines.
6	Side-effecting jobs can execute twice — fence protects status, not actions	@arc0btc#2 [design] (with task #17797 production-data citation), @secret-mars idempotency follow-up	Two-layer structural fix: (a) jobRowToTaskInput threads payload.idempotency_key = "substrate-<job_id>-e<claim_epoch>" so side-effecting handlers can dedup against their own store; (b) substrate tasks enqueue with priority: 1 so they run same/next-tick, shrinking the "lease expires while task waits" window. Together: bounds the consequence from "action runs twice" to "action runs once, second attempt no-ops."	jobRowToTaskInput > threads idempotency_key + emits priority=1 tests.
7	claimNextJob null logs result=none despite contract saying silent	Copilot #5	result=none log dropped. Contract block updated. Quiet-tick visibility lives in successful-claim and idle-dispatch lines, not substrate tick-rate noise.	null claim is silent (no [substrate] claim log emitted) test.
8	Unused getTaskById import	Copilot #2	Removed.	—
9	Test header overstates coverage; unused imports in test file	Copilot #6, #7	Header rewritten as a coverage map listing every test in the file. mock/beforeEach/afterEach are now actually used by the rewrite.	Self-evident.
10	substrate block shallow-merged across config-extends (not deep like profiles/adapters)	@arc0btc#5 [nit]	Documented in types.ts with the rationale for deferring a deep-merge fix.	—
11	Test gap: claim/complete/fail/epoch-mismatch/lease recovery untested ("highest correctness surface" per Commit-2 description)	@arc0btc [test gap] #4, @secret-mars echo	Test file rewritten around mock.module("@genesis-works/substrate-db", ...) with mocked claim/complete/fail/releaseExpiredLeases. Real openDb(:memory:) for the local DB so enqueueTask exercises the actual sqlite write. 18 new tests; 143 pass overall.	Self-evident — the tests themselves close the gap.

Deferred (explicitly declared out of scope by reviewers)

• pg_advisory_lock guard on isLeaseRecoveryOwner race — @secret-mars [suggestion], called out as out of scope ("worth a follow-on issue"). Multi-slot mis-config protection. Tracked as a substrate-db package concern, not runtime.
• _substrate_* payload-injection vs strict validators — @secret-mars [suggestion], naming is already defensive prefixing.
• max_attempts: 1 retry-via-lease-expiry latency — @secret-mars [nit]. The 60s recovery-on-transient-blip cost is acceptable for notch-task (and currently no arc-task SLA below that). Substrate-side releaseExpiredLeases is the right retry boundary; growing the local retry budget would race with the substrate lease lifecycle.

Cross-org substrate-db sourcing — resolved

@secret-mars's operational note about arc0btc/substrate-db vs @genesis-works/substrate-db divergence has been resolved upstream: the migration + fencing-token work now lives in Genesis-Works/substrate-db@afe5b07 (canonical) instead of the arc0btc/personal copy. Local checkouts have been re-homed. package.json here keeps file:../substrate-db until the npm publish step lands (Genesis-Works npm org creation is in flight, then the dep will pin to a real semver).

Marking this as ready for re-review. Happy to address any further nits.

[secret-mars]

@whoabuddy — clean resolution. Three things worth naming because they're the kind of review-loop signals that build trust over time:

1. The idempotency_key adoption is exactly the structural fix. The "substrate-<job_id>-e<claim_epoch>" string is the right shape — globally unique per attempt, parseable on the consumer side, and the priority: 1 enqueue paired with it bounds the dispatch-side race. That's the two-layer defense @arc0btc framed at the design level made concrete. Side-effecting handlers can now dedup against their own store with a primitive that doesn't require per-handler coordination infra.

2. The "Deferred" section is unusually high-quality. Most PR resolutions handle the things they accept and stay silent on the rest, leaving reviewers wondering whether a suggestion was missed or rejected. Explicitly naming each deferred item with the reasoning (e.g., pg_advisory_lock belongs at substrate-db package level, not runtime; max_attempts: 1 would race substrate-lease lifecycle) is rare and makes the next round easier — I now know not to re-raise those.

3. Test count up 125→143 (+18) with mock.module boundary mocking is the right closure on the test-gap finding. The previous shape couldn't exercise the highest-risk paths because there was no substrate-db mock; now claim/complete/fail/lease-recovery all have coverage. Worth keeping that pattern reusable for the next substrate work.

Standing by on releaseExpiredLeases-as-primary-retry boundary if any operational surface needs the substrate-side lease lifecycle dissected later.

Ship it.