Skip to content

fix(service-job): destroy the cron adapter on kernel eviction so scheduled flows re-bind (#8362) - #8462

Merged
os-zhuang merged 1 commit into
mainfrom
claude/issue-8362-cron-rebind-after-kernel-rebuild
Aug 13, 2026
Merged

fix(service-job): destroy the cron adapter on kernel eviction so scheduled flows re-bind (#8362)#8462
os-zhuang merged 1 commit into
mainfrom
claude/issue-8362-cron-rebind-after-kernel-rebuild

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Fixes#8362

Scheduled and time-relative flows failed to re-bind after every kernel rebuild, permanently, with one unwatched WARN as the only signal. Kernel eviction is routine in the cloud runtime (freshness probe every few seconds; every AI auto-publish bumps freshness), so "AI builds a scheduled automation, the user edits one piece of metadata, the automation is silently dead" was the normal path.

Root cause, re-verified on origin/main

DbJobAdapter.destroy() destroyed only this.inner, never the CronJobAdapter it was constructed with — while CronJobAdapter.destroy() had existed all along, carrying a doc comment naming the caller that never called it. croner keys named jobs in a process-global array, and Cron.stop() is what splices an entry out of it, so every evicted kernel left its timers running and its names claimed for the life of the process. The rebuilt kernel then hit name already taken forever.

The eviction chain above that point was already intact: KernelManager.evict() -> kernel.shutdown() -> plugin.destroy() -> JobServicePlugin.destroy() -> dbAdapter.destroy(). It just stopped one level short.

Changes

  1. DbJobAdapter.destroy() destroys the cron adapter too — the single-point fix. IJobService does not declare destroy(), so the call is structural, like the existing cancel forwarding. A failure to shut it down is reported at error, because at that point runtime state disagrees with every other surface and nothing looks wrong.
  2. JobServicePlugin releases the cron adapter it owns on the adapter: 'cron' path, where nothing else would have (the same leak, one branch over).
  3. CronJobAdapter scopes its croner registry key to the adapter INSTANCE (cronRegistryName() exposes the key; an optional namespace label, filled from OS_ENVIRONMENT_ID, only makes scheduledJobs readable). This independently fixes the second defect the issue names: two environments in one container binding the same AI-generated flow name used to collide with no eviction involved at all. Per-instance rather than per-environment on purpose — a rebuilt kernel reuses the environment id, which is the collision itself.
  4. Claiming a name a foreign job still holds now REPLACES it: the holder is stopped and the name retaken, instead of warn-and-give-up. Stopping is the load-bearing half — a leaked croner job is a live timer closed over a shut-down kernel, so taking the name while leaving it running would trade a silent death for a zombie double-write, which is strictly worse than the bug being fixed.
  5. Both triggers report a failed bind at error with the consequence and the remedy, via one shared helper.

On the log level (item 4), and where it deliberately stops

AGENTS.md decides warn vs error with one question: after the degradation, does the system still look normal from the outside while something it claims is in place has not landed? Here it does, completely — the flow stays published and active, Studio lists it, the metadata API serves it, verify_build passes, and nothing will ever fire it. That is persisted and runtime state disagreeing, which the rule puts in the error class. The neighbouring composition branch ("no job service registered at all") stays at warn; the rule names that exact message as correctly a warn, and the distinction is not severity but whether the outside can see it.

Per the triage constraint, item 4 stops at the log signal: no new status surface was invented. Writing bind failures into sys_job.last_status was considered and rejected — that column is a run status with decisions recorded in its TSDoc, and overloading it is exactly the shape the constraint names.

Reverse verification, measured

Every reading below is real output, not a prediction.

On pristine origin/main (tests added, implementation untouched) — 4 red, 67 pre-existing green:

  • DbJobAdapter — kernel rebuild > destroy() destroys the CRON adapter too: expect(job.isStopped()).toBe(true) -> expected false to be true. The destroy chain, measured.
  • a rebuilt kernel re-binds the same flow: Cron: Tried to initialize new named job 'flow-time-relative:xqao_contract_expiry_reminder_flow', but name already taken. — the reported production failure, reproduced byte-for-byte.
  • lets two live adapters hold the SAME job name: same throw, with no eviction involved (the cross-environment defect).
  • reclaims its registry name from a foreign holder: cronRegistryName is not a function — new capability, no main-side equivalent.

After the fix: service-job 71/71, trigger-schedule 46/46, both typecheck clean, ESLint clean.

Second direction — reverting ONLY the two trigger sources to origin/main, with the service-job fix left in place. Predicted before running: the two observability cases go red and the two trigger-seam rebind cases stay green, because that fix lives in the adapter, not in the triggers. Observed exactly that: 2 failed | 44 passed, both failures expected [] to have a length of 1 (the error channel is empty on main; it logged warn).

Vacuity traps addressed explicitly

  • Every case goes through the cron path. The issue's own control experiment showed interval uses setInterval and never enters croner's named registry, so an interval-shaped fixture would pass against a completely unfixed tree.
  • The first bind is asserted before the second.expect(registeredFor(JOB)).toHaveLength(1) runs before any rebind assertion, so a fixture that silently registered nothing cannot pass for the wrong reason.
  • Scheduled once and fires. The count is asserted (toHaveLength(1) — one live job, not one live plus one zombie) and the surviving croner job is then triggered, asserting the NEW kernel's callback ran and the evicted kernel's closure did not.
  • Zombie check asserts the old job is STOPPED, holding the captured Cron object across the rebuild and checking oldJob.isStopped() — not merely that a new job exists under a different name, which cannot distinguish "destroyed" from "still running".

Scope

Confined to packages/services/service-job/src/** and packages/triggers/trigger-schedule/src/**, plus one test-only croner devDependency on trigger-schedule (mirroring the existing service-automation devDep that backs its e2e test) so the trigger-seam pins run against the real process-global registry rather than a hand-modelled Map.

The architectural half — the clock living above the kernel, so scheduled work does not depend on a resident kernel — is not addressed here; out of scope: objectstack-ai/cloud#1288. Even with everything above landed, "scheduled jobs do not run while no kernel is resident" remains true and belongs there.

Gates

Re-derived against the actual diff with scripts/pm/dispatch-gates.mjs. All run locally and green: check:nul-bytes, check:docs-audit-scope, check:test-source-alias, check:type-source-resolution, check:engine-double-contract, check:durability-log-level, check:startup-registry-verdict, check:query-options-erasure, check:changeset-gate-self-tests, check:objectui-changeset, check-changeset-fixed.mjs, check-changeset-no-major.mjs.


Generated by Claude Code

…duled flows re-bind (#8362)
DbJobAdapter.destroy() only destroyed `inner`, never the CronJobAdapter it was
constructed with. Kernel eviction is routine in the cloud runtime, so every
evicted kernel left its croner timers running and holding their PROCESS-GLOBAL
croner names forever; the rebuilt kernel then failed to re-bind that flow
permanently, with one unwatched WARN as the only signal.
- DbJobAdapter.destroy() destroys the cron adapter too (the single-point cause),
reporting a failure to do so at error level (persisted vs runtime state
disagreeing, invisible everywhere else).
- JobServicePlugin releases the cron adapter it owns on the `adapter: 'cron'`
path, where nothing else would.
- CronJobAdapter scopes its croner registry key to the adapter INSTANCE, which
also fixes cross-environment collisions in one container with no eviction
involved. Per-instance rather than per-environment on purpose: a rebuilt
kernel reuses the environment id, which is the collision itself.
- Claiming a registry name a foreign job still holds now STOPS that job and
replaces it, rather than warning and giving up. Stopping matters: a leaked
croner job is a live timer closed over a shut-down kernel, so taking the name
while leaving it running would trade a silent death for a zombie double-write.
- Both triggers report a failed bind at error with consequence and remedy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARidKDYSCD56LaygrvDPnk
@vercel

vercelBot commented Aug 13, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
objectstackIgnoredIgnoredAug 13, 2026 2:16pm

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/service-job, @objectstack/trigger-schedule.

2 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/kernel/services-checklist.mdx(via @objectstack/service-job)
  • content/docs/plugins/packages.mdx(via @objectstack/service-job)

2 release-owned page(s) also reference the affected code. These are read-only:

  • content/docs/releases/implementation-status.mdx(via @objectstack/service-job, @objectstack/trigger-schedule)
  • content/docs/releases/v16.mdx(via @objectstack/trigger-schedule)

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file tests tooling labels Aug 13, 2026
@os-zhuang
os-zhuang marked this pull request as ready for review August 13, 2026 15:37
@os-zhuang
os-zhuang added this pull request to the merge queueAug 13, 2026
Merged via the queue into main with commit d37ac1cAug 13, 2026
27 checks passed
@os-zhuang
os-zhuang deleted the claude/issue-8362-cron-rebind-after-kernel-rebuild branch August 13, 2026 16:02
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependenciesPull requests that update a dependency filedocumentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

2 participants

@os-zhuang@claude