Uh oh!
There was an error while loading. Please reload this page.
fix(service-job): destroy the cron adapter on kernel eviction so scheduled flows re-bind (#8362) - #8462
Conversation
…duled flows re-bind (#8362) DbJobAdapter.destroy() only destroyed `inner`, never the CronJobAdapter it was constructed with. Kernel eviction is routine in the cloud runtime, so every evicted kernel left its croner timers running and holding their PROCESS-GLOBAL croner names forever; the rebuilt kernel then failed to re-bind that flow permanently, with one unwatched WARN as the only signal. - DbJobAdapter.destroy() destroys the cron adapter too (the single-point cause), reporting a failure to do so at error level (persisted vs runtime state disagreeing, invisible everywhere else). - JobServicePlugin releases the cron adapter it owns on the `adapter: 'cron'` path, where nothing else would. - CronJobAdapter scopes its croner registry key to the adapter INSTANCE, which also fixes cross-environment collisions in one container with no eviction involved. Per-instance rather than per-environment on purpose: a rebuilt kernel reuses the environment id, which is the collision itself. - Claiming a registry name a foreign job still holds now STOPS that job and replaces it, rather than warning and giving up. Stopping matters: a leaked croner job is a live timer closed over a shut-down kernel, so taking the name while leaving it running would trade a silent death for a zombie double-write. - Both triggers report a failed bind at error with consequence and remedy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ARidKDYSCD56LaygrvDPnk
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
📓 Docs Drift CheckThis PR changes 2 package(s): 2 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
⛔ 2 release-owned page(s) also reference the affected code. These are read-only:
|
Uh oh!
There was an error while loading. Please reload this page.
Fixes#8362
Scheduled and time-relative flows failed to re-bind after every kernel rebuild, permanently, with one unwatched WARN as the only signal. Kernel eviction is routine in the cloud runtime (freshness probe every few seconds; every AI auto-publish bumps freshness), so "AI builds a scheduled automation, the user edits one piece of metadata, the automation is silently dead" was the normal path.
Root cause, re-verified on
origin/mainDbJobAdapter.destroy()destroyed onlythis.inner, never theCronJobAdapterit was constructed with — whileCronJobAdapter.destroy()had existed all along, carrying a doc comment naming the caller that never called it. croner keys named jobs in a process-global array, andCron.stop()is what splices an entry out of it, so every evicted kernel left its timers running and its names claimed for the life of the process. The rebuilt kernel then hitname already takenforever.The eviction chain above that point was already intact:
KernelManager.evict()->kernel.shutdown()->plugin.destroy()->JobServicePlugin.destroy()->dbAdapter.destroy(). It just stopped one level short.Changes
DbJobAdapter.destroy()destroys the cron adapter too — the single-point fix.IJobServicedoes not declaredestroy(), so the call is structural, like the existingcancelforwarding. A failure to shut it down is reported aterror, because at that point runtime state disagrees with every other surface and nothing looks wrong.JobServicePluginreleases the cron adapter it owns on theadapter: 'cron'path, where nothing else would have (the same leak, one branch over).CronJobAdapterscopes its croner registry key to the adapter INSTANCE (cronRegistryName()exposes the key; an optionalnamespacelabel, filled fromOS_ENVIRONMENT_ID, only makesscheduledJobsreadable). This independently fixes the second defect the issue names: two environments in one container binding the same AI-generated flow name used to collide with no eviction involved at all. Per-instance rather than per-environment on purpose — a rebuilt kernel reuses the environment id, which is the collision itself.errorwith the consequence and the remedy, via one shared helper.On the log level (item 4), and where it deliberately stops
AGENTS.md decides
warnvserrorwith one question: after the degradation, does the system still look normal from the outside while something it claims is in place has not landed? Here it does, completely — the flow stays published and active, Studio lists it, the metadata API serves it,verify_buildpasses, and nothing will ever fire it. That is persisted and runtime state disagreeing, which the rule puts in theerrorclass. The neighbouring composition branch ("no job service registered at all") stays atwarn; the rule names that exact message as correctly awarn, and the distinction is not severity but whether the outside can see it.Per the triage constraint, item 4 stops at the log signal: no new status surface was invented. Writing bind failures into
sys_job.last_statuswas considered and rejected — that column is a run status with decisions recorded in its TSDoc, and overloading it is exactly the shape the constraint names.Reverse verification, measured
Every reading below is real output, not a prediction.
On pristine
origin/main(tests added, implementation untouched) — 4 red, 67 pre-existing green:DbJobAdapter — kernel rebuild > destroy() destroys the CRON adapter too:expect(job.isStopped()).toBe(true)->expected false to be true. The destroy chain, measured.a rebuilt kernel re-binds the same flow:Cron: Tried to initialize new named job 'flow-time-relative:xqao_contract_expiry_reminder_flow', but name already taken.— the reported production failure, reproduced byte-for-byte.lets two live adapters hold the SAME job name: same throw, with no eviction involved (the cross-environment defect).reclaims its registry name from a foreign holder:cronRegistryName is not a function— new capability, no main-side equivalent.After the fix: service-job 71/71, trigger-schedule 46/46, both
typecheckclean, ESLint clean.Second direction — reverting ONLY the two trigger sources to
origin/main, with the service-job fix left in place. Predicted before running: the two observability cases go red and the two trigger-seam rebind cases stay green, because that fix lives in the adapter, not in the triggers. Observed exactly that:2 failed | 44 passed, both failuresexpected [] to have a length of 1(the error channel is empty on main; it loggedwarn).Vacuity traps addressed explicitly
cronpath. The issue's own control experiment showedintervalusessetIntervaland never enters croner's named registry, so an interval-shaped fixture would pass against a completely unfixed tree.expect(registeredFor(JOB)).toHaveLength(1)runs before any rebind assertion, so a fixture that silently registered nothing cannot pass for the wrong reason.toHaveLength(1)— one live job, not one live plus one zombie) and the surviving croner job is then triggered, asserting the NEW kernel's callback ran and the evicted kernel's closure did not.Cronobject across the rebuild and checkingoldJob.isStopped()— not merely that a new job exists under a different name, which cannot distinguish "destroyed" from "still running".Scope
Confined to
packages/services/service-job/src/**andpackages/triggers/trigger-schedule/src/**, plus one test-onlycronerdevDependency ontrigger-schedule(mirroring the existingservice-automationdevDep that backs its e2e test) so the trigger-seam pins run against the real process-global registry rather than a hand-modelled Map.The architectural half — the clock living above the kernel, so scheduled work does not depend on a resident kernel — is not addressed here; out of scope: objectstack-ai/cloud#1288. Even with everything above landed, "scheduled jobs do not run while no kernel is resident" remains true and belongs there.
Gates
Re-derived against the actual diff with
scripts/pm/dispatch-gates.mjs. All run locally and green:check:nul-bytes,check:docs-audit-scope,check:test-source-alias,check:type-source-resolution,check:engine-double-contract,check:durability-log-level,check:startup-registry-verdict,check:query-options-erasure,check:changeset-gate-self-tests,check:objectui-changeset,check-changeset-fixed.mjs,check-changeset-no-major.mjs.Generated by Claude Code