world-postgres: startup recovery re-enqueues parked runs and accumulates duplicate jobs

[jiaruini]

Summary

@workflow/world-postgres treats every pending or running run as runnable during startup recovery.

createWorld().start() calls reenqueueActiveRuns(), which lists every active run and queues it without inspecting its persisted suspension state. A run that is durably parked on an unresolved hook or a future wait is therefore replayed on every process restart.

The recovery enqueue also gets a fresh message ID/job key when no idempotency key is supplied, so repeated restarts accumulate another outstanding Graphile job for the same run.

This reproduces in a single application with one namespace and one Postgres database. It is separate from #2978, which covers cross-application namespace isolation.

Versions

• @workflow/world-postgres@5.0.0-beta.28
• @workflow/world@5.0.0-beta.22
• graphile-worker@0.16.6
• Node.js 24

The same paths are still present on main at 62d570ed4bf38db333ae9fe9ba513c0d6a9d6b91:

• packages/world-postgres/src/index.ts: start() calls reenqueueActiveRuns(storage.runs, queue.queue, ...)
• packages/world/src/recovery.ts: recovery lists all pending and running runs
• packages/world-postgres/src/queue.ts: an enqueue without an explicit idempotency key uses a fresh message ID as its Graphile job key

Reproduction

Using a real Postgres database and the package's official migrations:

1. Persist 500 running runs whose only live state is an unresolved hook. They have no active step, received hook delivery, due wait, or existing queue job.
2. Persist 20 genuinely runnable runs with an active step.
3. Add one deliberately ambiguous active run.
4. Start and close the Postgres World once.
5. Inspect graphile_worker._private_jobs.

Observed: all 521 active runs get a job instead of only the 20 runnable runs.

A second fixture with 20 runnable runs shows the deduplication problem:

1. Start and close the same World five times without consuming the jobs.
2. Inspect outstanding Graphile jobs grouped by runId.

Observed: 100 jobs total, with five jobs per run. Expected: 20 jobs total, with at most one outstanding recovery job per run.

There is also a delivery/restart race: if a durable wake event and its normal delivery job are created around startup recovery, the same run can receive both a normal wake job and a recovery job because they do not share a persisted idempotency identity.

Expected behavior

• A durably suspended run should not be replayed merely because its top-level status is running.
• Persisted runnable work must still recover, including the crash window after a wake event is stored but before its queue job is inserted.
• Repeated startup recovery must not add more than one equivalent outstanding job for the same run/recovery cause.
• If persisted state cannot be classified safely, startup should report/fail rather than replay every active run.

At minimum, recovery needs a persisted runnable predicate plus durable queue deduplication. A stable key derived from the run and the persisted wake/event identity would cover repeated restarts without suppressing later legitimate wakes.

Impact

A service with many long-lived hook-backed sessions can enqueue hundreds of unnecessary replays on every container restart. Repeated restarts multiply the backlog, increase concurrent event-log replay, and can cause long startup delays or memory exhaustion.

Related issues

• Shared Postgres world: queue namespaces don't isolate claiming, and startup recovery re-enqueues other applications' active runs #2978 covers unscoped recovery across applications/namespaces; this issue reproduces with only one application.
• sleep() is not durable across worker crashes in Postgres World #679 and Postgres World - Workflows doesn't recover after process crash #1531 motivated restart recovery, but do not cover over-selection or duplicate recovery jobs.

[jiaruini]

Redacted production evidence

I restored a PostgreSQL workflow dump captured on 2026-07-27 into an isolated local container with no published ports or network access. Only aggregate system state was queried; no identifiers, tokens, inputs, outputs, payloads, or user data are included.

The correlated production versions were:

• eve@0.27.6
• @workflow/world@5.0.0-beta.22
• @workflow/world-postgres@5.0.0-beta.27
• graphile-worker@0.16.6
• Node.js 24

The snapshot contained 2,504 Runs, including 456 with top-level pending or running status:

• 451 running Sessions
• 4 running Turns
• 1 pending Turn

All 451 running Sessions had:

• zero running Steps;
• three or four open Hooks;
• at least one historical hook_received event.

Additionally:

• 429 Sessions had multiple historical hook_received events;
• one Session had 40 such events;
• the latest event was hook_disposed for 345 Sessions, hook_received for 102, and step_completed for 4.

I evaluated a candidate recovery predicate based on historical runnable-looking events. It still classified 449 of the 456 active Runs as runnable. This is a production counterexample showing that neither EXISTS(hook_received) nor the latest event can reliably distinguish an unconsumed delivery from a delivery that has already been replayed and consumed.

The persisted state appears to need an explicit durable consumed/acknowledged checkpoint, or another owner-level distinction between parked and runnable work. A query-only heuristic over existing event history does not appear sufficient.

The queue snapshot also contained 66 locked workflow jobs covering 64 Run IDs. Two Runs had two jobs each. Of those jobs, 55 referenced completed Runs, 4 failed Runs, 6 active Runs, and 1 no longer had a matching Run row. The stale-lock portion is probably related to #679 and may amplify the startup recovery problem, but the dump alone cannot prove causation.

Operator observations from the same incident:

• 71 container restarts;
• startup logged Re-enqueued 418 active run(s);
• approximately 20 application Sandboxes were actually running.

I can run additional read-only aggregate queries or provide a synthetic state fixture if maintainers need a specific invariant tested. The raw production dump cannot be shared publicly.

[olgen]

Production evidence from kindergeld.plus (eve self-host, 2026-08-06)

We hit the same class described here: ~105 zombie workflowEntry runs in our eve DB (status=running, 0 open waits, last event >1h) out of ~109 running workflowEntry rows. Fleet-wide ~198 running runs includes legitimate sessionTimeoutWorkflow parks.

Typical stall shape (subagent HITL handoff):

• Child turnWorkflow completes after input.requested (tool-approval)
• Parent + child each record a single hook_received, then silence — no open wait, no session.waiting
• AgentPanel stuck on „Läuft…“ for hours

Deploy correlation: we suspect #3119 startup recovery re-enqueue on every eve pod restart amplifies long-parked cross-run HITL (child parks on approval, parent proxies via resumeHook). Every deploy restarts eve → replays parked sessions.

Mitigations we're shipping:

• Auto-heal cron (cancel + clear zombies >30m stale)
• @workflow/world-postgres beta.20→beta.31 + workflow beta.39 (hook-resume migrations)
• Moving approval: always tool HITL off child sessions where possible (root-owned draft pattern)

Stack at investigation: eve 0.29.5, wp beta.20 (now upgrading). Watching #3162 — happy to test a local patch if it lands unmerged.

cc @vercel/workflow maintainers

[himself65]

Filed #3758 for a distinct harm on this same function, splitting it out the way this issue split from #2978.

Same call (reenqueueActiveRuns on start()), different failure: on a horizontally-scaled deployment a scale-out boot re-enqueues runs a live peer process is mid-replay, so two processes append to one event log and the run dies CORRUPTED_EVENT_LOG rather than merely being replayed redundantly. Cost us a completed 12-minute production run — both LLM turns paid for and discarded, all three business retries burned inside the same two-minute scale-out window.

It is additive to the asks here rather than covered by them: a correct runnable predicate plus durable job dedup narrows the window but cannot close it, because the duplicate arises from a second process, and inflightWorkflowRuns — the guard that states this exact invariant — is process-local.

Also worth noting for anyone landing here from the stable line rather than 5.0.0-beta: world@4.4.0's recovery.ts and world-postgres@4.3.4's start() are unchanged, so 4.x is affected identically.