fix(world-postgres): serialize keyed deliveries across workers

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[joeyhotz]

Description

@workflow/world-postgres currently passes QueueOptions.idempotencyKey to Graphile Worker as jobKey. With Graphile's default replace mode, adding the same key while its current job is locked clears the locked job's key, exhausts that row, and inserts a successor. The existing inflightMessages map prevents overlap inside one process, but separate Postgres World processes can claim the current job and its successor at the same time.

We observed this after a self-hosted deployment scaled out: startup recovery replayed active runs and redispatched their keyed steps. In one staging burst, 7 of 34 materialisation steps received a second start without an intervening failure (41 executions total). The same startup-recovery/duplicate-start signature appeared in production. This behavior is present in @workflow/world-postgres 4.3.3 and current main.

This PR assigns every keyed Graphile job to a deterministic named queue. A named queue is Graphile's native cross-worker concurrency primitive: while one worker owns the queue, a replacement remains pending. After the first delivery commits and releases the queue, the successor can replay the terminal state instead of overlapping the original side effect.

The implementation deliberately:

• Keeps Graphile's default replace mode. This preserves the durable successor created by delayed rescheduling and crash recovery.
• Applies only to messages that carry idempotencyKey. Unkeyed orchestrator messages and public World interfaces are unchanged.
• Uses 2,048 stable SHA-256 buckets per configured jobPrefix. Exact per-key queue names would grow Graphile's persistent queue table without bound; hashing the job task name keeps prefixes isolated and the physical name below Graphile's 128-character limit.
• Applies the same mapping when migrating pg-boss jobs that actually carry MessageData.idempotencyKey. Legacy unkeyed rows used messageId as singleton_key and remain unqueued.
• Does not run live GC_JOB_QUEUES; Graphile 0.16.6 cleanup raced concurrent producers in a local stress test and left 398 of 7,680 jobs orphaned (5.18%).

jobKeyMode: 'unsafe_dedupe' is not safe here. The handler enqueues a delayed successor before its current locked job returns; unsafe_dedupe would discard that successor, after which completing the current row can strand the workflow. It can also suppress the only replacement for a locked job that later dies on its final attempt.

This serializes competing deliveries; it does not claim exactly-once execution. The runtime's terminal-state replay remains the duplicate-suppression layer after the queue releases.

Operational trade-offs:

Case	Behavior
First mixed-version rollout or rollback	Existing/outgoing jobs without queueName do not own the new queue. Deployments must drain old keyed jobs and producers for a fully protected handoff.
Hard process death	A bucket can remain locked until Graphile's four-hour stale-lock recovery (or an explicit worker unlock). Graceful shutdown releases it normally.
Bucket collision	Unrelated keyed jobs can serialize. At the default concurrency of 50, 2,048 buckets produce about 0.60 expected colliding pairs, or roughly 1.2% expected slot loss.

A shuffled three-repetition Graphile 0.16.6 no-op benchmark at concurrency 50 / pool 8 measured median throughput of 181.5 jobs/s with no named queues, 141.1 jobs/s with all 2,048 buckets populated, and 108.2 jobs/s with 4,096. The VPS timings were noisy, so these are directional rather than a production capacity claim; 2,048 was consistently the better balance. Dropping to 1,024 would double expected collision loss at the default concurrency to roughly 2.4%.

This complements #3119 and #3162 but does not replace them. Those address selecting parked runs and accumulating startup-recovery jobs; this PR closes the separate locked-keyed-delivery concurrency gap. The queueName seam is independent of the HTTP loopback and can be carried through the in-process execution refactor in #3322.

How did you test your changes?

Added unit coverage for:

• Keyed producer jobs receiving the deterministic named queue.
• Delayed handler reschedules retaining the same jobKey, named queue, runAt, and 49-attempt budget.
• Unkeyed messages remaining outside named queues.
• Custom jobPrefix values receiving isolated queue scopes.
• pg-boss migration distinguishing real idempotency keys from message-ID singleton keys.

Added a real-PostgreSQL Testcontainers regression with two independent pools and two createQueue() instances. It blocks worker A's HTTP delivery, enqueues the same key through worker B, waits through the polling window, and asserts that both rows share one named queue while only the first is locked. Before the fix, the same test observed two concurrent HTTP deliveries (maxActiveRequests: 2); with the fix it observes one (maxActiveRequests: 1) and then drains both jobs sequentially.

Verification:

• pnpm exec vitest run packages/world-postgres/src (3 files, 30 tests)
• Real-PostgreSQL concurrency regression against an isolated local PostgreSQL 17 database (1 test)
• pnpm --filter @workflow/world-postgres typecheck
• pnpm --filter @workflow/world-postgres build
• Biome checks over the changed TypeScript files (only the two pre-existing warnings in queue.ts / queue.test.ts)
• pnpm changeset status

Additional recovery probe with Graphile Worker 0.16.6: a child worker was killed while holding the named queue, a same-key replacement was added, and after forceUnlockWorkers only the replacement executed.

PR Checklist - Required to merge

• 📦 pnpm changeset was run to create a changelog for this PR
  • Patch changeset for @workflow/world-postgres.
• 🔒 DCO sign-off passes (run git commit --signoff on your commits)
• 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

Draft for author review. No reviewers have been requested.

[changeset-bot]

🦋 Changeset detected

Latest commit: 163dbf4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@workflow/world-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.