Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotzjoeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown startsResult
Delivery finishes during Graphile's grace periodNormal completion
Delivery is aborted and attempt budget remainsSame row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attemptSame row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-postgresPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6CompareJuly 23, 2026 11:19
@joeyhotzjoeyhotz changed the title fix(world-postgres): gracefully stop active jobsfix(world-postgres): abort stalled HTTP delivery on shutdownJul 23, 2026
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77CompareJuly 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code ownersJuly 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Pinging @vercel/workflow

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment threadpackages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread.changeset/gentle-jobs-stop.md Outdated

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-postgres/src/index.ts Outdated
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.
A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986bCompareJuly 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:mainJul 24, 2026
64 of 105 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
github-actionsBot added a commit that referenced this pull request Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@joeyhotz@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix(world-postgres): abort stalled HTTP delivery on shutdown by joeyhotz · Pull Request #3064 · vercel/workflow · GitHub
Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotzjoeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown startsResult
Delivery finishes during Graphile's grace periodNormal completion
Delivery is aborted and attempt budget remainsSame row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attemptSame row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-postgresPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6CompareJuly 23, 2026 11:19
@joeyhotzjoeyhotz changed the title fix(world-postgres): gracefully stop active jobsfix(world-postgres): abort stalled HTTP delivery on shutdownJul 23, 2026
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77CompareJuly 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code ownersJuly 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Pinging @vercel/workflow

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment threadpackages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread.changeset/gentle-jobs-stop.md Outdated

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-postgres/src/index.ts Outdated
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.
A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986bCompareJuly 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:mainJul 24, 2026
64 of 105 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
github-actionsBot added a commit that referenced this pull request Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@joeyhotz@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(world-postgres): abort stalled HTTP delivery on shutdown by joeyhotz · Pull Request #3064 · vercel/workflow · GitHub
Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotzjoeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown startsResult
Delivery finishes during Graphile's grace periodNormal completion
Delivery is aborted and attempt budget remainsSame row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attemptSame row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-postgresPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6CompareJuly 23, 2026 11:19
@joeyhotzjoeyhotz changed the title fix(world-postgres): gracefully stop active jobsfix(world-postgres): abort stalled HTTP delivery on shutdownJul 23, 2026
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77CompareJuly 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code ownersJuly 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Pinging @vercel/workflow

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment threadpackages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread.changeset/gentle-jobs-stop.md Outdated

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-postgres/src/index.ts Outdated
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.
A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986bCompareJuly 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:mainJul 24, 2026
64 of 105 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
github-actionsBot added a commit that referenced this pull request Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@joeyhotz@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(world-postgres): abort stalled HTTP delivery on shutdown by joeyhotz · Pull Request #3064 · vercel/workflow · GitHub
Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotzjoeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown startsResult
Delivery finishes during Graphile's grace periodNormal completion
Delivery is aborted and attempt budget remainsSame row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attemptSame row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-postgresPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6CompareJuly 23, 2026 11:19
@joeyhotzjoeyhotz changed the title fix(world-postgres): gracefully stop active jobsfix(world-postgres): abort stalled HTTP delivery on shutdownJul 23, 2026
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77CompareJuly 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code ownersJuly 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Pinging @vercel/workflow

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment threadpackages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread.changeset/gentle-jobs-stop.md Outdated

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-postgres/src/index.ts Outdated
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.
A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986bCompareJuly 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:mainJul 24, 2026
64 of 105 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
github-actionsBot added a commit that referenced this pull request Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@joeyhotz@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' fix(world-postgres): abort stalled HTTP delivery on shutdown by joeyhotz · Pull Request #3064 · vercel/workflow · GitHub
Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotzjoeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown startsResult
Delivery finishes during Graphile's grace periodNormal completion
Delivery is aborted and attempt budget remainsSame row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attemptSame row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-postgresPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6CompareJuly 23, 2026 11:19
@joeyhotzjoeyhotz changed the title fix(world-postgres): gracefully stop active jobsfix(world-postgres): abort stalled HTTP delivery on shutdownJul 23, 2026
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77CompareJuly 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code ownersJuly 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Pinging @vercel/workflow

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment threadpackages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread.changeset/gentle-jobs-stop.md Outdated

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-postgres/src/index.ts Outdated
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.
A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986bCompareJuly 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:mainJul 24, 2026
64 of 105 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
github-actionsBot added a commit that referenced this pull request Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@joeyhotz@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(world-postgres): abort stalled HTTP delivery on shutdown by joeyhotz · Pull Request #3064 · vercel/workflow · GitHub
Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotzjoeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown startsResult
Delivery finishes during Graphile's grace periodNormal completion
Delivery is aborted and attempt budget remainsSame row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attemptSame row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-postgresPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6CompareJuly 23, 2026 11:19
@joeyhotzjoeyhotz changed the title fix(world-postgres): gracefully stop active jobsfix(world-postgres): abort stalled HTTP delivery on shutdownJul 23, 2026
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77CompareJuly 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code ownersJuly 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Pinging @vercel/workflow

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment threadpackages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread.changeset/gentle-jobs-stop.md Outdated

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-postgres/src/index.ts Outdated
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.
A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986bCompareJuly 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:mainJul 24, 2026
64 of 105 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
github-actionsBot added a commit that referenced this pull request Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@joeyhotz@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); })(); fix(world-postgres): abort stalled HTTP delivery on shutdown by joeyhotz · Pull Request #3064 · vercel/workflow · GitHub
Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotzjoeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown startsResult
Delivery finishes during Graphile's grace periodNormal completion
Delivery is aborted and attempt budget remainsSame row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attemptSame row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-postgresPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6CompareJuly 23, 2026 11:19
@joeyhotzjoeyhotz changed the title fix(world-postgres): gracefully stop active jobsfix(world-postgres): abort stalled HTTP delivery on shutdownJul 23, 2026
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77CompareJuly 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code ownersJuly 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Pinging @vercel/workflow

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment threadpackages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread.changeset/gentle-jobs-stop.md Outdated

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment threadpackages/world-postgres/src/index.ts Outdated
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.
A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotzforce-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986bCompareJuly 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
ContributorAuthor

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:mainJul 24, 2026
64 of 105 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 24, 2026
github-actionsBot added a commit that referenced this pull request Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@joeyhotz@VaguelySerious