Uh oh!
There was an error while loading. Please reload this page.
fix(world-postgres): abort stalled HTTP delivery on shutdown - #3064
Conversation
🦋 Changeset detectedLatest commit: 2342b1e The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
926a5fe to
856e2c6Compare856e2c6 to
2a98e77Comparejoeyhotz
commented
Jul 23, 2026
Pinging @vercel/workflow |
VaguelySerious
left a comment
There was a problem hiding this comment.
LGTM, doing another AI pass before approving
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: blocking issues found
Uh oh!
There was an error while loading. Please reload this page.
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown. A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits. Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior. Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
120943f to
2ed986bComparejoeyhotz
commented
Jul 24, 2026
Thanks @VaguelySerious, attended to the feedback. @vercel/workflow |
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Uh oh!
There was an error while loading. Please reload this page.
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Backport PR opened against |
Description
We observed this failure in a self-hosted deployment with a short termination grace period:
SIGTERMwhile Graphile Worker was delivering a step to the local Workflow HTTP route.helpers.abortSignal, butworld-postgresdiscarded the signal, so its HTTP request kept waiting.world.close()did not finish before the platform sentSIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.The same missing abort forwarding exists on current
main/ the v5 beta.This PR forwards
helpers.abortSignaltofetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.Graphile records an attempt when it claims a row, before the HTTP call:
So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.
The lifecycle changes are limited to what that shutdown requires:
world.close()stops the queue, waits forrunner.promise, then closes the streamer and internally owned pool.runner.promiseif Graphile.s automatic shutdown started first.world.close()call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1enables application-managed shutdown for the standard package target;applicationManagedShutdown: trueprovides the same opt-in for programmatic Worlds. The default remainsfalse.The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.
SIGKILLcan still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.How did you test your changes?
Added loopback HTTP tests for aborting:
Added lifecycle tests for:
stop()returns early or reports that shutdown already started.world.close()after a transient cleanup failure.Verification:
pnpm --filter @workflow/world-postgres test(163 tests, including Testcontainers PostgreSQL conformance tests)pnpm --filter @workflow/world-postgres typecheckpnpm --filter @workflow/world-postgres buildpackages/world-postgres/README.md(2 tests)pnpm changeset status --since=upstream/mainA real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (
maxAttempts: 1becomes unlocked/terminal;maxAttempts: 2becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.Docs Preview
The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.
PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRgit commit --signoffon your commits)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete