Uh oh!
There was an error while loading. Please reload this page.
fix(db): retry the migration connection on transient slot exhaustion - #5226
Conversation
Every push to main/staging ran db:migrate against the production/staging database even when the merge changed no schema, so a no-op migration would dial the DB and fail whenever it was at its connection limit (53300, slots reserved for SUPERUSER) — red-X'ing UI-only merges. Add a detect-migrations job (dorny/paths-filter on packages/db/migrations/**) and pass the result into the reusable migrations workflow, which now skips the apply step when no migration files changed. The migrate job still runs so downstream build/deploy jobs that need it are never skipped, and the flag defaults to 'true' so manual dispatch and any unknown value always apply migrations — the gate only ever skips a provably-empty change.
The migration opens its session on the first query (the advisory-lock
acquire). When the deploy database briefly exhausts every non-superuser
connection slot at peak, that connect fails with 53300 ("remaining connection
slots are reserved for roles with the SUPERUSER attribute") and the whole
deploy's migrate step errors out — even when the spike clears within seconds.
Add a bounded connectWithRetry() before acquiring the lock that retries 53300,
the 08xxx connection_exception class, and the driver's transport errors with
backoff (10 attempts, ~90s ceiling). Non-transient errors (auth, bad config)
still fail fast. The migration is a single short-lived session, so waiting out
a transient spike is far safer than failing the deploy.The latest updates on your projects. Learn more about Vercel for GitHub.
|
PR SummaryLow Risk Overview
Reviewed by Cursor Bugbot for commit ac8eeed. Configure here. |
Uh oh!
There was an error while loading. Please reload this page.
Greptile SummaryThis PR adds a bounded retry before database migrations acquire their lock. The main changes are:
Confidence Score: 5/5This looks safe to merge.
Important Files Changed
Reviews (2): Last reviewed commit: "ci: drop the migration paths-filter gate..." | Re-trigger Greptile |
Uh oh!
There was an error while loading. Please reload this page.
Revert the detect-migrations gate carried over from the closed CI PR; we are fixing the connection failure at its source (migrate.ts connection retry) rather than gating db:migrate, which the reviewers correctly noted could leave a previously-merged migration unapplied after a failed deploy.
waleedlatif1
commented
Jun 26, 2026
waleedlatif1
commented
Jun 26, 2026
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit ac8eeed. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Migrations fail their deploy whenever the production/staging database is momentarily at its connection cap. The migration opens its session on the first query (the advisory-lock acquire); if every non-superuser slot is taken, that connect throws
53300— "remaining connection slots are reserved for roles with the SUPERUSER attribute" — and the whole migrate step errors out, even though the spike clears within seconds.This adds a bounded connection retry before the lock is acquired.
Why this approach (investigated via PlanetScale)
The "give the migration role reserved slots" route is not available from here: on the prod cluster,
max_connections(200),superuser_reserved_connections(3), andreserved_connections(0) are allcontext = postmaster— they only change with a database restart and are managed by PlanetScale's cluster sizing, notALTER SYSTEM. The only online levers (capping the app role's connection limit, or grantingpg_use_reserved_connectionswhilereserved_connections = 0) are either ineffective or risk shifting the outage onto the app.The saturation is transient (the DB sits around ~30 connections and briefly spikes), and the migration is a single short-lived session — so retrying the connect is the correct, low-risk fix.
Change
connectWithRetry()runsSELECT 1beforeacquireMigrationLock(), retrying53300, the08xxxconnection_exception class, and the postgres-js transport codes with jittered backoff (10 attempts, ~90s ceiling). Non-transient errors (auth, bad host/db) still fail fast. Reuses the existingbackoffWithJitter/getPostgresErrorCodehelpers.Durable capacity follow-up (platform, not in this PR)
The root cause is the app saturating ~all 200 slots at peak. Options for the owner to weigh: scale the cluster (raises
max_connections), contain the app's pool / route through the pooler, or have PlanetScale setreserved_connections+ grant the migration rolepg_use_reserved_connections(and revoke it from the app role, which currently holds it). All require platform/cluster action.Type of Change
Testing
Checklist