Uh oh!
There was an error while loading. Please reload this page.
fix(cli): detect dead db connections (CLI-2207) - #6277
Conversation
Supabase CLI previewnpx --yes https://pkg.pr.new/supabase/cli/supabase@5a80fb2bdfe03d8fbca22a2763f3920a0d733e67Preview package for commit |
…ation-apply-sql-pipeline-stops
avallete
left a comment
There was a problem hiding this comment.
The core fix is sound — I traced the load-bearing pieces and they hold: the widened discard predicate strictly covers the old poisoned check, the only execBatch caller guards LegacyDbConnectError before touching statementIndex, keepalive reaches both the pool and the raw client, and pg honors the submit() error return so the writable check settles instead of hanging. I also chased two scary scenarios that turned out impossible: there's no residual hang on exec/query (pg's 'close' → _errorAllQueries fails them within a tick), and no applied-but-unrecorded migration (the history INSERT rides inside the batch).
I reproduced the findings below against this branch where marked; one is a process crash I'd fix before merge. The script used for the reproduced non-crash findings:
verify-findings.ts (run with bun from the repo root)
// Verification of the review findings against this branch's real modules.// Save at the repo root and run: bun verify-findings.tsimport{LegacyPgBatchQuery,legacyBatchFailureError,legacyToExecError,}from"./apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts";import{legacyFormatExecBatchError}from"./apps/cli/src/legacy/shared/legacy-migration-apply.ts";importtype*asPgfrom"pg";console.log("=== standalone exec renders dead connection as 'At statement: N' ===");constdeadConnErr=newError("Client has encountered a connection error and is not queryable");constexecMapped=legacyToExecError(deadConnErr);constrendered=legacyFormatExecBatchError(execMapped,3,"CREATE INDEX CONCURRENTLY idx ON t (c)");console.log(rendered.message);console.log("\n=== poisoned batch blames statement 0 ===");letbindCalls=0;constnoop=()=>{};constconn={stream: {writable: true,cork: noop,uncork: noop},parse: noop,bind: ()=>{bindCalls+=1;if(bindCalls===7)thrownewError("frame serialization blew up on statement 7");},describe: noop,execute: noop,sync: noop,}asunknownasPg.Connection;conststatements=Array.from({length: 10},(_,i)=>({sql: `SELECT ${i} /* stmt ${i} */`}));constbatch=newLegacyPgBatchQuery(statements,noop);constsubmitErr=batch.submit(conn);console.log("poisoned:",batch.poisoned,"submitted:",batch.submitted,"completed:",batch.completed);constfailure=legacyBatchFailureError(submitErr!,batch);console.log("mapped:",failure._tag,"statementIndex:",(failureas{statementIndex?: number}).statementIndex);console.log("\n=== batch-lost ConnectError has no suggestion ===");constunsent=newLegacyPgBatchQuery([{sql: "SELECT 1"}],noop);constunsentErr=unsent.submit({stream: {writable: false}}asunknownasPg.Connection);constconnectFailure=legacyBatchFailureError(unsentErr!,unsent);console.log(connectFailure._tag,"| suggestion:",(connectFailureas{suggestion?: string}).suggestion);console.log("\n=== message-equality sentinel stutters once wrapped ===");constwrapped=newError(`batch submit failed: ${unsentErr!.message}`);console.log(legacyBatchFailureError(wrapped,unsent).message);Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…pabase-db-resetstart-migration-apply-sql-pipeline-stops # Conflicts: # apps/cli/docs/go-cli-divergences.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:b7397db3f1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
avallete
left a comment
There was a problem hiding this comment.
Re-reviewed HEAD after the follow-ups. The hang-fix path holds: submit() refuses an already-unwritable socket, pg 8.23 delivers that Error through handleError, unsent batches surface as LegacyDbConnectError (with the local-stack hint) and discard the pooled client, and the raw-client idle error listener stops the process crash.
First-commit items look addressed. Not reopening the typed submit() subclass or the full execBatch+destroy wiring test.
Two non-blocking notes inline. Separate observation: keepalive idle is 5 minutes, so it will not save the ~2–3 minute CI timeouts in #6244 — the unwritable refusal is the CI-visible fix. Standalone exec still rendering a dead connection as At statement: N is documented and an intentional scope cut.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
TL;DR
fixes
supabase db resetandsupabase starthanging forever with no error when the database connection dies while migrations are being appliedwhats broken?
node-postgres silently discards every protocol frame once a socket stops being writable, while still reporting the write as successful, so the whole batch goes nowhere and the CLI waits forever on a reply the server was never asked for
it also leaves TCP keepalive off by default, where the Go CLI's driver had it on, so a peer that dies without a FIN or RST is never noticed either
fixed now by:
a server that stays alive but never answers still waits, matching the Go CLI, since that is indistinguishable from a long running statement
ref: