Skip to content

fix(cli): detect dead db connections (CLI-2207) - #6277

Merged
7ttp merged 12 commits into
developfrom
7ttp/cli-2207-supabase-db-resetstart-migration-apply-sql-pipeline-stops
Aug 25, 2026
Merged

fix(cli): detect dead db connections (CLI-2207)#6277
7ttp merged 12 commits into
developfrom
7ttp/cli-2207-supabase-db-resetstart-migration-apply-sql-pipeline-stops

Conversation

@7ttp

@7ttp7ttp commented Aug 19, 2026

Copy link
Copy Markdown
Member

TL;DR

fixes supabase db reset and supabase start hanging forever with no error when the database connection dies while migrations are being applied

whats broken?

node-postgres silently discards every protocol frame once a socket stops being writable, while still reporting the write as successful, so the whole batch goes nowhere and the CLI waits forever on a reply the server was never asked for

it also leaves TCP keepalive off by default, where the Go CLI's driver had it on, so a peer that dies without a FIN or RST is never noticed either

fixed now by:

  • failing a batch that never reached the wire as a connection error, carrying the driver's own reason instead of blaming the migration's first statement
  • discarding that pooled connection, so the next batch redials instead of writing into the same dead socket
  • turning TCP keepalive on for every connection the CLI opens, so a silently dead peer is eventually detected instead of waited on

a server that stays alive but never answers still waits, matching the Go CLI, since that is indistinguishable from a long running statement

ref:

@7ttp7ttp self-assigned this Aug 19, 2026
@7ttp
7ttp requested a review from a team as a code ownerAugust 19, 2026 22:33
@github-actions

github-actionsBot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@5a80fb2bdfe03d8fbca22a2763f3920a0d733e67

Preview package for commit 5a80fb2.

@avalleteavallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core fix is sound — I traced the load-bearing pieces and they hold: the widened discard predicate strictly covers the old poisoned check, the only execBatch caller guards LegacyDbConnectError before touching statementIndex, keepalive reaches both the pool and the raw client, and pg honors the submit() error return so the writable check settles instead of hanging. I also chased two scary scenarios that turned out impossible: there's no residual hang on exec/query (pg's 'close'_errorAllQueries fails them within a tick), and no applied-but-unrecorded migration (the history INSERT rides inside the batch).

I reproduced the findings below against this branch where marked; one is a process crash I'd fix before merge. The script used for the reproduced non-crash findings:

verify-findings.ts (run with bun from the repo root)
// Verification of the review findings against this branch's real modules.// Save at the repo root and run: bun verify-findings.tsimport{LegacyPgBatchQuery,legacyBatchFailureError,legacyToExecError,}from"./apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts";import{legacyFormatExecBatchError}from"./apps/cli/src/legacy/shared/legacy-migration-apply.ts";importtype*asPgfrom"pg";console.log("=== standalone exec renders dead connection as 'At statement: N' ===");constdeadConnErr=newError("Client has encountered a connection error and is not queryable");constexecMapped=legacyToExecError(deadConnErr);constrendered=legacyFormatExecBatchError(execMapped,3,"CREATE INDEX CONCURRENTLY idx ON t (c)");console.log(rendered.message);console.log("\n=== poisoned batch blames statement 0 ===");letbindCalls=0;constnoop=()=>{};constconn={stream: {writable: true,cork: noop,uncork: noop},parse: noop,bind: ()=>{bindCalls+=1;if(bindCalls===7)thrownewError("frame serialization blew up on statement 7");},describe: noop,execute: noop,sync: noop,}asunknownasPg.Connection;conststatements=Array.from({length: 10},(_,i)=>({sql: `SELECT ${i} /* stmt ${i} */`}));constbatch=newLegacyPgBatchQuery(statements,noop);constsubmitErr=batch.submit(conn);console.log("poisoned:",batch.poisoned,"submitted:",batch.submitted,"completed:",batch.completed);constfailure=legacyBatchFailureError(submitErr!,batch);console.log("mapped:",failure._tag,"statementIndex:",(failureas{statementIndex?: number}).statementIndex);console.log("\n=== batch-lost ConnectError has no suggestion ===");constunsent=newLegacyPgBatchQuery([{sql: "SELECT 1"}],noop);constunsentErr=unsent.submit({stream: {writable: false}}asunknownasPg.Connection);constconnectFailure=legacyBatchFailureError(unsentErr!,unsent);console.log(connectFailure._tag,"| suggestion:",(connectFailureas{suggestion?: string}).suggestion);console.log("\n=== message-equality sentinel stutters once wrapped ===");constwrapped=newError(`batch submit failed: ${unsentErr!.message}`);console.log(legacyBatchFailureError(wrapped,unsent).message);

Comment threadapps/cli/src/legacy/shared/legacy-migration-apply.ts
Comment threadapps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts Outdated
Comment threadapps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts Outdated
Comment threadapps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts Outdated
Comment threadapps/cli/src/legacy/shared/legacy-db-connection.sql-pg.integration.test.ts Outdated

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:b7397db3f1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment threadapps/cli/docs/go-cli-divergences.md Outdated
Comment threadapps/cli/docs/go-cli-divergences.md Outdated
@7ttp
7ttp requested a review from avalleteAugust 22, 2026 10:14

@avalleteavallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed HEAD after the follow-ups. The hang-fix path holds: submit() refuses an already-unwritable socket, pg 8.23 delivers that Error through handleError, unsent batches surface as LegacyDbConnectError (with the local-stack hint) and discard the pooled client, and the raw-client idle error listener stops the process crash.

First-commit items look addressed. Not reopening the typed submit() subclass or the full execBatch+destroy wiring test.

Two non-blocking notes inline. Separate observation: keepalive idle is 5 minutes, so it will not save the ~2–3 minute CI timeouts in #6244 — the unwritable refusal is the CI-visible fix. Standalone exec still rendering a dead connection as At statement: N is documented and an intentional scope cut.

@7ttp
7ttp added this pull request to the merge queueAug 25, 2026
Merged via the queue into develop with commit 11878aeAug 25, 2026
32 checks passed
@7ttp
7ttp deleted the 7ttp/cli-2207-supabase-db-resetstart-migration-apply-sql-pipeline-stops branch August 25, 2026 10:09
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

supabase db reset/start: migration-apply SQL pipeline stops mid-stream with no server-side counterpart

2 participants

@7ttp@avallete