Skip to content

postgres.js throws from its own error handler on a dead socket, killing the process #169

Description

@Hazzng

Summary

When a Postgres connection dies mid-script, postgres.js can throw out of its own error handler, escaping as an uncaught exception that kills the API process.

node_modules/postgres/src/connection.js:173 writes to the socket; on a nulled socket it throws. The catch at :179 then calls write(Sync) on the same dead socket, which throws again — escaping execute, escaping index.js:295, and terminating the process.

This is still live. A sibling crash mechanism in our own code (an abort racing the script-tx open) was fixed in aa60a5a; this one is inside the driver and needs a different approach.

Reproduction, with no admin command

ALTER DATABASE ... SET idle_in_transaction_session_timeout = '1500ms', then a three-write script with a sleep 3 in the middle. No kill is issued.

PG_POOL_MAX Result
1 HTTP 500, torn commit (see #TORN)
2 (the default) process CRASHED, 3/3 runs (HTTPCODE=000)

This matters because a script scope pins one backend idle in transaction for the entire duration of arbitrary user bash — any sleep, Python/JS step, or network fetch. idle_in_transaction_session_timeout is default-on or standard hardening on managed Postgres. A pooler dropping a server connection is the same class.

With the crash suppressed via a preload harness, the request hangs indefinitely instead (client timeout at 120 s, zero committed) — so the underlying handling is wrong either way, the crash is just the loudest symptom.

Impact

  • A single unlucky script kills the replica, dropping every other in-flight request on it.
  • In a pooled deployment, where idle in transaction backends are exactly what gets reaped, this is a crash-loop shape rather than a one-off.

Options

  1. Avoid handing queries to a dead connection at all — largely achieved by the fail-closed script-tx guard in 52cc836, which stops our code issuing further statements after the connection is lost. That narrows the window substantially but does not prove the driver can never reach connection.js:179 by another path.
  2. A process-level uncaughtException handler that logs and fails the in-flight request instead of exiting. Blunt, and it must not mask unrelated bugs.
  3. Upstream: report to postgres.js — the error path should not perform I/O on the socket it is handling an error for.
  4. Structural: stop holding a transaction open across arbitrary user code (script-tx pins a pooled connection for the whole script; deadlocks at default_pool_size/2 #166), which removes the condition that makes this reachable.

Provenance

Found during pre-merge load testing of #162. Not introduced by that PR. thoughts/shared/plans/2026-05-02_bulk-fs-ops-script-tx.md records an earlier, related crash ("Bug 2 — Unhandled rejection crash when Neon closes the connection mid-transaction") fixed on the sibling promise in May.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingseverity:highHigh severity

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions