From 5f318a358c444c2aec08590f51d7c4528742967e Mon Sep 17 00:00:00 2001 From: AndresL230 <190146319+AndresL230@users.noreply.github.com> Date: Fri, 31 Jul 2026 19:00:47 -0700 Subject: [PATCH 1/4] ci: the migrate secret must be the session-mode pooler, not the direct URI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found while running the migration by hand: `db..supabase.co` publishes ONLY an AAAA record. This machine has no global IPv6 address (link-local only, despite an RA default route), so a direct connection dies with "Network is unreachable" before it authenticates. Canopy already recorded this from the #481 work — "direct psycopg to staging is blocked, IPv6-only endpoint" — but the workflow I merged in #506 told you to use the direct string, which walks straight into it. GitHub-hosted runners have no outbound IPv6 either, so its first real run would have failed the same way. The pooler hosts do publish A records, so the fix is the SESSION-mode pooler (port 5432), not transaction mode (6543) which drops the session-level behaviour psycopg and DDL rely on. db/migrate.py's "NOT the pooler" warning is about transaction mode and predates the IPv6-only endpoint; session mode behaves like a direct connection. The pooler also changes the username to `postgres.`, which is easy to miss. Corrects both the header rationale and the skip notice, so the thing you read when the secret is missing points at a host that is actually reachable. Co-Authored-By: Claude Opus 5 --- .github/workflows/migrate-staging.yml | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/.github/workflows/migrate-staging.yml b/.github/workflows/migrate-staging.yml index 85644276..c062b4b1 100644 --- a/.github/workflows/migrate-staging.yml +++ b/.github/workflows/migrate-staging.yml @@ -12,6 +12,19 @@ # separate `production` branch promotion, and auto-applying irreversible DDL to # prod on merge is a different risk decision. This runner has no down # migrations. +# +# THE SECRET MUST BE THE SESSION-MODE POOLER URI, NOT THE DIRECT ONE. +# `db..supabase.co` publishes only an AAAA record, and GitHub-hosted +# runners have no outbound IPv6 — a direct string fails with "Network is +# unreachable" / "server closed the connection unexpectedly" before it ever +# authenticates. (Same wall on a home network without a global IPv6 address; +# it is why staging migrations had to be applied by hand.) +# +# Use the pooler host on port 5432 — SESSION mode. Not 6543 (transaction mode), +# which drops the session-level behaviour psycopg and DDL rely on. db/migrate.py +# still says "NOT the pooler"; that warning is about transaction mode and +# predates the IPv6-only endpoint. Session mode behaves like a direct +# connection. Note the pooler also changes the username to `postgres.`. name: Migrate (staging) on: @@ -56,7 +69,7 @@ jobs: SUPABASE_DB_URL: ${{ secrets.STAGING_SUPABASE_DB_URL }} run: | if [ -z "${SUPABASE_DB_URL}" ]; then - echo "::notice::STAGING_SUPABASE_DB_URL is not set — skipping. Add the secret (the DIRECT connection string, port 5432, not the pooler) to enable." + echo "::notice::STAGING_SUPABASE_DB_URL is not set — skipping. Add the secret to enable: the SESSION-mode pooler URI (aws-0-.pooler.supabase.com:5432, user postgres.). Not the direct db..supabase.co string — that is IPv6-only and unreachable from GitHub runners — and not port 6543, which is transaction mode." echo "skip=true" >> "$GITHUB_OUTPUT" exit 0 fi From 59c2a051e59daf362dc06fdd171920f4324371b2 Mon Sep 17 00:00:00 2001 From: AndresL230 <190146319+AndresL230@users.noreply.github.com> Date: Fri, 31 Jul 2026 19:41:24 -0700 Subject: [PATCH 2/4] ops: pooler-URI builder and a read-only migration drift report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both came out of actually trying to migrate staging, and both encode something that cost real time to rediscover. pooler_url.py builds the SESSION-mode pooler URI from the password already in an env file, so the secret never has to be copied by hand. It takes the pooler host PREFIX rather than a bare region, because Supabase assigns projects to numbered clusters (aws-0-, aws-1-) and the number is not derivable from the region — staging is aws-1-us-west-2, which an aws-0- assumption gets wrong. migration_drift_report.py answers the question you must answer before applying a backlog to an environment that has been touched outside the repo (#317): is the ledger merely BEHIND, or is it LYING? It reports pending files, orphans (recorded here but absent from the repo — flagging filename NUMBER COLLISIONS, the dangerous shape), and any object a pending migration would create that already exists, noting whether that migration is IF NOT EXISTS-safe or would fail the whole run. Object lists are parsed from the migration SQL itself, so there is nothing to keep in sync by hand. Read-only: runs no DDL, safe against production. Verified against a real database both ways — clean (0 pending, 0 orphans, "behind, not lying") and with staging's shape simulated (unrecorded migrations plus a colliding orphan), where it correctly names the collision. Co-Authored-By: Claude Opus 5 --- backend/scripts/migration_drift_report.py | 139 ++++++++++++++++++++++ backend/scripts/pooler_url.py | 84 +++++++++++++ 2 files changed, 223 insertions(+) create mode 100644 backend/scripts/migration_drift_report.py create mode 100644 backend/scripts/pooler_url.py diff --git a/backend/scripts/migration_drift_report.py b/backend/scripts/migration_drift_report.py new file mode 100644 index 00000000..a98e60f4 --- /dev/null +++ b/backend/scripts/migration_drift_report.py @@ -0,0 +1,139 @@ +"""Read-only drift report between backend/db/migrations/ and a live database. + +Answers the question you must answer before applying a backlog of migrations to +an environment that has been touched outside the repo (#317): **is the ledger +merely behind, or is it lying?** + + SUPABASE_DB_URL=... python scripts/migration_drift_report.py + +Writes nothing. Runs no DDL. Safe against production. + +Three sections: + + PENDING — on disk, not in schema_migrations. + ORPHANS — in schema_migrations, not on disk. Means the environment ran SQL + that never existed in this repo (dashboard editor, ad-hoc script). + A filename collision here is the dangerous shape: staging having + recorded `0032_retire_summer_2026.sql` while the repo's own + `0032_rooms_missing_columns.sql` is pending means two different + migrations share a number. + ALREADY — objects a PENDING migration would create that ALREADY EXIST. + Every row here is the ledger lying: the schema moved without + being recorded, so "pending" overstates what will actually run. + Migrations written with IF NOT EXISTS will no-op safely; ones + without it will fail the whole run. + +The object list is parsed from the migration SQL itself (CREATE TABLE / ADD +COLUMN / CREATE INDEX), so it stays correct as migrations are added — nothing +to keep in sync by hand. +""" +from __future__ import annotations + +import os +import pathlib +import re +import sys + +import psycopg + +MIGRATIONS = pathlib.Path(__file__).resolve().parent.parent / "db" / "migrations" + +RE_TABLE = re.compile(r"create\s+table\s+(?:if\s+not\s+exists\s+)?([a-z0-9_.]+)", re.I) +RE_COLUMN = re.compile( + r"alter\s+table\s+([a-z0-9_.]+)\s+add\s+column\s+(?:if\s+not\s+exists\s+)?([a-z0-9_]+)", + re.I, +) +RE_INDEX = re.compile(r"create\s+(?:unique\s+)?index\s+(?:if\s+not\s+exists\s+)?([a-z0-9_]+)", re.I) + + +def bare(name: str) -> str: + return name.split(".")[-1] + + +def main() -> int: + url = os.getenv("SUPABASE_DB_URL") + if not url: + print("SUPABASE_DB_URL is not set", file=sys.stderr) + return 2 + + files = sorted(p.name for p in MIGRATIONS.glob("*.sql")) + conn = psycopg.connect(url) + + has_ledger = conn.execute( + "SELECT count(*) FROM information_schema.tables WHERE table_name = %s", + ("schema_migrations",), + ).fetchone()[0] + if not has_ledger: + print("NO LEDGER — this database has never been migrated by db/migrate.py.") + print("Reconcile with `python -m db.migrate --baseline` against a schema you") + print("have verified is current, rather than applying.") + return 1 + + recorded = { + r[0] for r in conn.execute("SELECT filename FROM schema_migrations").fetchall() + } + pending = [f for f in files if f not in recorded] + orphans = sorted(recorded - set(files)) + + print(f"on disk {len(files)} | recorded {len(recorded)} | pending {len(pending)}\n") + + print("PENDING") + for p in pending: + print(f" {p}") + if not pending: + print(" none") + + print("\nORPHANS (recorded here, absent from the repo)") + for o in orphans: + clash = [f for f in files if f.split("_")[0] == o.split("_")[0]] + note = f" <-- NUMBER COLLIDES WITH {clash[0]}" if clash else "" + print(f" {o}{note}") + if not orphans: + print(" none") + + # Which objects the pending migrations would create, that already exist. + print("\nALREADY EXISTS (pending migration, object already present)") + found_any = False + for name in pending: + sql = (MIGRATIONS / name).read_text() + hits: list[str] = [] + + for tbl in {bare(t) for t in RE_TABLE.findall(sql)}: + n = conn.execute( + "SELECT count(*) FROM information_schema.tables WHERE table_name = %s", + (tbl,), + ).fetchone()[0] + if n: + hits.append(f"table {tbl}") + + for tbl, col in {(bare(t), c) for t, c in RE_COLUMN.findall(sql)}: + n = conn.execute( + "SELECT count(*) FROM information_schema.columns " + "WHERE table_name = %s AND column_name = %s", + (tbl, col), + ).fetchone()[0] + if n: + hits.append(f"column {tbl}.{col}") + + for idx in set(RE_INDEX.findall(sql)): + n = conn.execute( + "SELECT count(*) FROM pg_indexes WHERE indexname = %s", (idx,) + ).fetchone()[0] + if n: + hits.append(f"index {idx}") + + if hits: + found_any = True + idempotent = "if not exists" in sql.lower() + flag = "safe: uses IF NOT EXISTS" if idempotent else "!! NO IF NOT EXISTS — would fail" + print(f" {name} ({flag})") + for h in sorted(hits): + print(f" {h}") + if not found_any: + print(" none — the ledger is behind, not lying") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/scripts/pooler_url.py b/backend/scripts/pooler_url.py new file mode 100644 index 00000000..3f1df65b --- /dev/null +++ b/backend/scripts/pooler_url.py @@ -0,0 +1,84 @@ +"""Build the SESSION-mode pooler connection string for an environment. + +Why this exists: `db..supabase.co` publishes only an AAAA record, so a +direct connection is unreachable from any host without global IPv6 — which +includes this laptop and GitHub-hosted runners. The Supavisor pooler hosts +publish A records, so they are the reachable path. + + python scripts/pooler_url.py .env.staging aws-1-us-west-2 # print (masked) + python scripts/pooler_url.py .env.staging aws-1-us-west-2 --raw # print usable URI + +Pass the pooler host PREFIX, not a bare region: Supabase assigns projects to +numbered pooler clusters (`aws-0-…`, `aws-1-…`) and the number is not derivable +from the region — staging is `aws-1-us-west-2`. A bare region is still accepted +and assumes `aws-0-`, which is a guess; take the prefix from the dashboard's +Connect panel instead. A wrong prefix fails fast and harmlessly with +"Tenant or user not found", which is distinguishable from a bad password +("password authentication failed"). + +The URI is derived from the password already in the env file, so the secret +never has to be copied by hand. `--raw` is what you feed to db.migrate; without +it the password is masked so the value is safe to look at (and to paste into a +chat or an issue). + +SESSION mode is port 5432 — NOT 6543. Transaction mode drops the session-level +behaviour psycopg and DDL rely on. db/migrate.py's "NOT the pooler" warning is +about transaction mode and predates the IPv6-only direct endpoint. + +The pooler also requires the username to carry the project ref +(`postgres.`), which is the detail most easily missed when assembling this +by hand. +""" +from __future__ import annotations + +import pathlib +import re +import sys +from urllib.parse import quote, urlparse + +SESSION_PORT = 5432 + + +def pooler_host(prefix: str) -> str: + """`aws-1-us-west-2` -> full host. A bare region gets the `aws-0-` guess.""" + if prefix.endswith(".pooler.supabase.com"): + return prefix + if not prefix.startswith("aws-"): + prefix = f"aws-0-{prefix}" + return f"{prefix}.pooler.supabase.com" + + +def build(env_file: str, region: str) -> str: + lines = pathlib.Path(env_file).read_text().splitlines() + matches = [ln for ln in lines if ln.startswith("SUPABASE_DB_URL=")] + if not matches: + raise SystemExit(f"{env_file}: no SUPABASE_DB_URL") + parsed = urlparse(matches[0].split("=", 1)[1].strip()) + if not parsed.password: + raise SystemExit(f"{env_file}: SUPABASE_DB_URL carries no password") + host = parsed.hostname or "" + # db..supabase.co -> + ref = host.split(".")[1] if host.startswith("db.") else host.split(".")[0] + pw = quote(parsed.password, safe="") + return ( + f"postgresql://postgres.{ref}:{pw}" + f"@{pooler_host(region)}:{SESSION_PORT}/postgres" + ) + + +def mask(uri: str) -> str: + return re.sub(r"://([^:]+):[^@]+@", r"://\1:********@", uri) + + +def main() -> int: + args = [a for a in sys.argv[1:] if a != "--raw"] + if len(args) != 2: + print(__doc__) + return 2 + uri = build(args[0], args[1]) + print(uri if "--raw" in sys.argv[1:] else mask(uri)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 148c888af44b0f4d713691865a2ded7a61201396 Mon Sep 17 00:00:00 2001 From: AndresL230 <190146319+AndresL230@users.noreply.github.com> Date: Fri, 31 Jul 2026 20:58:11 -0700 Subject: [PATCH 3/4] ops: drift report also checks pending UNIQUE indexes against live data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A UNIQUE index is the one thing IF NOT EXISTS cannot make safe: it still fails if the rows already present violate it. That is a DATA problem, invisible to a schema diff, and it is what turns a clean-looking backlog into a half-applied run partway through. The report now parses pending migrations for CREATE UNIQUE INDEX (including the partial-index WHERE clause) and runs the equivalent GROUP BY ... HAVING count>1 against the live table, naming the offending rows. Generic — it follows whatever happens to be pending rather than hardcoding today's case. Verified both directions against a real database: clean data reports none, and an injected duplicate is caught with the row identified (0036_offering_null_section_unique -> ('rich-course-math210','summer-2026',2)). Co-Authored-By: Claude Opus 5 --- backend/scripts/migration_drift_report.py | 36 +++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/backend/scripts/migration_drift_report.py b/backend/scripts/migration_drift_report.py index a98e60f4..147d9568 100644 --- a/backend/scripts/migration_drift_report.py +++ b/backend/scripts/migration_drift_report.py @@ -45,6 +45,17 @@ ) RE_INDEX = re.compile(r"create\s+(?:unique\s+)?index\s+(?:if\s+not\s+exists\s+)?([a-z0-9_]+)", re.I) +# A UNIQUE index is the one thing in a pending migration that IF NOT EXISTS +# cannot make safe: it still fails if the live data already violates it. That +# is a DATA problem, invisible to a schema-only diff, and it is what turns a +# clean-looking backlog into a half-applied run. Parsed so the check follows +# whatever migrations are actually pending. +RE_UNIQUE_INDEX = re.compile( + r"create\s+unique\s+index\s+(?:if\s+not\s+exists\s+)?([a-z0-9_]+)\s+" + r"on\s+([a-z0-9_.]+)\s*\(([^)]*)\)(?:\s*where\s+([^;]+))?", + re.I | re.S, +) + def bare(name: str) -> str: return name.split(".")[-1] @@ -132,6 +143,31 @@ def main() -> int: if not found_any: print(" none — the ledger is behind, not lying") + # Data blockers: a pending UNIQUE index that live rows already violate. + print("\nDATA BLOCKERS (pending UNIQUE index vs rows already present)") + blocked = False + for name in pending: + sql = (MIGRATIONS / name).read_text() + for idx, table, cols, pred in RE_UNIQUE_INDEX.findall(sql): + cols_sql = ", ".join(c.strip() for c in cols.split(",")) + where = f" WHERE {pred.strip()}" if pred else "" + try: + dupes = conn.execute( + f"SELECT {cols_sql}, count(*) FROM {bare(table)}{where} " + f"GROUP BY {cols_sql} HAVING count(*) > 1 LIMIT 5" + ).fetchall() + except Exception as exc: # table may not exist yet — that's fine + conn.rollback() + print(f" {name}: {idx} — could not check ({type(exc).__name__})") + continue + if dupes: + blocked = True + print(f" {name}: {idx} WOULD FAIL — duplicate rows exist:") + for d in dupes: + print(f" {d}") + if not blocked: + print(" none") + return 0 From 7d31c5daf8e53cbcb34bb61d1de0ab67af62365e Mon Sep 17 00:00:00 2001 From: AndresL230 <190146319+AndresL230@users.noreply.github.com> Date: Fri, 31 Jul 2026 23:19:17 -0700 Subject: [PATCH 4/4] fix(ops): stop double-encoding pooler passwords; finish the direct-URI sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review of this PR found four things, all of which undercut the PR's own premise that the next person shouldn't have to re-derive any of this. pooler_url.py double-encoded the password. urlparse() returns it STILL percent-encoded, and quoting again turned `p%40ss` into `p%2540ss`, so the URI authenticated as the literal escape text. It fails as "password authentication failed" — indistinguishable from simply holding the wrong secret, which is the expensive kind of wrong. Supabase generates passwords with reserved characters, so this was not hypothetical; it was waiting for the next rotation. Decode then re-encode, and pin it with tests, because nothing about the output looks wrong until you try to connect. The workflow's skip-notice hardcoded `aws-0-` while pooler_url.py, added in the same PR, calls that a guess and records that staging is on `aws-1-`. Verified against both live projects: staging answers only on aws-1-us-west-2, production only on aws-0-us-west-2, same region. An operator copying the notice got "Tenant or user not found" — the exact failure class this PR exists to delete. The notice now points at the dashboard and at the builder script. db/migrate.py still told operators the opposite of the PR. Its docstring said "the direct connection string, NOT the pooler" and main()'s unset-variable error routed the reader to Connection string -> Direct. This PR's own repro was running `python -m db.migrate`, so that was the one path left misdocumented. The docstring now explains why the old warning existed (it is still right about transaction mode / 6543) and why it no longer decides the answer (the direct host went IPv6-only). Same correction in CLAUDE.md, README.md, and docs/staging/setup-checklist.md — the checklist being the document someone actually follows to set staging up. Two smaller things while in the same file: the drift report's docstring said "Three sections" after a fourth was added, and main() returned 0 even while printing orphans or data blockers. That second one is a trap for the obvious next refactor — having the workflow call this script instead of duplicating its preflight — which would have silently downgraded a fail-on-orphan gate into a report nobody checks. It now exits 1 on orphans, a non-idempotent collision, or a data blocker; PENDING alone stays clean, since being behind is not drift. Also folds in the one finding from #509's review that cleared review but landed after merge: CLAUDE.md's Commands section still said "add a new numbered file", which #509's own CI guard now rejects. Verification: 1550 passed, 38 skipped (8 new). ruff clean. Drift report re-run against live staging returns exit 1 and correctly names the 3 orphans. No request-path or schema change, so the e2e lanes have nothing to exercise here; the ledger reconciliation that follows will take the full cycle. Co-Authored-By: Claude Opus 5 --- .github/workflows/migrate-staging.yml | 17 ++-- CLAUDE.md | 8 +- README.md | 4 +- backend/db/migrate.py | 29 +++++-- backend/scripts/migration_drift_report.py | 31 +++++++- backend/scripts/pooler_url.py | 11 ++- backend/tests/test_pooler_url.py | 96 +++++++++++++++++++++++ docs/staging/setup-checklist.md | 7 +- 8 files changed, 184 insertions(+), 19 deletions(-) create mode 100644 backend/tests/test_pooler_url.py diff --git a/.github/workflows/migrate-staging.yml b/.github/workflows/migrate-staging.yml index c062b4b1..d7ac79ed 100644 --- a/.github/workflows/migrate-staging.yml +++ b/.github/workflows/migrate-staging.yml @@ -21,10 +21,17 @@ # it is why staging migrations had to be applied by hand.) # # Use the pooler host on port 5432 — SESSION mode. Not 6543 (transaction mode), -# which drops the session-level behaviour psycopg and DDL rely on. db/migrate.py -# still says "NOT the pooler"; that warning is about transaction mode and -# predates the IPv6-only endpoint. Session mode behaves like a direct -# connection. Note the pooler also changes the username to `postgres.`. +# which drops the session-level behaviour psycopg and DDL rely on. Session mode +# behaves like a direct connection. Note the pooler also changes the username to +# `postgres.`. +# +# Take the host from the dashboard's Connect panel rather than assembling it: +# projects are assigned to NUMBERED pooler clusters (`aws-0-`, `aws-1-`, ...) +# and the number is NOT derivable from the region — staging and production are +# both us-west-2 yet sit on different clusters. A wrong prefix fails with +# "Tenant or user not found", which is at least distinguishable from a bad +# password. `backend/scripts/pooler_url.py` builds the URI from an env file so +# the password is never copied by hand. name: Migrate (staging) on: @@ -69,7 +76,7 @@ jobs: SUPABASE_DB_URL: ${{ secrets.STAGING_SUPABASE_DB_URL }} run: | if [ -z "${SUPABASE_DB_URL}" ]; then - echo "::notice::STAGING_SUPABASE_DB_URL is not set — skipping. Add the secret to enable: the SESSION-mode pooler URI (aws-0-.pooler.supabase.com:5432, user postgres.). Not the direct db..supabase.co string — that is IPv6-only and unreachable from GitHub runners — and not port 6543, which is transaction mode." + echo "::notice::STAGING_SUPABASE_DB_URL is not set — skipping. Add the secret to enable: the SESSION-mode pooler URI on port 5432, user postgres.. Take the host from the dashboard's Connect panel — projects are assigned to NUMBERED pooler clusters (aws-0-, aws-1-, ...) and the number is not derivable from the region, so do not assume aws-0. 'python scripts/pooler_url.py .env.staging --raw' builds the URI from the env file. Not the direct db..supabase.co string — that is IPv6-only and unreachable from GitHub runners — and not port 6543, which is transaction mode." echo "skip=true" >> "$GITHUB_OUTPUT" exit 0 fi diff --git a/CLAUDE.md b/CLAUDE.md index 833fc93e..1807f71d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,14 +45,18 @@ python -m pytest tests/ -q # backend test suite Database (run from `backend/`; migrations are raw DDL, never dashboard SQL): ``` -python -m db.migrate # apply pending migrations (needs SUPABASE_DB_URL = direct conn string) +python -m db.migrate # apply pending migrations (SUPABASE_DB_URL = SESSION-mode pooler URI, port 5432) python -m db.migrate --baseline # record migrations as applied without running them python -m db.seed_staging # idempotent fake demo dataset on the new schema ``` The `db/` scripts read `.env` by default; for staging/prod ops run them under `dotenv -f .env.staging run -- python -m db.