Squash the migration chain at floor V135, add verify-and-repair, and repair the V56/V57 flush-order fallout (V164)

[timujinne]

Squash the migration chain at floor V135, ship verify-and-repair

Branch: squash-migrations. This document is the PR description — written for
the maintainer, who has not followed the multi-week working history behind it.
Full design rationale lives in dev_docs/plans/2026-07-14-squash-migrations-spec.md
(spec, REVIEWED DRAFT r2) and its companion
dev_docs/plans/2026-07-14-squash-inventory.md (per-version classification of
every migration folded into the baseline, plus addenda for everything added
since). This description summarizes; it does not replace either.

What changes, and why

PhoenixKit's migration chain had grown to 164 versioned modules
(lib/phoenix_kit/migrations/postgres/v01.ex .. v164.ex, ~28,000 lines) at a
rate of roughly 16 new versions a month. Nothing in that chain was wrong, but
almost none of it does anything on a fresh install except reproduce
intermediate shapes a later version immediately changes again — V01..V134
alone account for ~23,000 of those lines and every one of them has already run,
unmodified, on every install this project has committed to supporting.

This PR:

1. Replaces V01..V134 with one generated baseline, V135. It applies
   the cumulative schema those 134 versions produce, directly, in one step —
   no intermediate drops, renames, or backfills replayed. It was generated by
   tool (dev_docs/squash/generate_baseline.exs), not hand-written, from a
   version-by-version catalog introspection of a real migrated database, and
   is regenerable the same way after any future rebase.
2. Refuses to migrate a database below the floor, rather than guessing. A
   database at 0 < migrated_version < 135 gets a PhoenixKit.Migrations.BelowFloorError
   naming the bridge release, from up/1, down/1, and mix phoenix_kit.update's
   generation step — not a silent attempt to replay a chain that no longer
   exists. A fresh install (migrated_version == 0) instead clamps straight to
   the baseline, so an unpinned installer or a below-floor-pinned consumer
   wrapper both keep working unmodified (spec §5.2 D13, §5.3).
3. Ships a mix phoenix_kit.repair verify-and-repair capability, built
   from the same generated manifest the baseline uses
   (lib/phoenix_kit/migrations/expected_schema.ex) — additive-only,
   never re-executes a delta module, never touches data, reports (never
   auto-fixes) anything divergent. This exists independently of the squash and
   is useful against the full pre-squash chain too; it landed first, as
   preparatory work, and is what backs mix phoenix_kit.doctor's new
   verify-read-only pass.
4. Repairs a real, previously undetected data-shape defect as part of the
   same release — see "The V164/V163 repair" below. This is the one item in
   this PR that changes behavior on an already-migrated database, not just the
   shape of the migration files.

Everything else — @current_version, the version-comment tracking mechanism,
the delta modules above the floor (V136..V164, 29 files, untouched in
content) — is unchanged.

The floor: V135

Floor = 135, picked per the spec's D1/D2 rule (minimum confirmed
migrated_version across every install this maintainer has committed to
seamless-upgrade support — one install, the local host app, confirmed live at
V160 on 2026-08-04, well above 135; 135 was chosen for headroom over the
minimum, not as the minimum itself — see spec §4 for the full floor-candidate
comparison). Below the floor there is nothing to bridge additively — several
versions in that range perform non-additive transforms (V58's timestamp→
timestamptz retype, V74's bigint-id-to-uuid-PK promotion, V114's settings-key
identity rewrite) that only the real, unfolded 1.7.x chain can produce, which
is exactly why the guard raises instead of attempting a partial repair.

The bridge

The last 1.7.x release remains on Hex permanently (packages are
immutable) as the stopover for anything below the floor. On this branch that
release is {:phoenix_kit, "~> 1.7.235"} (mix.exs@version, unmodified by
this PR — version bumps are maintainer-owned, see "Open decisions" below).
Path for a below-floor consumer: pin the bridge, run mix phoenix_kit.update,
confirm the version comment reaches V135 or above on every environment
(dev/staging/prod each have their own database), then move the pin to ~> 2.0.
Full walkthrough: dev_docs/guides/2026-08-07-upgrading-to-2.0-guide.md.

The V164/V163 repair

Two migrations at the top of the chain repair schema defects that predate this
PR and are unrelated to the squash mechanism itself — one is this branch's own
work, the other is yours, merged in while this branch was in flight, and the
two are now load-bearing in a specific order.

V163 (yours, PR #688, "UUID primary-key integrity") is a catalog-driven
repair for any phoenix_kit_* table whose uuid column is the wrong type,
nullable, or not the primary key — a state V40/V56/V74 each assumed
impossible and a production install reached anyway (the reported case:
phoenix_kit_email_events.uuid as varchar(255), nullable, no PK, invisible
to V40's existence-only guard and to V56's later-added type conversion).

V164 (this branch's) repairs the fallout of a genuine ordering defect in
V56/V57: both called a shared helper's up/1 (queues ADD COLUMN for
~80 *_uuid FK columns) immediately followed by that helper's own
add_constraints/1, whose guards run immediateinformation_schema queries
— with no flush() between the two. On an incremental, per-version chain run
this is invisible (Ecto flushes between migration modules regardless), but
on a single-shot run — which is every fresh install mix phoenix_kit.install
produces, since it emits one unpinned wrapper for the whole chain — the guards
ran against database state that had not yet seen the columns just queued
moments earlier, and failed closed: ~46 *_uuid columns were left nullable
instead of NOT NULL, and ~67 of the 70 declared foreign keys were never
created at all.phoenix_kit_comments.fk_comments_user_uuid specifically was
never created; V72, finding it missing later, guessed ON DELETE CASCADE
instead of V56/V57's own already-declared SET NULL. V164 re-imposes
NOT NULL (only where the column currently has zero NULL rows — never
backfills live data with an invented value; otherwise warns with table, column
and row count and leaves it), creates the missing FKs NOT VALID + VALIDATE
(a failed validation is left NOT VALID with an orphan-count diagnostic, never
retried automatically, never a crash), and corrects fk_comments_user_uuid
from CASCADE to SET NULL. It also folds in two unrelated, unplanned-for-a-
second-migration fixes for objects that diverged between public and
named-schema installs because their DDL was not prefix-qualified
(idx_publishing_posts_group_slug's missing partial predicate from V68, and
the ..._subscription_plans_slug_uidx → ..._types_slug_uidx rename from
V65) — ships as one migration, not two, by explicit decision (see the
inventory's "Post-V161 additions" for the full reasoning). All three repairs
are additive/corrective and no-ops on a healthy install.

The order is load-bearing, not incidental.V163 runs immediately before
V164 in this merged chain, and it has to: a foreign key cannot reference a
column with no unique or primary key, so V164's FK repair depends on V163
having already promoted the affected uuid columns to primary keys. Both
apply in the same mix phoenix_kit.update run for a real upgrade; nobody
needs to sequence them by hand, but anyone re-deriving this chain (a fork, a
cherry-pick, a re-squash) needs to preserve that order.

Full operational detail for anyone running this against production — lock
behavior, expected orphan counts measured against a real pre-production
database, what a degraded (some constraints left NOT VALID) outcome looks
like, how to re-run after a partial failure — is in the upgrade guide, not
duplicated here.

How this was verified

dev_docs/squash/verify.exs implements the spec's §8.2 scenario matrix (S1
through S22); what each scenario actually proves, and what is deliberately
left unproven, is dev_docs/squash/COVERAGE.md — read that before trusting a
summary number. As of the full --mode b run on 2026-08-08: 21 PASS, 0
FAIL, three named SKIPs (s4_seed — needs the pre-squash checkout by
construction; s16 — Oban-delegation assertion body still pending; s18 —
needs a manually triggered concurrent migration, not automatable). Concretely,
among what that run covers:

• Fresh-install equivalence (S1/S2): a normalized pg_dump --schema-only
  diff and seed-row dump between the new (squashed) chain and the old
  (pre-squash) chain, empty modulo a committed, enumerated whitelist of known
  bimodal drift — any unlisted diff fails the scenario.
• Below-floor guard (S4): up, down, and ensure_current/2 on a
  persistent below-floor install each raise the specific BelowFloorError
  struct with the bridge message, not just "some error."
• Consumer wrapper replay (S5): replays the host application's real
  54-file wrapper set (1 unpinned installer + 48 pinned update wrappers + 5
  interleaved consumer-authored migrations that ALTER phoenix_kit tables),
  including a rollback→migrate round trip and the specific, documented failure
  mode an unguarded consumer migration produces against the new floor shape.
• The V164 repair itself (S21): damages a healthy install the way the
  flush defect did — drops every declared FK, drops every re-imposed
  NOT NULL, flips the comments FK back to CASCADE — runs the chain, and
  asserts full restoration; re-runs it and requires a byte-identical no-op;
  and, on a separate schema, leaves an orphan row in place and requires the
  constraint to land NOT VALID with the row untouched.
• Per-version composition (S22): applies each delta V136..V164 in its
  own separate migrator invocation and requires the result to equal one
  single-shot run — the property the historical V56/V57 defect violated.
• A second install-path axis (Mode A, public-schema installs, as opposed
  to the named-schema installs every other scenario uses by default): this is
  how the two prefix-unsafe objects V164 normalizes were actually found — a
  public install and a named-schema install diverged on two objects that
  every purely named-schema scenario is structurally blind to. COVERAGE.md's
  "Hazard: install-PATH bimodality" section has the mechanism.

What is not yet proven, stated as plainly as what is. The "21 PASS" run
above predates this branch's two most recent commits — the final renumber of
this branch's own repair delta to V164, and the merge that brought your
V163 in. The generated manifest
(lib/phoenix_kit/migrations/expected_schema.ex) was not regenerated for the
V163 merge (confirmed: git show <merge-commit> --stat -- lib/phoenix_kit/migrations/expected_schema.ex
is empty) and has no knowledge of V163's objects yet. Regenerating the
manifest and re-running the full matrix against current HEAD is an open item
before this PR should be considered fully verified — flagged, not fixed, by
this pass (see "Open decisions" below).

Deliberately never proven here, and why: a second PostgreSQL major (needs
an operator-provided container with a different major; until then a finding
that rests only on expression-rendering differences between majors is reported
informational, not drift); the reverse-direction concurrency race (a migration
starting mid-repair — the advisory lock only prevents the opposite order; this
one is detected via a comment re-read, not prevented, and needs a manually
triggered two-process trigger to exercise, spec §6.1).

Test-surface changes

v106_test.exs and v114_test.exs deleted with nothing ported (both pinned
mechanism for migrations that no longer exist as distinct steps; the schema
facts they protected are already correct-by-omission in the generated
manifest). v107_test.exs/v112_test.exs/v113_test.exs/v125_test.exs had
their load-bearing final-shape assertions (column types, index predicates, FK
delete actions, CHECK enforcement) ported verbatim into the new
v135_baseline_schema_test.exs; their upgrade-only backfill-correctness
assertions were dropped as unreachable below the floor. v145_test.exs
survives untouched. Full accounting: inventory's "What the squash changed"
section.

Upgrading

Full walkthrough, including lock behavior and maintenance-window guidance for
the V163/V164 repair pair: dev_docs/guides/2026-08-07-upgrading-to-2.0-guide.md.
In short: land on the bridge (~> 1.7.235) first, confirm the version comment
reaches V135 or above on every environment, then move the pin to ~> 2.0
and run mix phoenix_kit.update as usual — the repair applies automatically
as part of that run, not as a separate step.

Open decisions — yours, not this branch's

This branch does not touch CHANGELOG.md or mix.exs's @version
deliberately — both are maintainer-owned by this project's own convention.
Specifically outstanding:

1. The 2.0.0 version bump itself.mix.exs on this branch still reads
   1.7.235. Breaking upgrade contract (below-floor installs are refused, not
   migrated) argues for a MAJOR bump per the spec's D8; the actual number and
   the CHANGELOG entry are yours to write.
2. Whether this lands as one PR or two. The spec's original rollout plan
   (§7.2) called for two independently-mergeable PRs — a "pre-squash" one
   (repair engine, manifest generator, gen.migration/release_check fixes,
   zero contention with v*.ex) landing first, then a separate "atomic
   squash" PR. In practice both stages were built as sequential commits on
   this one branch, and nothing has merged to upstream/main from it yet —
   this PR is a single PR carrying both. Splitting it before review, if you'd
   rather review the repair engine independently of the deletions, is your
   call; it can be done by cherry-picking the pre-squash-labeled commits onto
   a separate branch.
3. Manifest regeneration + a fresh full verify run against current HEAD
   (see "What is not yet proven" above) — the V163 merge needs the manifest
   regenerated before the "21 PASS" result can be re-confirmed as still
   holding.
4. The ~14-package module-ecosystem pin-widening wave (spec §7.4). Every
   phoenix_kit_* module checked at /www still pins a bare ~> 1.7.x with
   nothing that resolves against 2.0 (re-verified 2026-08-08 — e.g.
   catalogue/locations~> 1.7.189, warehouse/entities~> 1.7.214,
   ecommerce/manufacturing/projects~> 1.7.231). {:phoenix_kit, "~> 2.0"}
   plus any of these modules is a hard resolver conflict until each ships a
   widened pin (a patch release per module, since none call migration
   internals directly). Sequencing that coordinated wave against the 2.0.0
   publish is a maintainer-level decision this branch cannot make.
5. Two small stale-text bugs left by this branch's own renumbering,
   confirmed live 2026-08-08 and not fixed by this docs pass (docs-only —
   no lib//test/ edits in this PR): v164.ex's moduledoc still names
   test/phoenix_kit/migrations/v163_relaxed_columns_test.exs and
   V163RelaxedColumnsTest (the real names are v164_relaxed_columns_test.exs
   / V164RelaxedColumnsTest), and — more visibly — the IO.warn a
   production run prints when a constraint is left NOT VALID says "the
   version comment now reads 163", which is simply wrong (up/1 self-stamps
   164). test/phoenix_kit/migrations/v164_relaxed_columns_test.exs carries
   the matching artifact, @exempt_version 163. None of this is a
   schema-correctness bug — only prose and one log message — but it should be
   fixed before merge, in a normal code commit, not this documentation pass.

A correction we owe you

Your V163's own moduledoc entry did not survive cleanly into this branch's
postgres.ex: after the merge, its heading landed positioned above V162's
body, and V162's own heading was lost — mechanical fallout of two
moduledoc-collapsing edits (this branch's V01..V134 collapse, and your
ordinary per-release entry) landing on the same lines from different
directions. This branch's merge resolution restored both headings, in the
correct order and with V163's content reconstructed from v163.ex's own
moduledoc (postgres.ex's V164/V163/V162/V161 block, as shipped in
this PR). This is not specific to anything this branch changed — your own
main, independent of this PR, likely has the same defect from whatever
follow-on release collapsed the moduledoc next; worth checking there too.

[timujinne]

Post-merge review of the migration work — Kimi K3, verified

Second-opinion pass over what landed in #688, #689 and the two fix commits on
top (b3ef57b2, 1aea3d28), by Kimi K3 at max effort. It was explicitly told
not to re-report two defects already known (the pk_sqv schema left behind by
verify.exs combined with baseline tests that query pg_indexes without a
schemaname filter; and the post-rename V164 drift where the comment reads 164
while the index V164 creates does not exist), so everything below is new.

I verified every finding against the tree before posting this. Where the
code's own comments prove the point, I quote them rather than paraphrase. One
finding it flagged as unverified I checked myself and it holds.

---

BUG - HIGH — a comment-less database is routed to the bridge replay the migrator itself calls destructive

lib/phoenix_kit/install/common.ex:270-272 → lib/mix/tasks/phoenix_kit.update.ex:463-464

query_version_directly/2 ends with, verbatim:

_-># Table exists but no version comment - assume version 11

So a database that is current but lost its comment — a half-installed or
adopted state that repair --adopt exists to handle — reports
{:current_version, 1}. update.ex:463 then sees 1 > 0 and 1 < 135 and emits
the below-floor notice: "Install the last PhoenixKit 1.7.x release (the
migration bridge) first."

The migrator refuses the same input, and its refusal text says why that advice
is dangerous (postgres.ex:731-747):

Pre-squash a missing comment was read as "V01 legacy" and the chain simply
replayed from the start; this release cannot do that — below-floor versions no
longer exist here, so the guess would route a possibly CURRENT database to the
1.7.x bridge, whose backfill overwrites still-NULL tracked columns with freshly
generated uuids pointing at nothing.

Two refusal messages for one state, pointing opposite ways, and the operator
meets the update task's one first. Downstream on the bridge, the replay is not
benign on a populated database: UUIDFKColumns.set_not_null/4 backfills
legitimately-NULL phoenix_kit_files rows with invented uuids, and
cleanup_orphaned_fk_refs/5 then deletes them for matching no user.

Note the other fallback in the same module was already hardened — its comment
records that it "used to fall back to a fabricated {:current_version, 1}…
Report honestly instead." The fabrication survives one level down.

Fix: distinguish "no comment" from "comment = 1" in
check_installation_status/1, and point the no-comment case at doctor +
restamp rather than the bridge.

BUG - HIGH — the manifest pins uuid_generate_v7's body to public.gen_random_bytes, so repair can never converge where pgcrypto lives elsewhere

lib/phoenix_kit/migrations/expected_schema.ex — the function object's stored
definition contains public.gen_random_bytes(10) and is compared raw in
repair/differ.ex. The self-heal path regenerates that body through
Helpers.ensure_uuid_v7_function/2, which writes that install's pgcrypto
schema.

Install with pgcrypto in a schema other than public — the DBA-provisioned
topology PhoenixKit.Migration's own moduledoc supports — and
mix phoenix_kit.repair reports :wrong_shape → :repaired on every run,
forever, because the reissued body can never match the pinned md5. --adopt is
not blocked by it (:repaired is repairable severity), so adopt stamps the
floor while the finding persists.

Fix: normalise the pgcrypto schema out of prosrc on both sides before
hashing, the same way the array-cast canonicalisation already works in Differ.

BUG - HIGH — the manifest's pre-164 shape for idx_publishing_posts_group_slug disagrees with every healthy public install

lib/phoenix_kit/migrations/expected_schema.ex:36115-36161

The earlier revisions carry predicate: nil; the partial form appears only at
revision 164. The correction comment sitting between them says:

Every real install has the predicate; V164 normalizes the rest onto it.

Both statements cannot hold. shape_at(object, 163) selects a predicate: nil
revision, so a public install at comment 135..163 — every host that finishes
the bridge and runs repair or doctor before migrating — gets :wrong_shape,
error severity, exit 2, reporting expected nil, got "(slug IS NOT NULL)" on a
byte-correct database. Named-schema installs verify clean; this false-positives
the dominant topology only.

The sibling case one table away was handled as :legacy_optional. This one
needs the same, or a bimodal revision at 68.

BUG - MEDIUM — V164's post-add check matches a constraint by name only, so a collision reads as :created

lib/phoenix_kit/migrations/postgres/v164.ex:644-660

fk_exists?/3 queries conname + relname + nspname — no contype, no shape:

SELECT EXISTS (SELECT1FROM pg_constraint c … WHEREc.conname='…'ANDt.relname='…'ANDn.nspname='…')

Pre-create detection is shape-based, so a differently-shaped constraint under
the expected name returns :absent; the guarded ADD then fails 42710 and is
swallowed by the EXCEPTION WHEN OTHERS handler; the verification sees the
colliding constraint and reports :created. One buried RAISE WARNING, no
{:failed, …}, no SUMMARY line, comment stamped 164, declared FK absent.

Same class as the known V164 drift — a verification step matching the wrong
thing — inside the migration whose purpose is repairing that class.

BUG - MEDIUM — repair dies on the anomaly it exists to diagnose

lib/phoenix_kit/migrations/repair/probe.ex:81 calls String.to_integer(version)
unguarded and untrimmed, eleven lines below a docstring that says "Never
raises" (:51). A hand-edited comment — 'v164', ' 164', the exact examples
postgres.ex:682-685 cites — crashes mix phoenix_kit.repair with a bare
** (ArgumentError) argument error. doctor survives only because its check
wraps everything in rescue.

The same unguarded parse is in install/common.ex:268, on the path described in
the first finding.

IMPROVEMENT - MEDIUM — the below-floor notice still does not name the bridge

lib/mix/tasks/phoenix_kit.update.ex:611 says "the last PhoenixKit 1.7.x
release (the migration bridge)". The fix commit threaded @bridge_version "1.7.236" into both BelowFloorError raise sites but not into the message
below-floor operators see first, at generation time.

IMPROVEMENT - MEDIUM — v163.ex scans before it checks the size guard

castable?/3 (a full-table count(*) … !~*) runs before the 2M-row guard, so
the deferral path for a huge broken varchar table still pays an unbounded
sequential scan. Read-only, so not the ACCESS EXCLUSIVE outage the guard exists
for — but on a PgBouncer-fronted pool it pins a connection for minutes on
exactly the tables the limit targets. Check estimated_rows/3 first.

The one it could not verify — I checked, and it holds

It asked whether dev_docs/squash/generate_baseline.exs encodes the hand-applied
"DECLARED POST-GENERATION CORRECTIONS" that live in the generated files.

It does not. The phrase appears in expected_schema.ex and nowhere in the
generator. So the next regeneration — which release_check's own failure
message instructs an operator to perform — silently drops the
users_tokens.user_uuid nullability correction and both V164 index entries, and
chain_hash will still match after restamp. A baseline that loses a correction
while continuing to verify clean is the most expensive failure mode available
here, because it is inherited by every install created afterwards.

---

What it checked and found sound

Worth recording, so the findings above are read as a short list rather than a
verdict on the whole body of work.

The floor. Spot-checked against the documented old-chain end state with no
disagreement found: newsletters_broadcasts.list_uuid NOT NULL with its
RESTRICT FK and index; users_tokens.user_uuid nullable with the V64 CHECK
inline — the corrected shape, not the broken V56/V57 one; fk_comments_user_uuid
as SET NULL rather than V72's guessed CASCADE; preferred_locale correctly
absent and :legacy_optional in the manifest; both prefix-embedded naming
conventions reproduced; down/1 dropping no shared public functions and never
the schema.

Prefix safety. No violations in anything the squash rewrote — bare index
names on CREATE and qualified on DROP throughout V135/V163/V164/ShapeSql, every
existence probe anchored on table_schema/schemaname/nspname including the
referenced side, no regclass casts in any live immediate check, extensions
through the privilege-aware helper, schema creation through the guarded
ensure_schema!/2. The two remaining ::regclass uses are in UUIDFKColumns
mutation entry points that are dead code since V56/V57 were deleted — worth
deleting rather than leaving a live-data backfill loaded.

Data safety elsewhere. V164's NOT-NULL repair is zero-NULL-gated; FK
creation is NOT VALID + VALIDATE with orphans reported and never deleted; the
comments-FK drop/re-add window is NOT-VALID-covered; Repair.Executor is
additive-only by construction — no DROP or DELETE verbs, and its single UPDATE
targets IS NULL on a column the same call just added.

The happy upgrade path verified end to end: comment ≥ 135 → wrapper
vNNN→v164 with @disable_ddl_transaction → plan_up → {:run_delta, …} →
uuid-function re-ensure → V161..V164 → stamp.

Verdict: NEEDS-WORK — the squash routing, baseline fidelity and prefix
discipline hold up under close reading. What does not is the edge: a
comment-less current database is advised toward a replay that deletes live rows,
and the manifest cannot report clean on two supported topologies.