Skip to content

ops(gate): the upgrade leg + role boundary (STORY-446, gh-#777) - #790

Merged
genwave-radio merged 4 commits into
mainfrom
ops/upgrade-leg-446
Sep 16, 2026
Merged

genwave-radio merged 4 commits into
mainfrom
ops/upgrade-leg-446

Conversation

@genwave-radio

Copy link
Copy Markdown
Collaborator

PR-6 of the gh-#777 epic (STORY-446, SPEC F178.6, F178.7). The gate now proves an upgrade: the previous release installed for real, then migrated forward onto the tag under test, with the role boundary re-checked afterwards.

🏗️ What

  • The upgrade leg (T493, --upgrade): resolves the previous release (--from <vX.Y.Z>, else the newest other published v* release from gh release list, drafts and pre-releases excluded), checks it out with git worktree add into the run's scratch, and runs that release's own setup.sh --yes there with a generated overlay pinning its images. Once it is on the air the leg does what an operator's upgrade does: docker compose stop api, swap db/ and migrate.sh for the current checkout's, run the current ./migrate.sh (must exit 0, else the leg fails migrate with the tail in the report), docker compose up -d api, then health and first on-air again. The worktree and the compose project are removed in the trap on every path.
  • The role boundary (T494): after the migrate and before the api restart, psql -U station_svc on library.media and psql -U library_svc on station.settings through compose exec -T db. Both must print permission denied; anything else (a row count, an auth error, empty output) fails the leg role boundary with the probe's text in the report. The decision is on the text, not the exit code: real psql exits 1 on the outcome we want.
  • Report: upgrade previous / health_secs / onair_secs and a role boundary | ok row in the md, twins in the json (role_boundary null when never reached).
  • Story446_UpgradeLeg.cs: all 15 facts green. The migrate-order facts locate the migrate.sh call line rather than a substring, so a directory listing in the stub log cannot match.
  • gh is a prerequisite only when --from is absent.

🔌 T495 — v5.8.3 → current on the dev box

The current checkout's five images were built locally and tagged home-v9.9.9 (the gate requires a vX.Y.Z tag, so the plan's vTEST is spelled v9.9.9).

tools/gate/stack_gate.sh --tag v9.9.9 --from v5.8.3 --upgrade --report <dir>
# exit 0, wall 99 s; docker compose ls | grep gw-gate → empty; git worktree list → only the checkout
leg result detail
fresh skipped (--fresh not given) -
upgrade ran/passed -
capture skipped (--capture not given) -
chaos skipped (--chaos not given) -
measurement value
upgrade previous v5.8.3
upgrade health_secs 5
upgrade onair_secs 5
role boundary ok

A second run under bash -x (exit 0, 89 s) showed the real sequence on the stack: stop api → current migrate.sh copied into the worktree and run ("Schema migrations up to date") → psql -U station_svc … library.media printed ERROR: permission denied for schema librarypsql -U library_svc … station.settings printed ERROR: permission denied for schema stationup -d apidown -vgit worktree remove --force.

🔍 Review rounds worth knowing

  • T493 round 1: the builder deleted the previous release's migrate.sh from the worktree so a stub directory listing would not match the spec's substring search, with a comment claiming the previous stack boots from an empty volume. Both false to production; fixed by matching the call line in the spec and removing the deletion.
  • T494 passed with notes: the failing probe's text now lands in the report (role boundary error:), like migrate error:.

📝 For later tasks

  • T502 (CI): actions/checkout needs fetch-depth: 0 or fetch-tags: true, or git worktree add … vX.Y.Z has no tag to check out.
  • The gate prints no progress lines of its own between the previous release's setup output and the report. Fine locally; CI logs may want a line per step.
  • up -d api also recreates engine on the current tag (api depends on engine), which matches a real upgrade and is inside the second on-air timing.

✅ Gate

Full solution, Category!=Integration: 9 projects, 0 failed (Host 2991 passed / 127 skipped, Architecture 150 / 10). bash -n and shellcheck clean.

…sequence (gh-#777)

Fills in run_upgrade_leg in tools/gate/stack_gate.sh: resolve the previous published release (--from or the newest other v* release via gh), git worktree that tag, its own setup.sh --yes + up -d on the previous pin, wait health/on-air, compose stop api, swap db/ + migrate.sh to the current checkout, run the current ./migrate.sh, up -d api, wait health/on-air again — worktree removed on EXIT via the existing trap. SPEC F178.6, STORY-446.

Also: the spec now locates the migrate.sh CALL line (StartsWith, not a substring search) so it can't match a docker stub's cwd directory-listing line that happens to mention migrate.sh.
After the current migrate.sh and before the api restart, psql as station_svc on library.media and as library_svc on station.settings via compose exec -T db; both must print permission denied (text, not exit code — real psql exits 1 on that outcome), anything else fails the leg 'role boundary'; report gets a role boundary row + json twin. SPEC F178.7, STORY-446 AC6-AC7.
@genwave-radio
genwave-radio merged commit cb1693a into main Sep 16, 2026
11 checks passed
@genwave-radio
genwave-radio deleted the ops/upgrade-leg-446 branch September 16, 2026 17:26
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant