autobuild: relaunch after install is a race, and a failed relaunch is silent

[radroid]

Symptom

On 2026-08-02 the autobuild built and installed aaedc21ab, then left nothing running.
The app and :3773 (the Tailscale remote surface) were down for 103 minutes, until it was
opened by hand. Every status indicator stayed green throughout.

Evidence chain

Time	Event	Source
01:03:41	osascript quit sent; bundle replaced (mtime 01:03:41–42)	auto-build-desktop.sh:401, stat
01:03:42	log: install: relaunching; open runs; lsd re-registers the bundle — no process launched	unified log
01:03:44	last trace record written by the old app	desktop.trace.ndjson
01:03:45.165	launchd: application.com.t3tools.t3code…[73031] exited due to exit(0), ran for 216120549ms	unified log
01:03:45 → 02:46:40	zero t3code launch activity, zero trace records	unified log + trace
02:46:40	launch originated by com.apple.coreservices.uiagent — opened by hand	unified log

Status file said {"result":"built","installed":true,"detail":"ok"} the entire time.

Root cause

:401 fires an asynchronous AppleScript quit; :431 calls open. Between them sit only
cp -R, rm -rf, mv, xattr — nothing waits for the old process to exit. The old instance
still held Electron's single-instance lock (DesktopClerk.ts:90,128, whose second-instance
handler only reveals the existing window), so open activated the dying instance and
returned 0.

The cp -R was the accidental sleep that made this work at all. Copy windows: 10s (07-27),
12s, 12s, then 4s (07-30) and 1s (08-02) as the artifact shrank 237MB → 150MB. The race is
now a coin flip; 07-30 won it, 08-02 lost it.

Two further causes of the same symptom, independent of the race

• The bundle is rm -rf'd under six live processes (main, server child,
  t3-resource-monitor, three helpers). :3773 is owned by the server child, not the main
  app, and DesktopBackendManager has a restart loop — a child killed mid-swap respawns from a
  path that is missing or is the new build under the old Electron main.
• Port squat → silent green. If that child outlives the quit, the new app walks up to
  :3774 while tailscale serve is pinned to 3773. Identical outage, app fully alive.
• xattr -dr com.apple.quarantine … || true (:429) swallows failure — Gatekeeper refuses the
  launch, open still exits 0.

Fixes, ranked

All in scripts/t3x/** and docs/t3x/** — fork-owned, zero upstream churn, no rebase cost.

1. Reorder install_dmg. cp -R first; capture the quit's output instead of discarding it;
   wait_for_exit polling ps -Ao pid=,comm= (pid=,command= matches its own pipeline and
   returns 9 pids instead of 6 — it would time out on every install); re-quit → SIGTERM; on
   timeout abandon the staged copy and return 1 without touching the installed app. Only
   then rm/mv/xattr/open. Fixes the race, the replace-under-live-process hazard, the
   child respawn and the port squat in one change. Reject open -n — the single-instance
   lock makes a second instance quit immediately after revealing the dying one.
2. Verify the relaunch and make it reportable. Require a new pid; poll
   curl -fsS http://127.0.0.1:3773/.well-known/t3/environment (the app's own readiness
   contract); assert the :3773 listener is in the bundle's pid set. Add relaunched,
   appPid, backendPort and a distinct installed-not-running result to the status JSON.
   Nothing consumes that file today, so this is purely additive.
3. Post-install launch watchdog — see spec below.

Watchdog spec (as decided)

A separate script, spawned by the autobuild once the install completes, detached so it
survives the parent:

• Wait a grace period, then poll every 5 minutes: if the app is not running, open it.
• Exit as soon as the app is confirmed up. Exit after a bounded number of attempts so it can
  never linger.
• Skip while the autobuild lock is held, so it cannot fight a concurrent install.
• Readiness = the :3773 probe and the listener pid belonging to the bundle, not just
  pgrep — otherwise a port squat reads as healthy.

Grace period: requested as 10 minutes. The app starts in ~5 seconds, so 10 minutes is up to
10 minutes of dead :3773 in the common case. Suggest a named constant defaulting to ~90s,
with 10 minutes a one-line change.

Known gap: post-install-only means it does not catch the app dying for any other reason
(crash, manual quit, OOM). A standing LaunchAgent would; this is the simpler shape and covers
the failure actually observed. No deferral for running Claude sessions — installs proceed and
the app is reopened afterwards.

Two traps to fix at the same time

• --print-launchd would silently undo the cargo fix. The generator hardcodes
  PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin (:625). grep -c cargo and
  grep -c KeepAlive on the script both return 0. The live plist carries the fnm node path
  and ~/.cargo/bin — the 07-30 spawn cargo ENOENT fix was hand-applied and never
  back-ported. Regenerating today reinstates a 12-hour silent build outage. Derive PATH from
  resolved node/pnpm/cargo/git, refuse to emit if any is missing, add --diff-launchd.
• The runbook will install a 2-minute loop. auto-build-runbook.md:127 documents the
  interval default as 60, and :200 — the --print-launchd example itself — uses
  --interval 120. The script is INTERVAL=43200 (:76). Following the docs to rebuild the
  agent produces a two-minute quit/replace/relaunch cycle on an app that hosts live Claude
  sessions.

Smaller items

KeepAlive + ThrottleInterval on the agent (if the watcher dies nothing restarts it and every
indicator stays green); dmg freshness check (newest_dmg is an mtime pick, so a stale build can
install fully green); move T3CODE_DESKTOP_OUTPUT_DIR off the main checkout (2.4GB currently
written into the tree the sync branch lives in); atomic write_status; -readonly on the
install-path hdiutil attach (:391 lacks it, the peek path :338 has it); INT TERM detach
trap; reject --relaunch without --install; self-heal when the app is missing (the marker
is a built-sha, not an installed-state, so deleting the bundle yields "no change since last
build" forever).

[radroid]

Priority triage — 2026-08-07: Rank 3 / 11 — Tier 1 (active reliability bug)

Already caused a real ~103-min outage — this is reliability of infra that is running right now, not a hypothetical. Batches naturally with the other tooling fixes after the #49 sync lands.

[radroid]

Triage 2026-08-07 — mostly superseded by the update-delivery work; two fixes still worth doing now

The update-delivery design (docs/superpowers/specs/2026-08-03-update-delivery-design.md, PR #51) names this issue in its Closes: line, and its rollout section says scripts/t3x/auto-build-desktop.sh is retired once delivery lands — "closing #41 at the source." The implementation exists on t3x/update-delivery-impl (13 commits, 51 files, ~5,600 insertions, pushed, no PR yet).

That changes what is worth fixing here. The ranked fixes in this issue — reorder install_dmg, wait_for_exit polling, relaunch verification, the post-install watchdog — all harden a script the fork intends to stop running. Recommend not building the watchdog.

The design does inherit the two real lessons: it keeps the copy→checksum→swap ordering documented here, and it wraps the shutdown wait in Effect.timeout(10s) explicitly because "the unbounded shutdown wait reintroduces #41."

The exception: two traps that bite before #51 lands

The autobuild is still the live delivery path today, and these two are docs/script-only, cost nothing in seam surface, and are not superseded:

1. --print-launchd silently reverts the cargo-PATH fix. The generator hardcodes PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin (:625); grep -c cargo on the script returns 0, while the live plist carries the fnm node path and ~/.cargo/bin. Regenerating the agent today reinstates a 12-hour silent build outage. This is also still uncommitted — scripts/t3x/auto-build-desktop.sh shows as modified in the working tree.
2. The runbook documents a 2-minute interval. auto-build-runbook.md:127 says the default is 60 and :200 uses --interval 120, against the script's INTERVAL=43200. Following the docs produces a two-minute quit/replace/relaunch cycle on an app hosting live Claude sessions.

Suggest closing the rest of this issue as superseded when #51 lands, and pulling those two into a small standalone fix now.