diff --git a/docs/superpowers/plans/2026-08-21-clean-room-vm.md b/docs/superpowers/plans/2026-08-21-clean-room-vm.md new file mode 100644 index 00000000..268a0182 --- /dev/null +++ b/docs/superpowers/plans/2026-08-21-clean-room-vm.md @@ -0,0 +1,2290 @@ +# L7 Clean-Room Testing (VM golden images + scripted walkthrough + second-user smoke) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship `rt-tray/vm/` — golden macOS VM images (14 + 26), a scripted first-install → five-screen setup → `rt verify` → Sparkle vN→vN+1 walkthrough that archives screenshots + logs per run, a second-macOS-user smoke script, and `scripts/e2e-cleanroom.sh` that runs the CI headless install recipe locally — so ruling 12's layers (a), (b), (c) exist and the release gate ((a)+(b) green) is runnable. + +**Architecture:** Everything is bash + one tiny Bun helper, orchestrated from the host. Tart (Apple Virtualization.framework) owns VM lifecycle; the golden VM is never run after it is built — every run is `tart clone` (APFS copy-on-write) → drive → `tart delete`. Host ↔ guest is ssh (key installed at golden time) plus a `--dir` virtiofs share that IS the run's artifacts directory, so every screenshot and log lands on the host as it is produced. The UI is driven by AppleScript/System Events UI scripting over ssh (Xcode-free fallback, first) with an XCUITest mode (`xcodebuild test` inside an `-xcode` golden) gated on Xcode existing. Screenshots are captured on the HOST from the Tart window (in-guest `screencapture` over ssh is unreliable on macOS 15+). Every phase writes a `pass|fail|skip` line with a reason; nothing is marked green on a guess. + +**Tech Stack:** Tart ≥ 2.x (`brew install openai/tools/tart`; FSL-1.1-ALv2 — see constraints), cirruslabs `macos-{sonoma,tahoe}-vanilla` OCI images, bash 3.2 (`/bin/bash` — macOS stock), `sshpass` (golden build only), `osascript` / System Events, `screencapture`, `hdiutil`, `ditto`, `PlistBuddy`, Bun (compiles `appcast-server`), Sparkle `generate_appcast` (update phase only), `gh` (test team, host only). + +**Spec:** `docs/superpowers/specs/2026-08-20-mattstack-app-installer-design.md` (§2 ruling 12 + invariants, §3–4 flow/screens, §9 permissions, §11 distribution, §12.2 testing, §13 lanes); contract `docs/superpowers/specs/2026-08-21-rt-setup-contract.md`; research `docs/superpowers/specs/research/2026-08-20-mattstack-app/research-onboarding-permissions-ux.md`, `research-sparkle-install-launchd.md`. + +**Execution worktree:** `/Users/matt/Documents/GitHub/repo-tools-l7-wt`, branch `goodwinmattheweric/mat-383-clean-room` off `origin/main`. Commit prefix `MAT-383:`; trailer `Co-Authored-By: Claude Fable 5 `. No monitor agent — every implementer runs `bash -n` and the task's dry-run themselves before committing. + +**Execution order** (binding; from `docs/superpowers/plans/2026-08-21-cross-plan-review.md` §3): Phase A — Tasks 1, 2, 4, 5, 7, 9, 10, 11, 14 (all under `rt-tray/vm/`) and Task 3 with its own recipes (re-pointed at `scripts/release/*` after L4 T6 merges). Phase B — Task 6 AFTER L3 T12–T16 merge (the AXIDs exist; ids are already written to L3's `AccessibilityIDs.swift` below and fail fast when missing); Task 8 AFTER L3 T10 merges (`--allow-appcast-override`); Task 12 after the CLI is agreed with L4 T8 (positional artifact — fixed in the review); Task 13 AFTER L3 T19 merges and Xcode exists. Merge order to main: L4 phase A → L3 T1–T11 → L1 → L4 T4/T5/T8 → L1 (rest) → L3 T12–T19 → **L7 (all)** → L4 T12 → L1 T31–T32 → MATT gates. L7 owns `rt-tray/vm/**` and `scripts/e2e-cleanroom.sh` (interface fixed by the review, §5 #46); `.github/workflows/release.yml` is L4's and only calls the script. + +## Global Constraints + +Copied from the spec; every task's requirements include these. + +- **macOS 14 floor; arm64-only** (ruling 13, §11). Golden images: macOS 14 (Sonoma) and 26 (Tahoe); 15 (Sequoia) optional. Host is Apple Silicon, macOS 26.6.1. +- **Identity frozen:** `com.mattstack.app` / `com.mattstack.app.dev`; agents `com.mattstack.daemon` / `.dev`, `com.mattstack.deck`; bundle `mattstack.app`; `~/.mattstack`; `~/.local/bin/rt` is a symlink into the bundle (ruling 13, §11). Tests assert these names, never `rt-tray.app` / `com.rt.daemon` / `~/.rt` ("pure canonical, no compat"). +- **Install location:** DMG target `/Applications`, fallback `~/Applications` when not writable; the app records `mattstack.appPath` (V3). +- **Permissions:** Full Disk Access + Login Items required; Notifications optional; no Accessibility/Screen Recording/Automation are requested by the APP (ruling 10). (The VM test *driver* needs Accessibility — that is granted to `sshd-keygen-wrapper`/`osascript` in the guest, never to mattstack.app.) +- **Honesty over magic:** every checklist row / phase reports what was actually checked; nothing is marked ready on a guess (invariant). Phases that cannot run say `skip` + reason, never `pass`. +- **rt owns mechanics, the app owns ceremony:** the walkthrough asserts through `rt verify --json`, `rt setup status --json`, and tray.sock routes (`GET /version`, `GET /services`, `GET /permissions`) — never by parsing UI text alone. +- **No user or employer data on mattstack-hosted infra; the relay sees ciphertext + timestamps only.** The test team uses a throwaway GitHub org; the PAT comes from env and is never written to the repo or to artifacts. +- **rt repo stealth:** rt never writes into target repos; all state under `~/.mattstack/`. The VM layer writes only under `rt-tray/vm/artifacts/` (gitignored) and `rt-tray/vm/.cache/` (gitignored) on the host. +- **Clean-room definition (ruling 12):** no Apple CLT, no Homebrew, one standard user, no Apple ID. +- **Release gate:** layers (a) GitHub Actions headless install + `rt verify --ci` and (b) VM walkthrough green on the candidate. +- **Clean-code comments rule** (`~/.claude/rules/clean-code-comments.md`): comments only for constraints the code cannot show; no task numbers/process narration in source. Decision records go in this plan / README, not scripts. +- **Logging:** scripts log to the run's artifacts dir; no new logging inside rt. +- **Rebase onto `origin/main` before every merge** (cross-plan review §3); L7 merges after L3 T12–T19. L7 edits only `rt-tray/vm/**`, `scripts/e2e-cleanroom.sh` and an appended section of `scripts/README.md` (L4 T6 appends its own section; whoever merges second rebases). +- **Artifacts are L4's names:** `mattstack-.dmg`, `mattstack-.zip`, `appcast.xml` (a GitHub Release asset at `https://github.com/m4ttstack/rt/releases/latest/download/appcast.xml` in prod; served on loopback in the VM), `*.delta`, `SHA256SUMS`. There is no `rt-darwin-arm64-*.tar.gz` any more — the zip is the headless artifact. `CFBundleVersion` is numeric `major*1000000+minor*1000+patch` (2.8.0 → `2008000`). +- **Tart licence constraint:** Tart's `LICENSE` on `main` is **Functional Source License 1.1 (ALv2 future licence), "Copyright 2022-2026 OpenAI"** — permitted purposes include "internal use and access"; tart.run/licensing: "Usage on personal computers including personal workstations is royalty-free"; organisations are free up to **100 CPU cores** for Tart, paid tiers above (Gold $12k/yr 500 cores…). Matt's single workstation is inside the free tier. Record in README; if L7 ever moves to a shared self-hosted fleet, recount cores. + +## Research record (what the scripts rely on — cited) + +- Tart now lives at `github.com/openai/tart`; brew tap `openai/tools/tart` (README on main). Older docs say `cirruslabs/cli/tart`; use the new tap, fall back to the old if brew can't find it. +- Images (tart.run quick-start): `ghcr.io/cirruslabs/macos-{tahoe,sequoia,sonoma}-{vanilla,base,xcode}:latest`; creds `admin`/`admin` for GUI and ssh. `macos-sonoma-vanilla:latest` = 14.8.7 (published ~3 months ago, still updated). **Use `-vanilla`**, not `-base`: `base` preinstalls Homebrew + gh/jq/node/rbenv + the tart guest agent (templates `base.pkr.hcl`); `vanilla` has none of that. The `vanilla-sequoia` template DOES run `softwareupdate --install 'Command Line Tools for Xcode-*'`, so the golden provisioner must remove CLT (`/Library/Developer/CommandLineTools`) on every image and verify with `xcode-select -p` failing. +- Vanilla templates preconfigure: auto-login `admin` (`/etc/kcpassword` + `autoLoginUser`), passwordless sudo (`/etc/sudoers.d/admin-nopasswd`), screensaver off, `systemsetup -setsleep Off`, `sysadminctl -screenLock off`, Remote Login + Screen Sharing ON, `defaults write NSGlobalDomain AppleKeyboardUIMode -int 3`, **Gatekeeper DISABLED** (`sudo spctl --global-disable` + Settings toggle). The golden must re-enable Gatekeeper (`sudo spctl --global-enable`, verify `spctl --status` = "assessments enabled") so the DMG path is real. +- `tart clone` = file copy on APFS → copy-on-write ("a cloned VM won't actually claim all the space right away"), so restore-per-run = clone from golden, delete after. `--stacked` needs macOS 27; not used. +- `tart run` flags used: `--no-graphics`, `--dir=:[:ro]` (guest automount `/Volumes/My Shared Files/`; host+guest ≥ 13), `--no-audio`, `--vnc` (Screen Sharing), `--net-softnet` (NOT used: blocks guest→host). Default NAT: host reachable from guest at the router IP (`route -n get default` in the guest). +- `tart exec` needs the tart-guest-agent, which ships only in non-vanilla images → **ssh only** in this plan. `tart ip ` gives the guest IP. Commands available: clone create delete exec export get import ip list login logout prune pull push rename run set stop suspend — **no screenshot command**. +- Nested virtualisation: Virtualization.framework supports it only for Linux guests on M3+/macOS 15 → GitHub-hosted macOS runners (themselves VMs) cannot run Tart macOS VMs. Local only; a self-hosted Apple-Silicon runner later. +- UI scripting over ssh: `osascript … tell application "System Events"` fails with "not allowed assistive access" until `/usr/libexec/sshd-keygen-wrapper` (and `/usr/bin/osascript`) are in Privacy & Security → **Accessibility**, and the **Automation** ("sshd-keygen-wrapper wants to control System Events") prompt is approved once for the ssh user. TCC cannot be pre-approved from a script (`tccutil` can only *reset*; PPPC profiles need MDM) → **one-time manual click in the guest GUI at golden-image time**, then every clone inherits it. macOS 26.1/26.2 had a bug adding CLI tools to Accessibility (fixed 26.3); the golden verifier probes it. +- `screencapture` over ssh on macOS 15+ prompts/returns blank even with Screen Recording granted to sshd-keygen-wrapper (Apple forum thread 764789) → **capture on the host** from the Tart window (`screencapture -x -l `; host Terminal needs Screen Recording once — MATT step). +- `launchctl bootstrap gui/` needs a logged-in GUI session ("Bootstrap failed: 5" otherwise) → second-user smoke requires the second user to be logged in (Fast User Switching) before `rt daemon install` can register; `su -l` alone can install files but the daemon row reports not-booted. +- Sparkle: feed override via `SPUUpdaterDelegate feedURLString(for:)`; http feeds need `NSAppTransportSecurity` → `NSAllowsLocalNetworking` (loopback/.local) — serving from the HOST at 192.168.64.x would additionally trigger Sequoia's Local Network Privacy prompt (Sparkle discussion #2732), so the appcast is served **inside the guest on 127.0.0.1**. Enclosures must be EdDSA-signed with the key matching the build's `SUPublicEDKey`; Sparkle compares numeric `CFBundleVersion`. +- `rt --post-install` (current `commands/post-install.ts`) installs `~/.local/bin/rt`, copies `mattstack.app` → `~/Applications`, **launches the app with `open`**, then `rt daemon install` (→ tray socket `/daemon/start`, SMAppService). Running it on a host whose console user already has mattstack registered would launch a second tray under the same bundle id → `scripts/e2e-cleanroom.sh` refuses that by default. (L1 T27 replaces the body with legacy sweep + headless `rt setup apply`; `--no-launch` is explicit and implied by `--ci`/`CI=true` — Task 12 passes it.) +- `rt verify --ci` (commands/verify.ts) = human output, no colour, exit 1 on any critical fail; daemon-not-booted is a *warn* under `CI=true`. +- VirtualBuddy (insidegui): GUI only, no CLI; APFS duplicate via Finder ⌘D; guest app for clipboard/sharing. Documented as the manual alternative, not scripted. + +## Dependencies on other lanes (the walkthrough is built to the contract; phases `skip` with the reason until these land) + +| Need | Lane | Walkthrough phase that skips without it | +|---|---|---| +| DMG + zip + appcast artifacts (`mattstack-.dmg`, `mattstack-.zip`), Sparkle in the app, `SUPublicEDKey`; build accepts `SPARKLE_PUBLIC_ED_KEY` override for test keys | L4 | update (P7); install uses `vm/run/make-dmg.sh` from a locally built `.app` meanwhile | +| Five screens + AXIdentifiers from the L3 plan's `rt-tray/Sources/Setup/AccessibilityIDs.swift` (`2026-08-21-mattstack-app-shell.md` T12; the same plan defines the XCUITest target `mattstackUITests` and the stub envs `RT_STUB_SCENARIO`/`RT_STUB_PATH` (required)/`RT_STUB_BUN`/`RT_STUB_STATE_DIR`): screens `setup..screen`; `setup..continue`, `setup..back`, `setup.checklist.continueLimited`; team `setup.team.card.create\|join\|restore`, `setup.team.create.name\|others\|useGh\|owner\|remote`, `setup.team.join.code`, `setup.team.restore.repo\|key`; checklist `setup.checklist.row.`, `.row..action`, `.row..status`, `setup.checklist.recheck`, `setup.checklist.relaunch`, `setup.checklist.connect.field.`, `setup.checklist.connect.submit\|cancel`; install `setup.install.step.`, `setup.install.retry`; done `setup.done.continue`, `setup.done.openBoard`, `setup.done.invite`. Row ids are L1's (`perm.fda`, `perm.login-items`, `perm.notifications`, `account.github`, `tool.clt`, …). Also from L3: `MATTSTACK_APPCAST_URL` honoured only for dev builds or when launched with `--allow-appcast-override` (T10), ATS `NSAllowsLocalNetworking`, `GET /version` (`build` numeric) on tray.sock | L3 | screens (P5), update (P7) | +| `rt setup plan/status/apply --json`, `rt --post-install --non-interactive --team-of-one --no-launch` (`--no-launch` also implied by `--ci`/`CI=true`), `rt team invite/join` | L1 | assert (P6 uses `rt verify --json` today), invite scenario | +| `/v1/invites*` on the shared relay | L6 | join scenario (stub until then) | + +--- + +## File structure + +``` +rt-tray/vm/ + README.md what the layer verifies / cannot; how to run; MATT steps; licence note + .gitignore artifacts/ .cache/ + lib/common.sh shared bash: logging, phase ledger, tart/ssh helpers, run dirs + golden/build-golden.sh host: pull vanilla image → mattstack-golden- → provision → verify + golden/provision-guest.sh guest (as admin): no CLT, no brew, Gatekeeper on, tester user, ssh keys, sleep/lock off + golden/verify-golden.sh host: asserts every golden property over ssh incl. the manual TCC grants + run/walkthrough.sh host orchestrator: clone → boot → stage → install → screens → assert → update → teardown → report + run/guest/install-app.sh guest: quarantine DMG, mount, copy to /Applications (admin) , launch as tester with env + run/guest/ax.sh guest: osascript helpers (find by AXIdentifier, click, type, wait, admin-auth, notification prompts) + run/guest/drive-setup.sh guest: screens 1–5 + FDA/Login Items/Notifications dances, screenshot hooks + run/guest/assert-installed.sh guest: rt verify --json, tray.sock /version, daemon pid/label, symlink target + run/guest/trigger-update.sh guest: loopback appcast server, POST /update/check, drive Sparkle, assert vN+1 + daemon restart + run/host/capture.sh host: screenshot the Tart window into the run dir + run/host/winid.swift host: CGWindowList lookup of the Tart window id + run/helpers/appcast-server.ts Bun static file server (compiled per run, copied into the guest) + run/make-dmg.sh host: wrap a built mattstack.app in a DMG (pre-L4 runs) + run/make-appcast.sh host: bump CFBundleVersion copy → zip → generate_appcast with a test EdDSA key + run/team-setup.sh host, ORCHESTRATOR/MATT: reset the throwaway org's repos; mint invite (real or stub) + run/second-user.sh host: layer (c) — run e2e-cleanroom as the second macOS user (MATT creates the user) + run/xcuitest.sh host: layer (b) XCUITest mode, gated on Xcode + the -xcode golden + artifacts/ gitignored; / per run: screenshots/, logs/, phases.jsonl, report.md +scripts/e2e-cleanroom.sh layer (a) locally: artifact → extract → rt --post-install → rt daemon install → rt verify --ci +``` + +Phase ledger format (`artifacts//phases.jsonl`, one object per phase, appended by `vm_phase_end`): +`{"phase":"install","status":"pass|fail|skip","reason":"…","at":"","seconds":12,"screenshots":["screenshots/03-install.png"]}`. +`report.md` is rendered from it at teardown. A run's exit code is 1 if any phase failed, 0 if all passed or skipped, and the summary line states the skips so a skip never reads as green. + +--- + +### Task 1: Scaffold `rt-tray/vm/` — README skeleton, gitignore, `lib/common.sh` + +**Files:** +- Create: `rt-tray/vm/README.md` +- Create: `rt-tray/vm/.gitignore` +- Create: `rt-tray/vm/lib/common.sh` +- Create: `rt-tray/vm/lib/__tests__/common.test.sh` + +**Interfaces:** +- Produces (sourced by every later script via `source "$(dirname "$0")/../lib/common.sh"` or `…/lib/common.sh`): + - `VM_ROOT` (abs path of `rt-tray/vm`), `VM_ARTIFACTS` (`$VM_ROOT/artifacts`), `VM_CACHE` (`$VM_ROOT/.cache`) + - `vm_log `, `vm_warn `, `vm_die ` (exit 1) + - `vm_require_cmd []` + - `vm_golden_name ` → `mattstack-golden-`; `vm_image_for ` → OCI ref (14→sonoma-vanilla, 15→sequoia-vanilla, 26→tahoe-vanilla) + - `vm_run_init