Consider migrating ansible/deploy.yml to Fabric (Python)

[ineedjet]

Context

ansible/deploy.yml doesn't actually use Ansible idiomatically. It's a
strictly sequential procedure — download bundle, extract, pull+merge app
packages, pull+decrypt+merge env packages, symlink switch, run
./deploy.sh, prune old releases — not a converging desired-state
playbook. "Recreate release directory" even does state: absent then
recreates unconditionally on every run, the opposite of idempotent
convergence. This matches #107's framing exactly: flightdeck is a
Capistrano-style deployment system (versioned releases, current symlink,
retained history), not a Nomad/Kubernetes-style reconciler. Capistrano's
own model — SSH in, run a sequence of tasks, switch a symlink — is what
Fabric provides for Python, the same way Capistrano provides it for Ruby.

Concrete friction with Ansible specifically

• No shared-function mechanism across tasks. Each shell: task is an
  independent subprocess — there's no way to define a bash function in one
  task and reuse it in another without a checked-in script file. This is
  the direct cause of Deduplicate release-ref resolution logic in ansible/deploy.yml #103: four near-identical ~30-line "parse
  owner/repo@tag[:asset], resolve @latest via gh release view,
  download" blocks, hand-copied because Ansible has no better answer short
  of extracting a separate shell script. In Fabric this is just an
  importable Python function — the duplication wouldn't exist to begin
  with, and Deduplicate release-ref resolution logic in ansible/deploy.yml #103 would become moot rather than needing its own fix.
• Zero unit test coverage of the deploy logic itself. encrypt-env
  and load-yaml-matrix are both Python, both have real unit tests
  (unittest, PyYAML). ansible/deploy.yml is YAML + Jinja + embedded
  bash — the only verification it gets is --syntax-check and manually
  tracing a generated matrix through by hand (done for feat: deploy rybbit for rubykatzen.com through flightdeck itself #102/feat: move apps to targets, support env_refs as a list #110, but
  that's not the same as testing the ref-parsing/collision-detection logic
  itself). A Fabric-based deploy script would be ordinary Python, testable
  the same way the rest of this repo's automation already is.
• The inventory is ceremony, not a real feature. deploy-shared.yml
  builds a throwaway JSON inventory via jq on every run purely to
  satisfy Ansible's -i requirement — nothing here uses Ansible's actual
  host-grouping/inventory capabilities.
• Real, recurring cost for capabilities we don't use. ansible-core
  is installed fresh via unpinned pip install on every single deploy run
  (deploy-shared.yml's "Install ansible-core" step) — paying setup cost
  for idempotency/convergence/templating machinery the playbook doesn't
  actually lean on.

Refined direction: push-based deploy, decided during #114's design

Discussion while designing #114 (per-app vault delivery) converged on a
concrete shape for what a Fabric-based ansible/deploy.yml replacement
should actually look like — this is now the leading design, not just "port
the playbook to Python":

• All ref-resolution/download logic moves to the CI runner. Parsing
  owner/repo@tag[:asset], resolving @latest via the GitHub API,
  downloading the resolved asset — all of it runs as ordinary Python on
  the GitHub Actions runner, written once, reused for the machinery
  bundle, every app bundle, and every encrypted vault asset. The server
  never calls gh itself and doesn't need it installed — a real
  prerequisite reduction from what README currently documents.
• Fabric pushes files to the server instead of the server pulling
  them. Downloaded app bundles and still-encrypted vault blobs are
  transferred to the server directly (SSH/SCP), rather than the server
  reaching out to GitHub itself.
• Decryption stays strictly server-side. The server's private age key
  never leaves it, same as today. The server's only vault-related
  capability is running sops decrypt on a single file it's given — a
  generic, dumb, reusable primitive with no knowledge of refs, targets, or
  merging.
• No composition/merge logic lives on the server at all — but collision
  detection still happens, in CI, before anything is pushed. SOPS's
  dotenv output format only encrypts values; key names stay in
  cleartext (APPS_DOMAIN=ENC[...]). So when an app's env_refs has more
  than one entry, Fabric downloads each .sops.env asset and compares key
  names across them directly — no decryption needed for this check at
  all. Any duplicate key fails the build right there, before anything
  touches the server. Only once that check passes does Fabric push the
  files and tell the server to decrypt each one and plain cat them
  together into apps/{app}/.env. The server still never understands
  composition; it just never sees a colliding set in the first place.
• Update, superseding the below: this went further than "almost
  nothing" — the session that implemented this also decided flightdeck
  will have no manual administration flow at all, ever (no server SSH
  console access, no local quick-start either). With that decided,
  decryption and config-template rendering moved to the CI runner too
  (see Consider decrypting vaults on the CI runner instead of server-side #116), and literally nothing server-side survives:
  up.sh/down.sh/restart.sh/deploy.sh/generate-env.sh/lib.sh
  were all deleted outright, no replacements. The target host's only
  dependencies are Docker and Docker Compose. deploy/deploy.py issues
  docker compose pull/up directly per app over SSH; there is no
  server-side script layer of any kind left to describe.

Considerations against

• Ansible's SSH connection handling, privilege escalation (become), and
  retry semantics are battle-tested. Re-implementing that in Fabric means
  re-verifying connection and privilege-escalation behavior against real
  hosts, not just a drop-in rewrite.
• Ansible is more broadly known in ops contexts than Fabric — a real
  discoverability/onboarding tradeoff, not just a technical one.
• This touches the most safety-critical path in the repo (the actual
  production deploy mechanism) — needs a careful verification plan, not a
  quick swap, and shouldn't block anything currently in flight (Normalize the hawkeye/rybbit deploy: configure prerequisites and get the first real deployment running #106's
  first real hawkeye deployment).

Proposal

Implemented, going further than originally scoped here (see the "Update"
note above) — ansible/ is deleted, deploy/deploy.py is the entire
deploy mechanism, verified via real unittest coverage of every module
plus a matrix trace-through of the updated targets//vaults/ schema.
No live deploy to hawkeye yet (still gated on infra prep), same
verification-rigor bar as originally proposed here.

Related

#107 (Capistrano-style framing this migration leans into), #103 (the
concrete duplication problem this resolves at the root instead of
patching — becomes moot once ref-resolution is a Python function), #114
(per-app vault delivery is the concrete feature this push-based model was
designed to carry).

[ineedjet]

Merged via #115. ansible/ is deleted; deploy/deploy.py is the entire deploy mechanism now, with real unittest coverage across every module (resolve.py, collisions.py, vault.py, render.py, deploy.py — 50 tests). Went further than originally scoped here: decryption and config-template rendering also moved to the runner (#116), and the shell-script layer is gone entirely, not just ansible/deploy.yml.