Normalize the hawkeye/rybbit deploy: configure prerequisites and get the first real deployment running

[ineedjet]

Problem

#102 built the tooling (release automation, vaults//targets/ manifests,
ansible/deploy.yml) to deploy Rybbit for rubykatzen.com on hawkeye
through flightdeck's own pipeline, but it never got to a live deploy — the
PR's own Verification checklist left these unchecked:

• Publish the first real Hawkeye env asset after external prerequisites
  are configured
• Run the first real Hawkeye deployment

Nothing has run against production yet, and the prerequisites listed in
#102's PR description haven't been confirmed as actually configured.

Prerequisites to configure (from #102's PR description)

• GitHub Secret DEPLOY_SSH_PRIVATE_KEY
• GitHub Secret HAWKEYE_AGE_PRIVATE_KEY — new prerequisite, added
  when decryption and config rendering moved from hawkeye itself to the
  CI runner (Consider decrypting vaults on the CI runner instead of server-side #116, done alongside Consider migrating ansible/deploy.yml to Fabric (Python) #111/One vault per app: drop the APPS_/{app}_ prefix convention for vault-sourced env #114). Content is the private
  key matching keys/hawkeye.pub (previously only ever needed
  server-side, in ~/.config/sops/age/keys.txt; now needed here
  instead, not there).
• GitHub Variable TAILSCALE_OAUTH_CLIENT_ID and Secret
  TAILSCALE_OAUTH_SECRET (if Tailscale is used to reach hawkeye)
• Every GitHub Secret/Variable referenced by vaults/hawkeye-traefik.yml's
  and vaults/hawkeye-rybbit.yml's env: mappings: RUBYKATZEN_COM_DOMAIN,
  RUBYKATZEN_COM_ADMIN_MAIL, RUBYKATZEN_COM_CERT_RESOLVER,
  RUBYKATZEN_COM_CLOUDFLARE_TOKEN, RUBYKATZEN_COM_DATABASE_PASSWORD,
  RUBYKATZEN_COM_KEY_HEX_32, RUBYKATZEN_COM_TRAEFIK_HTTP_PORT,
  RUBYKATZEN_COM_TRAEFIK_HTTPS_PORT
• Server-side rubykatzen-com user on hawkeye, with SSH access matching
  DEPLOY_SSH_PRIVATE_KEY. No longer needed on the server: the age
  private key, sops, gh — all moved to the CI runner side (Consider decrypting vaults on the CI runner instead of server-side #116).
  Docker and Docker Compose are the only remaining host dependencies.
• External Cloudflare Tunnel configuration for rubykatzen.com
• Separate open question, not blocking: apps/traefik/docker-compose.yml
  carries the Watchtower label, but hawkeye's apps doesn't include a
  watchtower app — as implemented, the automated deploy path would
  never actually run docker compose up for traefik. Needs its own
  decision (add watchtower to hawkeye's apps, or drop the label)
  before the first real deploy.

Once prerequisites are in place

• Publish the first real hawkeye.sops.env asset (via a real release,
  or by manually running the encrypt-vaults/encrypt jobs)
• Run the first real deployment to hawkeye (deploy.yml
workflow_dispatch, or let it ride the next real release)
• Confirm traefik + rybbit actually come up and rubykatzen.com
  resolves correctly end to end

Separately: triage currently failing/blocked workflow runs on main

Found while checking the post-merge state of #102.

• Release — action_required, zero jobs ran, on every push to main
  through 1f6117d. Root cause confirmed: repo Settings → Actions →
  General → Workflow permissions was set to "Read repository contents
  and packages permissions" — a hard cap that release.yml's own
  explicit permissions: contents: write / issues: write / pull-requests: write block could not exceed (proven by the fact it
  stayed blocked despite already declaring those permissions itself).
  Fixed by switching to "Read and write permissions" + "Allow GitHub
  Actions to create and approve pull requests" (needed separately for
  release-please's own release PR creation). Confirmed via API
  (default_workflow_permissions: "write",
  can_approve_pull_request_reviews: true). Couldn't force a live
  re-run (gh run rerun refuses runs that never started any jobs, and
  release.yml has no workflow_dispatch) — will get a real
  confirmation on the next push to main / next release-please cycle.
• Notify Telegram PR — startup_failure on every single run
  (push, PR, and daily schedule) back through history. Same root
  cause: the GitHub error banner named it exactly — nested job
  notify requesting pull-requests: read while the repo only
  allowed pull-requests: none. Fixed by the same Settings change.
  Confirmed live via manual workflow_dispatch after the fix —
  run succeeded.
• Dependabot update-check runs ("pip in /.", "bundler in /.") on
  58d431e — both failure. Not yet investigated; may or may not be
  the same permissions root cause (worth a quick check next).

Related

Follow-up from #102.

[ineedjet]

Real test today: merged the release-please PR, v0.4.0 published successfully (release, upload-apps, deploy-targets all green). encrypt failed exactly as expected, on the first missing secret from the prerequisites checklist above:

```
error: missing GitHub Secrets/Variables: RUBYKATZEN_COM_CLOUDFLARE_TOKEN
```

`deploy` was skipped (depends on `encrypt` succeeding) — never got far enough to attempt a real connection to hawkeye. Confirms the prerequisites list above is accurate; `RUBYKATZEN_COM_CLOUDFLARE_TOKEN` is the first one actually hit. Once the GitHub Secrets/Variables above are set, re-run encrypt/deploy manually via deploy.yml's workflow_dispatch rather than waiting for the next release.

[ineedjet]

Closing — the core ask ("get the first real deployment running") is done, even though the specifics here are stale (hawkeye was fully retired and replaced by heimdall in a later session, along with the RUBYKATZEN_COM_* prefix convention, KEY_HEX_32, and the watchtower label mentioned above - all superseded by later refactors).

Current state: traefik + rybbit + cloudflared are live on heimdall (root@100.75.53.13), all containers healthy, all prerequisites configured (SSH key, SOPS age key, Tailscale OAuth, every vault-referenced secret/variable). First real deploy succeeded via the standard release pipeline.