Support encrypted-at-rest inventory secrets (ansible-vault / SOPS)

[thiras]

Goal

Give operators a supported, committed, encrypted-at-rest way to carry the one operator-supplied secret in this repo — decdn_rpc_url (may embed a provider API key) — instead of the current "keep it out of git" posture. Today that value lives in a git-ignored secret.yml, which keeps plaintext out of the repo but leaves the secret unencrypted, unversioned, and un-shareable on each operator's control machine.

This must not weaken the repo's hard rule (AGENTS.md §1): no plaintext secrets in tracked files, ever. Encryption is the mechanism that lets a secret be tracked while still honoring that rule.

Scope: this is about the operator-supplied secret path only (decdn_rpc_url). The host-generated secrets — eth keystore, node.secret, keystore password — are minted on the host at 0600 by roles/decdn_node/tasks/main.yml and intentionally never touch the control machine; they are out of scope and should stay that way.

Why there's work to do

The current model (see ansible/inventory/host_vars/decdn-node-1/):

• main.yml — committed, non-secret per-node config.
• secret.yml.example → operator copies to secret.yml, git-ignored via ansible/.gitignore:9 (inventory/host_vars/*/secret.*), holds decdn_rpc_url.

Ignoring the file solves "don't commit plaintext" but creates three gaps:

1. Unencrypted at rest — the secret sits in cleartext on every operator's disk / control machine.
2. Unversioned & unshared — it's outside git, so there's no history and no first-class way to hand it to a teammate or reproduce it on a second control machine; every operator reconstructs it out-of-band.
3. Doesn't scale to multi-node — each new host_vars/<node>/secret.yml is another untracked, hand-copied file.

There is no ansible-vault or SOPS usage anywhere in the repo today (no vault_id, no --vault-password-file wiring in the ansible/ Makefile, no .sops.yaml). Adding first-class encrypted-secret support closes the gaps above without loosening the plaintext rule.

Deliverables

1. Choose & wire an encryption mechanism (see Open questions — ansible-vault is the native default).
   • Encrypted secrets become committable (e.g. inventory/host_vars/<node>/secret.yml as a vaulted file, or a vaulted var inside main.yml).
   • ansible-playbook decrypts transparently at run time from an operator-held key/password never in the repo.
2. Update the ansible/ Makefile + inventory wiring so make check / make deploy pick up the vault password (e.g. ANSIBLE_VAULT_PASSWORD_FILE / --vault-password-file, or a SOPS-decrypt lookup) with a clear failure when the key is absent.
3. Refresh the secret.yml.example flow + docs (ansible/README.md, roles/decdn_node/README.md, inventory/host_vars/decdn-node-1/): show creating/editing an encrypted secret (ansible-vault create/edit) instead of copy-a-plaintext-file, and document key distribution.
4. .gitignore reconciliation — once secrets are encrypted-and-committed, the blanket inventory/host_vars/*/secret.* ignore (ansible/.gitignore:9) must be narrowed so vaulted files can be tracked while plaintext still can't. Keep a defense-in-depth guard against committing a decrypted file (e.g. pre-commit check that secret.yml is vault-encrypted / not plaintext).
5. CI/lint compatibility — confirm ansible-lint (production profile), KICS, and galaxy-build are happy with vaulted files, and that molecule converges without the real vault key (fixture/dummy secret).

Current secret surface (grounding)

Secret	Origin	Where today	In scope?
decdn_rpc_url (may embed API key)	operator-supplied	git-ignored host_vars/<node>/secret.yml; rendered to 0600 decdn.env (DECDN_RPC_URL) on host, no_log: true	yes
eth keystore keystore.json	host-generated	/var/lib/decdn, 0600	no
node.secret (node identity)	host-generated	/var/lib/decdn, 0600	no
keystore password file	host-generated (openssl rand)	host, 0600	no

Open questions

• ansible-vault vs SOPS — ansible-vault is the zero-dependency native fit (just a password/key file; works with existing tooling). SOPS (with age/PGP/KMS) gives per-recipient keys and cloud-KMS backing, better for a multi-operator team, at the cost of a community.sops dependency + .sops.yaml. Which posture do we want? Default recommendation: ansible-vault now, leave SOPS as a documented upgrade path.
• Granularity — whole-file vaulted secret.yml, or inline !vault var so main.yml stays a single diffable file? Inline keeps one file per node but makes main.yml partially opaque.
• Key distribution — how do teammates/CI get the vault password? (1Password/pass, an ops-only shared file, CI secret env). Out of band by definition — but the docs should name a recommended path.
• Backward compatibility — keep supporting the git-ignored plaintext secret.yml for solo operators, or migrate everyone to vault? Prefer: support both, document vault as recommended.

Out of scope

• Host-generated secrets (keystore / node.secret / password) — they never leave the host by design.
• Any change to how the daemon consumes DECDN_RPC_URL at runtime (the 0600 env-file + no_log path stays).