You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now a target has one (or more, via env_refs) vault(s) whose combined env: covers every app running on that target, merged into one shared .env. Downstream, generate-env.sh/lib.sh's generate_env() filters
that single canvas per app by name prefix (APP_NAME, APPS_* shared, or {APPNAME}_* own) before writing apps/{app}/.env — this is how apps are
kept from seeing each other's variables today. The prefix convention exists
specifically because everything gets poured into one shared canvas first
and has to be sorted back out afterward, and it leaks into the compose
files themselves: apps/traefik/docker-compose.yml references ${TRAEFIK_HTTP_PORT} directly, not a vanilla ${HTTP_PORT} — flightdeck
compose files already aren't vanilla-compose-compatible because of this.
An earlier version of this issue proposed fixing that by naming vaults
after the app they belong to (vaults/traefik.yml) and relying on the
filename matching the app name as the link. That's wrong: it reintroduces
exactly the kind of implicit, convention-based wiring this repo has
consistently avoided elsewhere (explicit app_refs/env_refs, "matching
filenames are a convenience, not an implicit relationship" for
vault/target pairing). It also doesn't resolve on its own — see below.
Proposal
Make the target the explicit wiring point between an app and the vault(s)
that feed it, the same way it already explicitly wires app_refs and env_refs today — just scoped per app instead of per target:
replacing the current flat apps: [traefik, rybbit] list — the app names
are just the mapping's keys now, so nothing needed for deploy.sh's APPS= synthesis is lost.
Each app's vault can then declare its env output names directly, with zero
prefix and zero indirection — e.g. vaults/hawkeye-traefik.yml:
HTTP_PORT is exactly what docker-compose.yml can reference — no
prefix stripped anywhere downstream, no hidden rename. Since the vault is
referenced explicitly by ref (not discovered by filename), its filename
doesn't need to match the app name at all — asset names just need to be
unique within a release, same as any other bundle asset.
This resolves what the filename-convention version of this issue couldn't:
No asset-naming collision risk across targets. Nothing depends on
the vault's filename matching the app's name, so two different targets
running the same app never fight over an asset name.
Collisions are detected exactly where they're real, not where they
aren't. Fail-loud detection (from Support multiple env sources (env_refs) merged per target, like app_refs #104/feat: move apps to targets, support env_refs as a list #110) still applies within one
app's own env_refs list. Across different apps there was never a
real merge to begin with — a value like DOMAIN duplicated as the same
mapping in two apps' vaults is structurally guaranteed to resolve to the
same GitHub Secret value; the only way it drifts is a typo in one
mapping, which is an ordinary authoring mistake, not a lost safety net.
Cross-repo secret composition falls out for free, no new mechanism
needed. This is what made Let a vault manifest import other vaults by ref (cross-repo secret composition) #113 (vault-importing-vault, with its own
CI-side decrypt key and encrypt-env changes) unnecessary: a third repo
that needs to contribute e.g. a Telegram bot token just needs a copy of
the target's public key (keys/hawkeye.pub — not secret, freely
copyable to any repo, any org) and to run the already-published encrypt-env action itself in its own release workflow. It publishes
its own encrypted asset to its own release. The target then simply adds
that ref to whichever app's env_refs needs it: owner3/repo3@latest:telegram.sops.env. ansible/deploy.yml's env_refs loop already parses the repo out of the ref itself (not
hardcoded to flightdeck's own repo) and decrypts with the target's own SOPS_AGE_KEY_FILE regardless of which repo published the asset —
decryption is keyed by recipient, not by source repo. Zero code
changes needed for this to already work today. Closed Let a vault manifest import other vaults by ref (cross-repo secret composition) #113 as
superseded by this.
Resolved: how per-app env actually reaches the server
Settled during discussion with #111 (Ansible→Fabric), then superseded by
a bigger decision made in the same implementation session: flightdeck has no manual administration flow at all, ever — the "manual local
quick-start" path described below was deleted outright, not kept as a
separate untouched path.
Automated path is now the only path. Ref-resolution and downloading
run entirely on the CI runner (deploy/deploy.py, see Consider migrating ansible/deploy.yml to Fabric (Python) #111); the CI
runner pushes the finished result to the server rather than the server
pulling anything.
Decryption and config-template rendering both happen on the runner
too (see Consider decrypting vaults on the CI runner instead of server-side #116, decided alongside this) — not server-side as originally
planned here. The server never runs sops, never holds an age key,
never runs envsubst. It receives a real, already-decrypted .env and
already-rendered config directly.
Collision detection still happens in CI, from ciphertext, before
anything is decrypted or pushed — this part of the design didn't
change. SOPS's dotenv output only encrypts values — key names stay in
cleartext in the downloaded .sops.env asset itself
(APPS_DOMAIN=ENC[...]). So when an app's env_refs has more than one
entry, deploy/collisions.py compares key names across the downloaded
encrypted files directly, no decryption needed. Any duplicate key fails
the build before anything is decrypted.
Net result: no shell scripting left on the server at all. up.sh/down.sh/restart.sh/deploy.sh/generate-env.sh/lib.sh
are all deleted, no replacements. The server's only dependencies are
Docker and Docker Compose.
Related
Builds on #104/#108/#110 (env_refs as a list, apps living on the
target). Supersedes and closes #113 (cross-vault import) — same goal,
solved by this design without new machinery. Implemented together with #111 (Ansible→Fabric) — this is the concrete feature the push-based
model in #111 was designed to carry, not built separately in Ansible.
Also affects #103 (ansible ref-resolution dedup), which becomes moot
under #111 rather than needing its own fix.
Implemented, together with #111 and #116, going further than originally
scoped here (see the updated section above).
Context
Right now a target has one (or more, via
env_refs) vault(s) whose combinedenv:covers every app running on that target, merged into one shared.env. Downstream,generate-env.sh/lib.sh'sgenerate_env()filtersthat single canvas per app by name prefix (
APP_NAME,APPS_*shared, or{APPNAME}_*own) before writingapps/{app}/.env— this is how apps arekept from seeing each other's variables today. The prefix convention exists
specifically because everything gets poured into one shared canvas first
and has to be sorted back out afterward, and it leaks into the compose
files themselves:
apps/traefik/docker-compose.ymlreferences${TRAEFIK_HTTP_PORT}directly, not a vanilla${HTTP_PORT}— flightdeckcompose files already aren't vanilla-compose-compatible because of this.
An earlier version of this issue proposed fixing that by naming vaults
after the app they belong to (
vaults/traefik.yml) and relying on thefilename matching the app name as the link. That's wrong: it reintroduces
exactly the kind of implicit, convention-based wiring this repo has
consistently avoided elsewhere (explicit
app_refs/env_refs, "matchingfilenames are a convenience, not an implicit relationship" for
vault/target pairing). It also doesn't resolve on its own — see below.
Proposal
Make the target the explicit wiring point between an app and the vault(s)
that feed it, the same way it already explicitly wires
app_refsandenv_refstoday — just scoped per app instead of per target:replacing the current flat
apps: [traefik, rybbit]list — the app namesare just the mapping's keys now, so nothing needed for
deploy.sh'sAPPS=synthesis is lost.Each app's vault can then declare its env output names directly, with zero
prefix and zero indirection — e.g.
vaults/hawkeye-traefik.yml:HTTP_PORTis exactly whatdocker-compose.ymlcan reference — noprefix stripped anywhere downstream, no hidden rename. Since the vault is
referenced explicitly by ref (not discovered by filename), its filename
doesn't need to match the app name at all — asset names just need to be
unique within a release, same as any other bundle asset.
This resolves what the filename-convention version of this issue couldn't:
the vault's filename matching the app's name, so two different targets
running the same app never fight over an asset name.
aren't. Fail-loud detection (from Support multiple env sources (env_refs) merged per target, like app_refs #104/feat: move apps to targets, support env_refs as a list #110) still applies within one
app's own
env_refslist. Across different apps there was never areal merge to begin with — a value like
DOMAINduplicated as the samemapping in two apps' vaults is structurally guaranteed to resolve to the
same GitHub Secret value; the only way it drifts is a typo in one
mapping, which is an ordinary authoring mistake, not a lost safety net.
needed. This is what made Let a vault manifest import other vaults by ref (cross-repo secret composition) #113 (vault-importing-vault, with its own
CI-side decrypt key and
encrypt-envchanges) unnecessary: a third repothat needs to contribute e.g. a Telegram bot token just needs a copy of
the target's public key (
keys/hawkeye.pub— not secret, freelycopyable to any repo, any org) and to run the already-published
encrypt-envaction itself in its own release workflow. It publishesits own encrypted asset to its own release. The target then simply adds
that ref to whichever app's
env_refsneeds it:owner3/repo3@latest:telegram.sops.env.ansible/deploy.yml'senv_refsloop already parses the repo out of the ref itself (nothardcoded to flightdeck's own repo) and decrypts with the target's own
SOPS_AGE_KEY_FILEregardless of which repo published the asset —decryption is keyed by recipient, not by source repo. Zero code
changes needed for this to already work today. Closed Let a vault manifest import other vaults by ref (cross-repo secret composition) #113 as
superseded by this.
Resolved: how per-app env actually reaches the server
Settled during discussion with #111 (Ansible→Fabric), then superseded by
a bigger decision made in the same implementation session: flightdeck has
no manual administration flow at all, ever — the "manual local
quick-start" path described below was deleted outright, not kept as a
separate untouched path.
run entirely on the CI runner (
deploy/deploy.py, see Consider migrating ansible/deploy.yml to Fabric (Python) #111); the CIrunner pushes the finished result to the server rather than the server
pulling anything.
too (see Consider decrypting vaults on the CI runner instead of server-side #116, decided alongside this) — not server-side as originally
planned here. The server never runs
sops, never holds an age key,never runs
envsubst. It receives a real, already-decrypted.envandalready-rendered config directly.
anything is decrypted or pushed — this part of the design didn't
change. SOPS's dotenv output only encrypts values — key names stay in
cleartext in the downloaded
.sops.envasset itself(
APPS_DOMAIN=ENC[...]). So when an app'senv_refshas more than oneentry,
deploy/collisions.pycompares key names across the downloadedencrypted files directly, no decryption needed. Any duplicate key fails
the build before anything is decrypted.
up.sh/down.sh/restart.sh/deploy.sh/generate-env.sh/lib.share all deleted, no replacements. The server's only dependencies are
Docker and Docker Compose.
Related
Builds on #104/#108/#110 (
env_refsas a list,appsliving on thetarget). Supersedes and closes #113 (cross-vault import) — same goal,
solved by this design without new machinery. Implemented together with
#111 (Ansible→Fabric) — this is the concrete feature the push-based
model in #111 was designed to carry, not built separately in Ansible.
Also affects #103 (ansible ref-resolution dedup), which becomes moot
under #111 rather than needing its own fix.
Implemented, together with #111 and #116, going further than originally
scoped here (see the updated section above).