From b9a5033ef87bd74a2a192da9fe6fbcfe04ed2654 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Wed, 26 Aug 2026 00:27:11 +0200 Subject: [PATCH 01/22] feat: prototype a folder-driven renovate workflow (#121) Deliberately a different input contract from deploy-shared.yml's per-target matrix, to compare the two before picking one: renovate.yml takes an app name and a targets directory, and deploy/renovate.py itself finds which targets currently run that app and re-pulls/recreates its containers there - no app_refs/env_refs re-resolution, no release rebuild, no vault decryption. Known gap: no Tailscale support yet. --- .github/workflows/renovate.yml | 36 +++++++ README.md | 20 ++++ deploy/renovate.py | 98 +++++++++++++++++++ deploy/tests/test_renovate.py | 168 +++++++++++++++++++++++++++++++++ 4 files changed, 322 insertions(+) create mode 100644 .github/workflows/renovate.yml create mode 100644 deploy/renovate.py create mode 100644 deploy/tests/test_renovate.py diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml new file mode 100644 index 0000000..0afed53 --- /dev/null +++ b/.github/workflows/renovate.yml @@ -0,0 +1,36 @@ +name: Renovate +on: + workflow_dispatch: + inputs: + app: + description: App name to renovate (re-pull its image and recreate its containers) wherever it's currently deployed. + type: string + required: true + targets-directory: + description: Directory of target manifests to search for the app. + type: string + default: targets +jobs: + renovate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - name: Install deploy dependencies + shell: bash + run: pip install --user --break-system-packages -r deploy/requirements.txt + - name: Start SSH agent + shell: bash + run: | + eval "$(ssh-agent -s)" + echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" + echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" + - name: Run renovate + shell: bash + env: + APP: ${{ inputs.app }} + TARGETS_DIRECTORY: ${{ inputs.targets-directory }} + GITHUB_SECRETS_JSON: ${{ toJson(secrets) }} + run: | + jq -n --arg app "$APP" --arg targets_directory "$TARGETS_DIRECTORY" \ + '{app: $app, targets_directory: $targets_directory}' \ + | python3 deploy/renovate.py diff --git a/README.md b/README.md index 7c5e2ee..3b21ea5 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,7 @@ flightdeck/ │ ├── deploy/ │ ├── deploy.py # Push-based deploy entrypoint (runs on the CI runner) +│ ├── renovate.py # Prototype: re-pull/recreate one app's containers across every matching target │ ├── resolve.py # owner/repo@tag[:asset] release ref resolution/download │ ├── collisions.py # Ciphertext-based env key collision detection │ ├── vault.py # SOPS decryption @@ -72,6 +73,7 @@ flightdeck/ │ │ └── load-yaml-matrix/ # Read a directory of YAML manifests into a workflow matrix │ └── workflows/ │ ├── deploy-shared.yml # Reusable deployment workflow +│ ├── renovate.yml # Prototype: folder-driven, non-matrix app renovation │ └── release.yml # Release Please + publish Flightdeck assets │ ├── vaults/ # Encrypted env asset configurations, one per app @@ -362,6 +364,24 @@ jobs: The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/deploy.py` itself. `app-refs` entries are separate and don't have to match the workflow pin. +--- + +### `renovate.yml` (prototype) + +**Experimental — a deliberately different contract shape from `deploy-shared.yml`, kept around to compare against before settling on one approach for #120/#121.** Not a reusable `workflow_call` yet; it's a repo-local `workflow_dispatch` in this repository, since `targets/` already lives here. + +`deploy-shared.yml` is invoked once per target by the *caller's* own matrix (see `deploy.yml`) — the caller resolves each target's secrets before the reusable workflow ever runs. `renovate.yml` inverts that: it takes an `app` name and a `targets-directory` (default `targets`), and [`deploy/renovate.py`](deploy/renovate.py) itself reads every manifest in that directory, finds which targets currently run that app, and renovates each match — no matrix, no per-target `workflow_call`. This is the only way to express "find this app on every target that runs it" as an input, since a GitHub Actions matrix has to be resolved by the caller before the job starts, and the whole point here is that the caller doesn't know which targets match ahead of time. + +Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. Because the job resolves secrets for potentially several matched targets itself (not per-matrix-cell), it reads the whole `GITHUB_SECRETS_JSON` blob (same pattern `encrypt-env` uses) and looks up each matched target's `credentials.secrets.ssh_private_key` by name at runtime, loading it into the job's SSH agent per target. + +**Known gap:** unlike `deploy-shared.yml`, this has no Tailscale support yet — a matched target only reachable over a tailnet can't be renovated by this workflow today. Solving that per-target, inside one job, needs either multiple sequential `tailscale up`/`tailscale down` cycles or driving Tailscale's OAuth-to-authkey exchange directly instead of the marketplace action (which only runs once per job at the YAML level). Deferred until this contract shape is the chosen one. + +```yaml +# workflow_dispatch inputs: +# app: beszel # required +# targets-directory: targets # optional, default shown +``` + ## License Flightdeck is released under the [MIT License](LICENSE). diff --git a/deploy/renovate.py b/deploy/renovate.py new file mode 100644 index 0000000..2a28559 --- /dev/null +++ b/deploy/renovate.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Renovate: re-pull and recreate one app's containers, wherever it's +already deployed, without touching versions - no new app bundle, no new +vault-sourced env, no rebuilt release tree. Just `docker compose pull && +docker compose up -d` against that app's already-current release on each +matching target host. + +Unlike deploy/deploy.py (one target per invocation, resolved by the +caller's own GitHub Actions matrix), this reads every target manifest +under a directory itself and finds which ones run the given app - the +input is a directory, not a pre-resolved single target's config. See +README's "Renovate (prototype)" section for the exact contract. + +Reads a JSON config from stdin: {"app": "", "targets_directory": "targets"}. +Needs GITHUB_SECRETS_JSON in the environment to resolve each matched +target's `credentials.secrets.ssh_private_key` GitHub Secret name to its +actual value - there's no per-target workflow_call here to do that ahead +of time. +""" +import json +import os +import shlex +import subprocess +import sys +from pathlib import Path + +import paramiko +import yaml +from fabric import Connection + + +class RenovateError(Exception): + pass + + +def load_targets(directory): + directory = Path(directory) + paths = sorted(directory.glob("*.yml")) + sorted(directory.glob("*.yaml")) + if not paths: + raise RenovateError(f"no target manifests found in {directory}") + return [(path.stem, yaml.safe_load(path.read_text())) for path in paths] + + +def find_matching_targets(targets, app): + return [(name, manifest) for name, manifest in targets if app in (manifest.get("apps") or {})] + + +def resolve_secret(secrets, name): + if name not in secrets: + raise RenovateError(f"GitHub Secret not found: {name}") + return secrets[name] + + +def load_ssh_key(secrets, secret_name, run=subprocess.run): + key_text = resolve_secret(secrets, secret_name) + result = run(["ssh-add", "-"], input=key_text, capture_output=True, text=True) + if result.returncode != 0: + raise RenovateError(f"Failed to load SSH key {secret_name}: {result.stderr.strip()}") + + +def expand_home(path, home): + return home + path[1:] if path.startswith("~") else path + + +def renovate_host(host, base_path, app): + connection = Connection(host) + connection.client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + home = connection.run("echo $HOME", hide=True).stdout.strip() + compose_dir = f"{expand_home(base_path, home)}/current/apps/{app}" + connection.run(f"cd {shlex.quote(compose_dir)} && docker compose pull && docker compose up -d --remove-orphans") + + +def renovate_target(name, manifest, app, secrets, run=subprocess.run): + ssh_secret_name = manifest["credentials"]["secrets"]["ssh_private_key"] + load_ssh_key(secrets, ssh_secret_name, run=run) + base_path = manifest.get("path", "~/flightdeck") + for host in manifest["hosts"]: + print(f"Renovating {app} on {name} ({host})") + renovate_host(host, base_path, app) + + +def main(): + config = json.load(sys.stdin) + app = config["app"] + targets_directory = config.get("targets_directory", "targets") + secrets = json.loads(os.environ["GITHUB_SECRETS_JSON"]) + + targets = load_targets(targets_directory) + matches = find_matching_targets(targets, app) + if not matches: + raise RenovateError(f"app {app!r} not found in any target manifest under {targets_directory}") + + for name, manifest in matches: + renovate_target(name, manifest, app, secrets) + + +if __name__ == "__main__": + main() diff --git a/deploy/tests/test_renovate.py b/deploy/tests/test_renovate.py new file mode 100644 index 0000000..f42246f --- /dev/null +++ b/deploy/tests/test_renovate.py @@ -0,0 +1,168 @@ +import importlib.util +import io +import json +import os +import sys +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +DEPLOY_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(DEPLOY_DIR)) + +MODULE_PATH = DEPLOY_DIR / "renovate.py" +SPEC = importlib.util.spec_from_file_location("renovate_entrypoint", MODULE_PATH) +renovate = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(renovate) + + +def write_target(directory, name, content): + path = Path(directory) / f"{name}.yml" + path.write_text(content) + return path + + +class FakeConnection: + """Stand-in for fabric.Connection - records commands instead of opening + a real SSH session, mirroring test_deploy.py's FakeConnection.""" + + def __init__(self, host): + self.host = host + self.client = SimpleNamespace(set_missing_host_key_policy=lambda policy: None) + self.commands = [] + + def run(self, command, hide=False): + self.commands.append(command) + if command == "echo $HOME": + return SimpleNamespace(stdout="/home/deploy\n") + return SimpleNamespace(stdout="") + + +class LoadTargetsTest(unittest.TestCase): + def test_reads_every_manifest_in_directory(self): + with tempfile.TemporaryDirectory() as directory: + write_target(directory, "heimdall", "apps:\n traefik: {}\n") + write_target(directory, "mainframe", "apps:\n rybbit: {}\n") + + targets = renovate.load_targets(directory) + + self.assertEqual( + sorted(targets), + [("heimdall", {"apps": {"traefik": {}}}), ("mainframe", {"apps": {"rybbit": {}}})], + ) + + def test_raises_when_directory_has_no_manifests(self): + with tempfile.TemporaryDirectory() as directory, self.assertRaises(renovate.RenovateError): + renovate.load_targets(directory) + + +class FindMatchingTargetsTest(unittest.TestCase): + def test_matches_targets_running_the_app(self): + targets = [ + ("heimdall", {"apps": {"traefik": {}, "beszel": {}}}), + ("mainframe", {"apps": {"rybbit": {}}}), + ] + + matches = renovate.find_matching_targets(targets, "beszel") + + self.assertEqual(matches, [("heimdall", {"apps": {"traefik": {}, "beszel": {}}})]) + + def test_ignores_targets_without_an_apps_mapping(self): + targets = [("empty", {})] + + self.assertEqual(renovate.find_matching_targets(targets, "beszel"), []) + + +class LoadSshKeyTest(unittest.TestCase): + def test_adds_resolved_secret_to_the_agent(self): + calls = [] + + def fake_run(args, input=None, capture_output=None, text=None): + calls.append((args, input)) + return SimpleNamespace(returncode=0, stderr="") + + renovate.load_ssh_key({"HEIMDALL_SSH_KEY": "-----KEY-----"}, "HEIMDALL_SSH_KEY", run=fake_run) + + self.assertEqual(calls, [(["ssh-add", "-"], "-----KEY-----")]) + + def test_raises_on_missing_secret(self): + with self.assertRaises(renovate.RenovateError): + renovate.load_ssh_key({}, "MISSING_SECRET") + + def test_raises_when_ssh_add_fails(self): + def fake_run(args, input=None, capture_output=None, text=None): + return SimpleNamespace(returncode=1, stderr="bad key") + + with self.assertRaises(renovate.RenovateError): + renovate.load_ssh_key({"KEY": "text"}, "KEY", run=fake_run) + + +class RenovateTargetTest(unittest.TestCase): + def test_renovates_every_host_without_touching_versions(self): + manifest = { + "hosts": ["deploy@app1.example.com", "deploy@app2.example.com"], + "path": "~/flightdeck", + "credentials": {"secrets": {"ssh_private_key": "DEPLOY_SSH_PRIVATE_KEY"}}, + } + secrets = {"DEPLOY_SSH_PRIVATE_KEY": "-----KEY-----"} + fakes = {} + + def fake_connection(host): + fakes[host] = FakeConnection(host) + return fakes[host] + + ssh_calls = [] + + def fake_run(args, input=None, capture_output=None, text=None): + ssh_calls.append(input) + return SimpleNamespace(returncode=0, stderr="") + + with patch.object(renovate, "Connection", side_effect=fake_connection): + renovate.renovate_target("mainframe", manifest, "beszel", secrets, run=fake_run) + + self.assertEqual(ssh_calls, ["-----KEY-----"]) + for host in manifest["hosts"]: + joined = "\n".join(fakes[host].commands) + self.assertIn("cd /home/deploy/flightdeck/current/apps/beszel", joined) + self.assertIn("docker compose pull && docker compose up -d --remove-orphans", joined) + + +class MainTest(unittest.TestCase): + def _run_main(self, directory, app, secrets): + stdin = io.StringIO(json.dumps({"app": app, "targets_directory": str(directory)})) + with ( + patch.object(sys, "stdin", stdin), + patch.dict(os.environ, {"GITHUB_SECRETS_JSON": json.dumps(secrets)}), + ): + renovate.main() + + def test_raises_when_app_matches_no_target(self): + with tempfile.TemporaryDirectory() as directory: + write_target(directory, "heimdall", "apps:\n traefik: {}\n") + + with self.assertRaises(renovate.RenovateError): + self._run_main(directory, "beszel", {}) + + def test_renovates_every_matching_target(self): + with tempfile.TemporaryDirectory() as directory: + write_target( + directory, + "heimdall", + "apps:\n beszel: {}\nhosts: [deploy@host]\ncredentials:\n secrets:\n ssh_private_key: KEY\n", + ) + write_target(directory, "mainframe", "apps:\n rybbit: {}\nhosts: [deploy@other]\n") + + with ( + patch.object(renovate, "renovate_target") as fake_renovate_target, + ): + self._run_main(directory, "beszel", {"KEY": "-----KEY-----"}) + + fake_renovate_target.assert_called_once() + name, manifest, app, secrets = fake_renovate_target.call_args[0] + self.assertEqual((name, app), ("heimdall", "beszel")) + + +if __name__ == "__main__": + unittest.main() From bcce474e4d4d2c8f8c8d953b329866febb9c8c83 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Wed, 26 Aug 2026 00:34:27 +0200 Subject: [PATCH 02/22] refactor: split renovate into a thin trigger + a shared reusable workflow Match deploy.yml/deploy-shared.yml's two-file shape - renovate.yml is now just a workflow_dispatch trigger, and all the logic (checkout, running deploy/renovate.py) lives in renovate-shared.yml, called with secrets: inherit since it resolves secrets for matched targets itself rather than per-matrix-cell like deploy-shared.yml does. --- .github/workflows/renovate-shared.yml | 36 +++++++++++++++++++++++++++ .github/workflows/renovate.yml | 27 ++++---------------- README.md | 28 ++++++++++++++------- 3 files changed, 60 insertions(+), 31 deletions(-) create mode 100644 .github/workflows/renovate-shared.yml diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml new file mode 100644 index 0000000..e15401b --- /dev/null +++ b/.github/workflows/renovate-shared.yml @@ -0,0 +1,36 @@ +name: Renovate (shared) +on: + workflow_call: + inputs: + app: + description: App name to renovate (re-pull its image and recreate its containers) wherever it's currently deployed. + type: string + required: true + targets-directory: + description: Directory of target manifests to search for the app. + type: string + default: targets +jobs: + renovate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - name: Install deploy dependencies + shell: bash + run: pip install --user --break-system-packages -r deploy/requirements.txt + - name: Start SSH agent + shell: bash + run: | + eval "$(ssh-agent -s)" + echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" + echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" + - name: Run renovate + shell: bash + env: + APP: ${{ inputs.app }} + TARGETS_DIRECTORY: ${{ inputs.targets-directory }} + GITHUB_SECRETS_JSON: ${{ toJson(secrets) }} + run: | + jq -n --arg app "$APP" --arg targets_directory "$TARGETS_DIRECTORY" \ + '{app: $app, targets_directory: $targets_directory}' \ + | python3 deploy/renovate.py diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index 0afed53..6762f7b 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -12,25 +12,8 @@ on: default: targets jobs: renovate: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v7 - - name: Install deploy dependencies - shell: bash - run: pip install --user --break-system-packages -r deploy/requirements.txt - - name: Start SSH agent - shell: bash - run: | - eval "$(ssh-agent -s)" - echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" - echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" - - name: Run renovate - shell: bash - env: - APP: ${{ inputs.app }} - TARGETS_DIRECTORY: ${{ inputs.targets-directory }} - GITHUB_SECRETS_JSON: ${{ toJson(secrets) }} - run: | - jq -n --arg app "$APP" --arg targets_directory "$TARGETS_DIRECTORY" \ - '{app: $app, targets_directory: $targets_directory}' \ - | python3 deploy/renovate.py + uses: $/.github/workflows/renovate-shared.yml + with: + app: ${{ inputs.app }} + targets-directory: ${{ inputs.targets-directory }} + secrets: inherit diff --git a/README.md b/README.md index 3b21ea5..8c84315 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,8 @@ flightdeck/ │ │ └── load-yaml-matrix/ # Read a directory of YAML manifests into a workflow matrix │ └── workflows/ │ ├── deploy-shared.yml # Reusable deployment workflow -│ ├── renovate.yml # Prototype: folder-driven, non-matrix app renovation +│ ├── renovate.yml # Prototype: thin workflow_dispatch trigger for renovate-shared.yml +│ ├── renovate-shared.yml # Prototype: folder-driven, non-matrix reusable renovation workflow │ └── release.yml # Release Please + publish Flightdeck assets │ ├── vaults/ # Encrypted env asset configurations, one per app @@ -366,20 +367,29 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl --- -### `renovate.yml` (prototype) +### `renovate.yml` / `renovate-shared.yml` (prototype) -**Experimental — a deliberately different contract shape from `deploy-shared.yml`, kept around to compare against before settling on one approach for #120/#121.** Not a reusable `workflow_call` yet; it's a repo-local `workflow_dispatch` in this repository, since `targets/` already lives here. +**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s matrix approach before settling on one contract for #120/#121.** Mirrors that same two-file split — a thin `workflow_dispatch` trigger (`renovate.yml`) calling a reusable `workflow_call` workflow (`renovate-shared.yml`) that holds all the actual logic — but the *contract* between the two files is deliberately different, because the problem shape is different. -`deploy-shared.yml` is invoked once per target by the *caller's* own matrix (see `deploy.yml`) — the caller resolves each target's secrets before the reusable workflow ever runs. `renovate.yml` inverts that: it takes an `app` name and a `targets-directory` (default `targets`), and [`deploy/renovate.py`](deploy/renovate.py) itself reads every manifest in that directory, finds which targets currently run that app, and renovates each match — no matrix, no per-target `workflow_call`. This is the only way to express "find this app on every target that runs it" as an input, since a GitHub Actions matrix has to be resolved by the caller before the job starts, and the whole point here is that the caller doesn't know which targets match ahead of time. +`deploy.yml` resolves a matrix from `targets/` itself (via `load-yaml-matrix`) and calls `deploy-shared.yml` once per target, with that target's secrets already picked out by name (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — `deploy-shared.yml` never reads `targets/` itself, it only ever sees one already-resolved target. `renovate.yml` can't do that: "find every target currently running this app" can't be answered by the caller before the job starts, since a GitHub Actions matrix has to be fully resolved ahead of time and that's exactly the thing we don't know yet. So `renovate.yml` just passes `app`/`targets-directory` straight through with `secrets: inherit`, and [`deploy/renovate.py`](deploy/renovate.py) — running inside `renovate-shared.yml` — reads every manifest in that directory itself, finds the matches, and renovates each one in a single job. -Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. Because the job resolves secrets for potentially several matched targets itself (not per-matrix-cell), it reads the whole `GITHUB_SECRETS_JSON` blob (same pattern `encrypt-env` uses) and looks up each matched target's `credentials.secrets.ssh_private_key` by name at runtime, loading it into the job's SSH agent per target. +Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. Because one job may act on several matched targets (not one per matrix cell), `renovate-shared.yml` needs `secrets: inherit` from its caller and reads the whole `GITHUB_SECRETS_JSON` blob (same pattern `encrypt-env` uses) so `deploy/renovate.py` can look up each matched target's `credentials.secrets.ssh_private_key` by name at runtime, loading it into the job's SSH agent per target. -**Known gap:** unlike `deploy-shared.yml`, this has no Tailscale support yet — a matched target only reachable over a tailnet can't be renovated by this workflow today. Solving that per-target, inside one job, needs either multiple sequential `tailscale up`/`tailscale down` cycles or driving Tailscale's OAuth-to-authkey exchange directly instead of the marketplace action (which only runs once per job at the YAML level). Deferred until this contract shape is the chosen one. +**Known gaps:** + +- No Tailscale support yet — a matched target only reachable over a tailnet can't be renovated by this workflow today. Solving that per-target, inside one job, needs either multiple sequential `tailscale up`/`tailscale down` cycles or driving Tailscale's OAuth-to-authkey exchange directly instead of the marketplace action (which only runs once per job at the YAML level). +- `renovate-shared.yml` only checks out its caller's own repository, so it currently only works when the caller's repo *is* this one (as it is here, since `targets/heimdall.yml` lives in this repo). Genuine cross-repo reuse - a private consumer repo calling `rubykatzen/flightdeck/.github/workflows/renovate-shared.yml@vX` for its own `targets/` - would need a second checkout of flightdeck's own ref (like `deploy-shared.yml` does) to get `deploy/renovate.py` alongside the caller's `targets/`, into separate paths. + +Both deferred until this contract shape is the chosen one. ```yaml -# workflow_dispatch inputs: -# app: beszel # required -# targets-directory: targets # optional, default shown +jobs: + renovate: + uses: $/.github/workflows/renovate-shared.yml + with: + app: beszel # required + targets-directory: targets # optional, default shown + secrets: inherit ``` ## License From f0dfce290274369b3f1259f25971587f6f234cbc Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Wed, 26 Aug 2026 00:57:45 +0200 Subject: [PATCH 03/22] refactor: compute renovate's target matrix inside renovate-shared.yml Keep load-yaml-matrix a dumb, generic directory-to-matrix reader (no app-filtering added to it) - instead renovate-shared.yml gets a second job, mirroring deploy.yml/deploy-shared.yml's split: find-targets loads every target unfiltered, then renovate fans out over that matrix one job per target, with native per-matrix-cell secret/Tailscale resolution. Every target gets dispatched regardless of whether it runs the requested app; deploy/renovate.py itself decides to no-op when its one target's apps mapping doesn't contain it. This drops the directory-reading and GITHUB_SECRETS_JSON-parsing deploy/renovate.py used to do - it's now scoped to one already-resolved target, same shape as deploy/deploy.py. --- .github/workflows/renovate-shared.yml | 36 +++++- README.md | 16 +-- deploy/renovate.py | 92 ++++----------- deploy/tests/test_renovate.py | 160 +++++++------------------- 4 files changed, 106 insertions(+), 198 deletions(-) diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml index e15401b..89a2535 100644 --- a/.github/workflows/renovate-shared.yml +++ b/.github/workflows/renovate-shared.yml @@ -11,26 +11,52 @@ on: type: string default: targets jobs: + find-targets: + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.matrix.outputs.matrix }} + count: ${{ steps.matrix.outputs.count }} + steps: + - uses: actions/checkout@v7 + - uses: $/.github/actions/load-yaml-matrix + id: matrix + with: + directory: ${{ inputs.targets-directory }} renovate: + needs: find-targets + if: needs.find-targets.outputs.count != '0' + strategy: + matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 + with: + repository: ${{ job.workflow_repository }} + ref: ${{ job.workflow_sha }} - name: Install deploy dependencies shell: bash run: pip install --user --break-system-packages -r deploy/requirements.txt - - name: Start SSH agent + - uses: tailscale/github-action@v4 + if: matrix.credentials.variables.tailscale_oauth_client_id + with: + oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} + oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} + tags: tag:ci + - name: Load SSH key shell: bash run: | eval "$(ssh-agent -s)" echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" + ssh-add - <<< "${{ secrets[matrix.credentials.secrets.ssh_private_key] }}" - name: Run renovate shell: bash env: APP: ${{ inputs.app }} - TARGETS_DIRECTORY: ${{ inputs.targets-directory }} - GITHUB_SECRETS_JSON: ${{ toJson(secrets) }} + HOSTS: ${{ toJson(matrix.hosts) }} + TARGET_PATH: ${{ matrix.path || '~/flightdeck' }} + TARGET_APPS: ${{ toJson(matrix.apps) }} run: | - jq -n --arg app "$APP" --arg targets_directory "$TARGETS_DIRECTORY" \ - '{app: $app, targets_directory: $targets_directory}' \ + jq -n --arg app "$APP" --argjson hosts "$HOSTS" --arg path "$TARGET_PATH" --argjson apps "$TARGET_APPS" \ + '{app: $app, hosts: $hosts, path: $path, apps: $apps}' \ | python3 deploy/renovate.py diff --git a/README.md b/README.md index 8c84315..5b60221 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ flightdeck/ │ ├── deploy/ │ ├── deploy.py # Push-based deploy entrypoint (runs on the CI runner) -│ ├── renovate.py # Prototype: re-pull/recreate one app's containers across every matching target +│ ├── renovate.py # Prototype: re-pull/recreate one app's containers on one already-resolved target │ ├── resolve.py # owner/repo@tag[:asset] release ref resolution/download │ ├── collisions.py # Ciphertext-based env key collision detection │ ├── vault.py # SOPS decryption @@ -74,7 +74,7 @@ flightdeck/ │ └── workflows/ │ ├── deploy-shared.yml # Reusable deployment workflow │ ├── renovate.yml # Prototype: thin workflow_dispatch trigger for renovate-shared.yml -│ ├── renovate-shared.yml # Prototype: folder-driven, non-matrix reusable renovation workflow +│ ├── renovate-shared.yml # Prototype: computes its own target matrix, then fans out like deploy-shared.yml │ └── release.yml # Release Please + publish Flightdeck assets │ ├── vaults/ # Encrypted env asset configurations, one per app @@ -369,18 +369,18 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl ### `renovate.yml` / `renovate-shared.yml` (prototype) -**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s matrix approach before settling on one contract for #120/#121.** Mirrors that same two-file split — a thin `workflow_dispatch` trigger (`renovate.yml`) calling a reusable `workflow_call` workflow (`renovate-shared.yml`) that holds all the actual logic — but the *contract* between the two files is deliberately different, because the problem shape is different. +**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-file split as deploy (a thin `workflow_dispatch` trigger calling a reusable `workflow_call` workflow), and now the *same* matrix mechanics too — the difference that's actually being compared is narrower than it first looks: where the target matrix gets computed. -`deploy.yml` resolves a matrix from `targets/` itself (via `load-yaml-matrix`) and calls `deploy-shared.yml` once per target, with that target's secrets already picked out by name (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — `deploy-shared.yml` never reads `targets/` itself, it only ever sees one already-resolved target. `renovate.yml` can't do that: "find every target currently running this app" can't be answered by the caller before the job starts, since a GitHub Actions matrix has to be fully resolved ahead of time and that's exactly the thing we don't know yet. So `renovate.yml` just passes `app`/`targets-directory` straight through with `secrets: inherit`, and [`deploy/renovate.py`](deploy/renovate.py) — running inside `renovate-shared.yml` — reads every manifest in that directory itself, finds the matches, and renovates each one in a single job. +`deploy.yml` computes its matrix from `targets/` itself and calls `deploy-shared.yml` once per already-known target. `renovate.yml` can't do that — "find every target currently running this app" isn't answerable by the outer trigger before the job starts, since a matrix has to be fully resolved ahead of time and that's exactly the thing we don't know yet. So `renovate.yml` just passes `app`/`targets-directory` straight through with `secrets: inherit`, and `renovate-shared.yml` computes the matrix *itself*, in its own first job (`find-targets`, plain `load-yaml-matrix` over every manifest — deliberately not filtered by app, so that utility stays exactly as dumb and generic as it already is). Its second job (`renovate`) then fans out over that matrix exactly like `deploy-shared.yml` does — one job per target, with native per-matrix-cell secret/Tailscale resolution (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — except every target gets dispatched unconditionally, whether or not it actually runs the requested app. [`deploy/renovate.py`](deploy/renovate.py) itself makes that call: if `app` isn't a key in the target's own `apps` mapping, it just logs and no-ops instead of erroring, since a target-matrix fan-out has no other way to say "skip me." -Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. Because one job may act on several matched targets (not one per matrix cell), `renovate-shared.yml` needs `secrets: inherit` from its caller and reads the whole `GITHUB_SECRETS_JSON` blob (same pattern `encrypt-env` uses) so `deploy/renovate.py` can look up each matched target's `credentials.secrets.ssh_private_key` by name at runtime, loading it into the job's SSH agent per target. +Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. **Known gaps:** -- No Tailscale support yet — a matched target only reachable over a tailnet can't be renovated by this workflow today. Solving that per-target, inside one job, needs either multiple sequential `tailscale up`/`tailscale down` cycles or driving Tailscale's OAuth-to-authkey exchange directly instead of the marketplace action (which only runs once per job at the YAML level). -- `renovate-shared.yml` only checks out its caller's own repository, so it currently only works when the caller's repo *is* this one (as it is here, since `targets/heimdall.yml` lives in this repo). Genuine cross-repo reuse - a private consumer repo calling `rubykatzen/flightdeck/.github/workflows/renovate-shared.yml@vX` for its own `targets/` - would need a second checkout of flightdeck's own ref (like `deploy-shared.yml` does) to get `deploy/renovate.py` alongside the caller's `targets/`, into separate paths. +- If `app` matches no target at all (a typo, say), every matrix job just no-ops and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. +- `find-targets` checks out its caller's own repository (to read `targets/`), while `renovate` checks out flightdeck's own ref (to get `deploy/renovate.py`, like `deploy-shared.yml` does) — so genuine cross-repo reuse (a private consumer repo calling `rubykatzen/flightdeck/.github/workflows/renovate-shared.yml@vX` for its own `targets/`) should already work, but hasn't been exercised outside this repo yet. -Both deferred until this contract shape is the chosen one. +Deferred until this contract shape is the chosen one. ```yaml jobs: diff --git a/deploy/renovate.py b/deploy/renovate.py index 2a28559..aa9e11c 100644 --- a/deploy/renovate.py +++ b/deploy/renovate.py @@ -1,63 +1,31 @@ #!/usr/bin/env python3 -"""Renovate: re-pull and recreate one app's containers, wherever it's -already deployed, without touching versions - no new app bundle, no new +"""Renovate: re-pull and recreate one app's containers on one already- +resolved target, without touching versions - no new app bundle, no new vault-sourced env, no rebuilt release tree. Just `docker compose pull && docker compose up -d` against that app's already-current release on each -matching target host. - -Unlike deploy/deploy.py (one target per invocation, resolved by the -caller's own GitHub Actions matrix), this reads every target manifest -under a directory itself and finds which ones run the given app - the -input is a directory, not a pre-resolved single target's config. See -README's "Renovate (prototype)" section for the exact contract. - -Reads a JSON config from stdin: {"app": "", "targets_directory": "targets"}. -Needs GITHUB_SECRETS_JSON in the environment to resolve each matched -target's `credentials.secrets.ssh_private_key` GitHub Secret name to its -actual value - there's no per-target workflow_call here to do that ahead -of time. +of the target's hosts. + +One invocation is scoped to one target, same as deploy/deploy.py - the +matrix fan-out across every target under a directory, and per-target +secret/Tailscale resolution, live in renovate-shared.yml's own two-job +matrix (see README's "Renovate" section), not here. This script doesn't +know about targets/ or GitHub Secrets at all. + +Reads a JSON config from stdin: {"app": "", "hosts": [...], "path": +"...", "apps": {...}}. `apps` is this one target's own `apps` mapping, as +declared in its targets/*.yml manifest - since the matrix fans out to +every target regardless of whether it actually runs the requested app, +`app` not being a key in it just means this target is a clean no-op, not +an error. """ import json -import os import shlex -import subprocess import sys -from pathlib import Path import paramiko -import yaml from fabric import Connection -class RenovateError(Exception): - pass - - -def load_targets(directory): - directory = Path(directory) - paths = sorted(directory.glob("*.yml")) + sorted(directory.glob("*.yaml")) - if not paths: - raise RenovateError(f"no target manifests found in {directory}") - return [(path.stem, yaml.safe_load(path.read_text())) for path in paths] - - -def find_matching_targets(targets, app): - return [(name, manifest) for name, manifest in targets if app in (manifest.get("apps") or {})] - - -def resolve_secret(secrets, name): - if name not in secrets: - raise RenovateError(f"GitHub Secret not found: {name}") - return secrets[name] - - -def load_ssh_key(secrets, secret_name, run=subprocess.run): - key_text = resolve_secret(secrets, secret_name) - result = run(["ssh-add", "-"], input=key_text, capture_output=True, text=True) - if result.returncode != 0: - raise RenovateError(f"Failed to load SSH key {secret_name}: {result.stderr.strip()}") - - def expand_home(path, home): return home + path[1:] if path.startswith("~") else path @@ -70,28 +38,18 @@ def renovate_host(host, base_path, app): connection.run(f"cd {shlex.quote(compose_dir)} && docker compose pull && docker compose up -d --remove-orphans") -def renovate_target(name, manifest, app, secrets, run=subprocess.run): - ssh_secret_name = manifest["credentials"]["secrets"]["ssh_private_key"] - load_ssh_key(secrets, ssh_secret_name, run=run) - base_path = manifest.get("path", "~/flightdeck") - for host in manifest["hosts"]: - print(f"Renovating {app} on {name} ({host})") - renovate_host(host, base_path, app) - - def main(): config = json.load(sys.stdin) app = config["app"] - targets_directory = config.get("targets_directory", "targets") - secrets = json.loads(os.environ["GITHUB_SECRETS_JSON"]) - - targets = load_targets(targets_directory) - matches = find_matching_targets(targets, app) - if not matches: - raise RenovateError(f"app {app!r} not found in any target manifest under {targets_directory}") - - for name, manifest in matches: - renovate_target(name, manifest, app, secrets) + apps = config.get("apps") or {} + if app not in apps: + print(f"{app!r} is not deployed on this target, skipping") + return + + base_path = config.get("path", "~/flightdeck") + for host in config["hosts"]: + print(f"Renovating {app} on {host}") + renovate_host(host, base_path, app) if __name__ == "__main__": diff --git a/deploy/tests/test_renovate.py b/deploy/tests/test_renovate.py index f42246f..a065f56 100644 --- a/deploy/tests/test_renovate.py +++ b/deploy/tests/test_renovate.py @@ -1,9 +1,7 @@ import importlib.util import io import json -import os import sys -import tempfile import unittest from pathlib import Path from types import SimpleNamespace @@ -18,12 +16,6 @@ SPEC.loader.exec_module(renovate) -def write_target(directory, name, content): - path = Path(directory) / f"{name}.yml" - path.write_text(content) - return path - - class FakeConnection: """Stand-in for fabric.Connection - records commands instead of opening a real SSH session, mirroring test_deploy.py's FakeConnection.""" @@ -40,128 +32,60 @@ def run(self, command, hide=False): return SimpleNamespace(stdout="") -class LoadTargetsTest(unittest.TestCase): - def test_reads_every_manifest_in_directory(self): - with tempfile.TemporaryDirectory() as directory: - write_target(directory, "heimdall", "apps:\n traefik: {}\n") - write_target(directory, "mainframe", "apps:\n rybbit: {}\n") - - targets = renovate.load_targets(directory) - - self.assertEqual( - sorted(targets), - [("heimdall", {"apps": {"traefik": {}}}), ("mainframe", {"apps": {"rybbit": {}}})], - ) - - def test_raises_when_directory_has_no_manifests(self): - with tempfile.TemporaryDirectory() as directory, self.assertRaises(renovate.RenovateError): - renovate.load_targets(directory) - - -class FindMatchingTargetsTest(unittest.TestCase): - def test_matches_targets_running_the_app(self): - targets = [ - ("heimdall", {"apps": {"traefik": {}, "beszel": {}}}), - ("mainframe", {"apps": {"rybbit": {}}}), - ] - - matches = renovate.find_matching_targets(targets, "beszel") - - self.assertEqual(matches, [("heimdall", {"apps": {"traefik": {}, "beszel": {}}})]) - - def test_ignores_targets_without_an_apps_mapping(self): - targets = [("empty", {})] - - self.assertEqual(renovate.find_matching_targets(targets, "beszel"), []) - +class ExpandHomeTest(unittest.TestCase): + def test_expands_tilde_prefix(self): + self.assertEqual(renovate.expand_home("~/flightdeck", "/home/deploy"), "/home/deploy/flightdeck") -class LoadSshKeyTest(unittest.TestCase): - def test_adds_resolved_secret_to_the_agent(self): - calls = [] + def test_leaves_absolute_path_untouched(self): + self.assertEqual(renovate.expand_home("/opt/flightdeck", "/home/deploy"), "/opt/flightdeck") - def fake_run(args, input=None, capture_output=None, text=None): - calls.append((args, input)) - return SimpleNamespace(returncode=0, stderr="") - renovate.load_ssh_key({"HEIMDALL_SSH_KEY": "-----KEY-----"}, "HEIMDALL_SSH_KEY", run=fake_run) +class RenovateHostTest(unittest.TestCase): + def test_pulls_and_recreates_without_touching_the_release(self): + fake = FakeConnection("deploy@host") + with patch.object(renovate, "Connection", return_value=fake): + renovate.renovate_host("deploy@host", "~/flightdeck", "beszel") - self.assertEqual(calls, [(["ssh-add", "-"], "-----KEY-----")]) - - def test_raises_on_missing_secret(self): - with self.assertRaises(renovate.RenovateError): - renovate.load_ssh_key({}, "MISSING_SECRET") - - def test_raises_when_ssh_add_fails(self): - def fake_run(args, input=None, capture_output=None, text=None): - return SimpleNamespace(returncode=1, stderr="bad key") - - with self.assertRaises(renovate.RenovateError): - renovate.load_ssh_key({"KEY": "text"}, "KEY", run=fake_run) - - -class RenovateTargetTest(unittest.TestCase): - def test_renovates_every_host_without_touching_versions(self): - manifest = { - "hosts": ["deploy@app1.example.com", "deploy@app2.example.com"], - "path": "~/flightdeck", - "credentials": {"secrets": {"ssh_private_key": "DEPLOY_SSH_PRIVATE_KEY"}}, - } - secrets = {"DEPLOY_SSH_PRIVATE_KEY": "-----KEY-----"} - fakes = {} - - def fake_connection(host): - fakes[host] = FakeConnection(host) - return fakes[host] - - ssh_calls = [] - - def fake_run(args, input=None, capture_output=None, text=None): - ssh_calls.append(input) - return SimpleNamespace(returncode=0, stderr="") - - with patch.object(renovate, "Connection", side_effect=fake_connection): - renovate.renovate_target("mainframe", manifest, "beszel", secrets, run=fake_run) - - self.assertEqual(ssh_calls, ["-----KEY-----"]) - for host in manifest["hosts"]: - joined = "\n".join(fakes[host].commands) - self.assertIn("cd /home/deploy/flightdeck/current/apps/beszel", joined) - self.assertIn("docker compose pull && docker compose up -d --remove-orphans", joined) + joined = "\n".join(fake.commands) + self.assertIn("cd /home/deploy/flightdeck/current/apps/beszel", joined) + self.assertIn("docker compose pull && docker compose up -d --remove-orphans", joined) class MainTest(unittest.TestCase): - def _run_main(self, directory, app, secrets): - stdin = io.StringIO(json.dumps({"app": app, "targets_directory": str(directory)})) - with ( - patch.object(sys, "stdin", stdin), - patch.dict(os.environ, {"GITHUB_SECRETS_JSON": json.dumps(secrets)}), - ): + def _run_main(self, config): + with patch.object(sys, "stdin", io.StringIO(json.dumps(config))): renovate.main() - def test_raises_when_app_matches_no_target(self): - with tempfile.TemporaryDirectory() as directory: - write_target(directory, "heimdall", "apps:\n traefik: {}\n") - - with self.assertRaises(renovate.RenovateError): - self._run_main(directory, "beszel", {}) - - def test_renovates_every_matching_target(self): - with tempfile.TemporaryDirectory() as directory: - write_target( - directory, - "heimdall", - "apps:\n beszel: {}\nhosts: [deploy@host]\ncredentials:\n secrets:\n ssh_private_key: KEY\n", + def test_skips_a_target_that_does_not_run_the_app(self): + with patch.object(renovate, "renovate_host") as fake_renovate_host: + self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"traefik": {}}}) + + fake_renovate_host.assert_not_called() + + def test_renovates_every_host_when_the_target_runs_the_app(self): + with patch.object(renovate, "renovate_host") as fake_renovate_host: + self._run_main( + { + "app": "beszel", + "hosts": ["deploy@app1.example.com", "deploy@app2.example.com"], + "path": "~/flightdeck", + "apps": {"beszel": {}}, + } ) - write_target(directory, "mainframe", "apps:\n rybbit: {}\nhosts: [deploy@other]\n") - with ( - patch.object(renovate, "renovate_target") as fake_renovate_target, - ): - self._run_main(directory, "beszel", {"KEY": "-----KEY-----"}) + self.assertEqual( + fake_renovate_host.call_args_list, + [ + unittest.mock.call("deploy@app1.example.com", "~/flightdeck", "beszel"), + unittest.mock.call("deploy@app2.example.com", "~/flightdeck", "beszel"), + ], + ) + + def test_defaults_path_when_omitted(self): + with patch.object(renovate, "renovate_host") as fake_renovate_host: + self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"beszel": {}}}) - fake_renovate_target.assert_called_once() - name, manifest, app, secrets = fake_renovate_target.call_args[0] - self.assertEqual((name, app), ("heimdall", "beszel")) + fake_renovate_host.assert_called_once_with("deploy@host", "~/flightdeck", "beszel") if __name__ == "__main__": From 980456a1ab5f6e5c9d9b855fb86152452bebcb6a Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Wed, 26 Aug 2026 01:09:47 +0200 Subject: [PATCH 04/22] feat: notify Telegram when renovate actually finds an updated image deploy/renovate.py now compares `docker compose images -q` before/after the pull per host, so it can tell an actual update from a no-op pull, and reports updated/updated_hosts via $GITHUB_OUTPUT. renovate-shared.yml sends a Telegram message (via baseline's generic send-telegram-message action, same one the existing notify-telegram-*.yml workflows use) only when a host's image changed - silent on every no-op target-matrix cell. --- .github/workflows/renovate-shared.yml | 8 ++++ README.md | 2 +- deploy/renovate.py | 30 +++++++++++-- deploy/tests/test_renovate.py | 62 +++++++++++++++++++-------- 4 files changed, 81 insertions(+), 21 deletions(-) diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml index 89a2535..26f1edf 100644 --- a/.github/workflows/renovate-shared.yml +++ b/.github/workflows/renovate-shared.yml @@ -50,6 +50,7 @@ jobs: echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" ssh-add - <<< "${{ secrets[matrix.credentials.secrets.ssh_private_key] }}" - name: Run renovate + id: run shell: bash env: APP: ${{ inputs.app }} @@ -60,3 +61,10 @@ jobs: jq -n --arg app "$APP" --argjson hosts "$HOSTS" --arg path "$TARGET_PATH" --argjson apps "$TARGET_APPS" \ '{app: $app, hosts: $hosts, path: $path, apps: $apps}' \ | python3 deploy/renovate.py + - name: Notify Telegram + if: steps.run.outputs.updated == 'true' + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Renovate: ${{ inputs.app }} updated on ${{ matrix.name }} (${{ steps.run.outputs.updated_hosts }})" + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/README.md b/README.md index 5b60221..125b1f2 100644 --- a/README.md +++ b/README.md @@ -373,7 +373,7 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl `deploy.yml` computes its matrix from `targets/` itself and calls `deploy-shared.yml` once per already-known target. `renovate.yml` can't do that — "find every target currently running this app" isn't answerable by the outer trigger before the job starts, since a matrix has to be fully resolved ahead of time and that's exactly the thing we don't know yet. So `renovate.yml` just passes `app`/`targets-directory` straight through with `secrets: inherit`, and `renovate-shared.yml` computes the matrix *itself*, in its own first job (`find-targets`, plain `load-yaml-matrix` over every manifest — deliberately not filtered by app, so that utility stays exactly as dumb and generic as it already is). Its second job (`renovate`) then fans out over that matrix exactly like `deploy-shared.yml` does — one job per target, with native per-matrix-cell secret/Tailscale resolution (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — except every target gets dispatched unconditionally, whether or not it actually runs the requested app. [`deploy/renovate.py`](deploy/renovate.py) itself makes that call: if `app` isn't a key in the target's own `apps` mapping, it just logs and no-ops instead of erroring, since a target-matrix fan-out has no other way to say "skip me." -Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. +Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), [`deploy/renovate.py`](deploy/renovate.py) compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts` via `$GITHUB_OUTPUT`. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed, including every target-matrix cell that no-ops because it doesn't run the app at all. **Known gaps:** diff --git a/deploy/renovate.py b/deploy/renovate.py index aa9e11c..fce41a3 100644 --- a/deploy/renovate.py +++ b/deploy/renovate.py @@ -17,8 +17,13 @@ every target regardless of whether it actually runs the requested app, `app` not being a key in it just means this target is a clean no-op, not an error. + +Writes `updated`/`updated_hosts` to $GITHUB_OUTPUT so the calling +workflow can notify only when a host's image actually changed, rather +than on every run. """ import json +import os import shlex import sys @@ -34,8 +39,21 @@ def renovate_host(host, base_path, app): connection = Connection(host) connection.client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) home = connection.run("echo $HOME", hide=True).stdout.strip() - compose_dir = f"{expand_home(base_path, home)}/current/apps/{app}" - connection.run(f"cd {shlex.quote(compose_dir)} && docker compose pull && docker compose up -d --remove-orphans") + compose_dir = shlex.quote(f"{expand_home(base_path, home)}/current/apps/{app}") + command = ( + f"cd {compose_dir} && before=$(docker compose images -q) && docker compose pull " + f'&& after=$(docker compose images -q) && docker compose up -d --remove-orphans ' + f'&& if [ "$before" != "$after" ]; then echo RENOVATE_UPDATED; fi' + ) + result = connection.run(command) + return "RENOVATE_UPDATED" in result.stdout + + +def write_github_output(name, value): + output_path = os.environ.get("GITHUB_OUTPUT") + if output_path: + with open(output_path, "a", encoding="utf-8") as output: + output.write(f"{name}={value}\n") def main(): @@ -44,12 +62,18 @@ def main(): apps = config.get("apps") or {} if app not in apps: print(f"{app!r} is not deployed on this target, skipping") + write_github_output("updated", "false") return base_path = config.get("path", "~/flightdeck") + updated_hosts = [] for host in config["hosts"]: print(f"Renovating {app} on {host}") - renovate_host(host, base_path, app) + if renovate_host(host, base_path, app): + updated_hosts.append(host) + + write_github_output("updated", "true" if updated_hosts else "false") + write_github_output("updated_hosts", ",".join(updated_hosts)) if __name__ == "__main__": diff --git a/deploy/tests/test_renovate.py b/deploy/tests/test_renovate.py index a065f56..4f00dd0 100644 --- a/deploy/tests/test_renovate.py +++ b/deploy/tests/test_renovate.py @@ -1,7 +1,9 @@ import importlib.util import io import json +import os import sys +import tempfile import unittest from pathlib import Path from types import SimpleNamespace @@ -20,16 +22,17 @@ class FakeConnection: """Stand-in for fabric.Connection - records commands instead of opening a real SSH session, mirroring test_deploy.py's FakeConnection.""" - def __init__(self, host): + def __init__(self, host, pull_stdout=""): self.host = host self.client = SimpleNamespace(set_missing_host_key_policy=lambda policy: None) self.commands = [] + self.pull_stdout = pull_stdout def run(self, command, hide=False): self.commands.append(command) if command == "echo $HOME": return SimpleNamespace(stdout="/home/deploy\n") - return SimpleNamespace(stdout="") + return SimpleNamespace(stdout=self.pull_stdout) class ExpandHomeTest(unittest.TestCase): @@ -48,23 +51,46 @@ def test_pulls_and_recreates_without_touching_the_release(self): joined = "\n".join(fake.commands) self.assertIn("cd /home/deploy/flightdeck/current/apps/beszel", joined) - self.assertIn("docker compose pull && docker compose up -d --remove-orphans", joined) + self.assertIn("before=$(docker compose images -q)", joined) + self.assertIn("docker compose pull", joined) + self.assertIn("after=$(docker compose images -q)", joined) + self.assertIn("docker compose up -d --remove-orphans", joined) + + def test_returns_true_when_the_image_changed(self): + fake = FakeConnection("deploy@host", pull_stdout="RENOVATE_UPDATED\n") + with patch.object(renovate, "Connection", return_value=fake): + self.assertTrue(renovate.renovate_host("deploy@host", "~/flightdeck", "beszel")) + + def test_returns_false_when_the_image_was_already_current(self): + fake = FakeConnection("deploy@host", pull_stdout="") + with patch.object(renovate, "Connection", return_value=fake): + self.assertFalse(renovate.renovate_host("deploy@host", "~/flightdeck", "beszel")) class MainTest(unittest.TestCase): def _run_main(self, config): - with patch.object(sys, "stdin", io.StringIO(json.dumps(config))): - renovate.main() + with tempfile.TemporaryDirectory() as directory: + output_path = Path(directory) / "outputs" + with ( + patch.object(sys, "stdin", io.StringIO(json.dumps(config))), + patch.dict(os.environ, {"GITHUB_OUTPUT": str(output_path)}), + ): + renovate.main() + return output_path.read_text() if output_path.exists() else "" def test_skips_a_target_that_does_not_run_the_app(self): with patch.object(renovate, "renovate_host") as fake_renovate_host: - self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"traefik": {}}}) + outputs = self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"traefik": {}}}) fake_renovate_host.assert_not_called() + self.assertIn("updated=false\n", outputs) - def test_renovates_every_host_when_the_target_runs_the_app(self): - with patch.object(renovate, "renovate_host") as fake_renovate_host: - self._run_main( + def test_reports_updated_hosts_when_the_image_changed(self): + def fake_renovate_host(host, base_path, app): + return host == "deploy@app1.example.com" + + with patch.object(renovate, "renovate_host", side_effect=fake_renovate_host): + outputs = self._run_main( { "app": "beszel", "hosts": ["deploy@app1.example.com", "deploy@app2.example.com"], @@ -73,16 +99,18 @@ def test_renovates_every_host_when_the_target_runs_the_app(self): } ) - self.assertEqual( - fake_renovate_host.call_args_list, - [ - unittest.mock.call("deploy@app1.example.com", "~/flightdeck", "beszel"), - unittest.mock.call("deploy@app2.example.com", "~/flightdeck", "beszel"), - ], - ) + self.assertIn("updated=true\n", outputs) + self.assertIn("updated_hosts=deploy@app1.example.com\n", outputs) + + def test_reports_not_updated_when_every_host_was_already_current(self): + with patch.object(renovate, "renovate_host", return_value=False): + outputs = self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"beszel": {}}}) + + self.assertIn("updated=false\n", outputs) + self.assertIn("updated_hosts=\n", outputs) def test_defaults_path_when_omitted(self): - with patch.object(renovate, "renovate_host") as fake_renovate_host: + with patch.object(renovate, "renovate_host", return_value=False) as fake_renovate_host: self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"beszel": {}}}) fake_renovate_host.assert_called_once_with("deploy@host", "~/flightdeck", "beszel") From 16ddbb6ec9a11edd278f880b6398b7f356d7074a Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Wed, 26 Aug 2026 11:08:04 +0200 Subject: [PATCH 05/22] refactor: move renovate's matrix and secret resolution out of secrets: inherit renovate.yml now computes the target matrix itself and resolves each matched target's secrets by name (secrets[matrix.credentials.secrets.X]), exactly like deploy.yml/deploy-shared.yml - renovate-shared.yml goes back to a single job with explicitly declared workflow_call secrets, so it only ever sees the one target's own resolved secret values, never the full secrets context. Dropped an earlier attempt to skip dispatching to a non-matching target via a matrix-conditioned job `if:` - actionlint (correctly) rejects `matrix` in the `if:` of a job that calls a reusable workflow via `uses:`, only github/inputs/needs/vars are available there. deploy/renovate.py's own `app in apps` check already makes a non-matching target a cheap no-op before any SSH connection happens, so this is a non-issue. --- .github/workflows/renovate-shared.yml | 78 ++++++++++++++++----------- .github/workflows/renovate.yml | 27 +++++++++- README.md | 35 +++++++----- 3 files changed, 94 insertions(+), 46 deletions(-) diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml index 26f1edf..bb85653 100644 --- a/.github/workflows/renovate-shared.yml +++ b/.github/workflows/renovate-shared.yml @@ -3,30 +3,48 @@ on: workflow_call: inputs: app: - description: App name to renovate (re-pull its image and recreate its containers) wherever it's currently deployed. + description: App name to renovate (re-pull its image and recreate its containers). type: string required: true - targets-directory: - description: Directory of target manifests to search for the app. + target-name: + description: Target name, used only to label the Telegram notification. type: string - default: targets + required: true + hosts: + description: JSON array of user@host SSH destinations to renovate. + type: string + required: true + apps: + description: JSON object of this target's own apps mapping, as declared in its targets/*.yml manifest. + type: string + required: true + path: + description: Base path on the target host for the current symlink. + type: string + default: "~/flightdeck" + tailscale-oauth-client-id: + description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). + type: string + default: "" + tailscale-tags: + description: Comma-separated ACL tags for the ephemeral tailnet node. + type: string + default: tag:ci + secrets: + ssh-private-key: + description: SSH private key used to connect to the hosts. + required: true + tailscale-oauth-secret: + description: Tailscale OAuth client secret used to join the tailnet. Required only when tailscale-oauth-client-id is set. + required: false + telegram-bot-token: + description: Telegram bot token used to notify when an update is found. + required: true + telegram-chat-id: + description: Telegram chat ID used to notify when an update is found. + required: true jobs: - find-targets: - runs-on: ubuntu-latest - outputs: - matrix: ${{ steps.matrix.outputs.matrix }} - count: ${{ steps.matrix.outputs.count }} - steps: - - uses: actions/checkout@v7 - - uses: $/.github/actions/load-yaml-matrix - id: matrix - with: - directory: ${{ inputs.targets-directory }} renovate: - needs: find-targets - if: needs.find-targets.outputs.count != '0' - strategy: - matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -37,26 +55,26 @@ jobs: shell: bash run: pip install --user --break-system-packages -r deploy/requirements.txt - uses: tailscale/github-action@v4 - if: matrix.credentials.variables.tailscale_oauth_client_id + if: inputs.tailscale-oauth-client-id != '' with: - oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} - oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} - tags: tag:ci + oauth-client-id: ${{ inputs.tailscale-oauth-client-id }} + oauth-secret: ${{ secrets.tailscale-oauth-secret }} + tags: ${{ inputs.tailscale-tags }} - name: Load SSH key shell: bash run: | eval "$(ssh-agent -s)" echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" - ssh-add - <<< "${{ secrets[matrix.credentials.secrets.ssh_private_key] }}" + ssh-add - <<< "${{ secrets.ssh-private-key }}" - name: Run renovate id: run shell: bash env: APP: ${{ inputs.app }} - HOSTS: ${{ toJson(matrix.hosts) }} - TARGET_PATH: ${{ matrix.path || '~/flightdeck' }} - TARGET_APPS: ${{ toJson(matrix.apps) }} + HOSTS: ${{ inputs.hosts }} + TARGET_PATH: ${{ inputs.path }} + TARGET_APPS: ${{ inputs.apps }} run: | jq -n --arg app "$APP" --argjson hosts "$HOSTS" --arg path "$TARGET_PATH" --argjson apps "$TARGET_APPS" \ '{app: $app, hosts: $hosts, path: $path, apps: $apps}' \ @@ -65,6 +83,6 @@ jobs: if: steps.run.outputs.updated == 'true' uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 with: - message: "Renovate: ${{ inputs.app }} updated on ${{ matrix.name }} (${{ steps.run.outputs.updated_hosts }})" - telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} - telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} + message: "Renovate: ${{ inputs.app }} updated on ${{ inputs.target-name }} (${{ steps.run.outputs.updated_hosts }})" + telegram-bot-token: ${{ secrets.telegram-bot-token }} + telegram-chat-id: ${{ secrets.telegram-chat-id }} diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index 6762f7b..f582ed5 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -11,9 +11,32 @@ on: type: string default: targets jobs: + find-targets: + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.matrix.outputs.matrix }} + count: ${{ steps.matrix.outputs.count }} + steps: + - uses: actions/checkout@v7 + - uses: $/.github/actions/load-yaml-matrix + id: matrix + with: + directory: ${{ inputs.targets-directory }} renovate: + needs: find-targets + if: needs.find-targets.outputs.count != '0' + strategy: + matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} uses: $/.github/workflows/renovate-shared.yml with: app: ${{ inputs.app }} - targets-directory: ${{ inputs.targets-directory }} - secrets: inherit + target-name: ${{ matrix.name }} + hosts: ${{ toJson(matrix.hosts) }} + apps: ${{ toJson(matrix.apps) }} + path: ${{ matrix.path || '~/flightdeck' }} + tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} + secrets: + ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} + tailscale-oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/README.md b/README.md index 125b1f2..d6480de 100644 --- a/README.md +++ b/README.md @@ -73,8 +73,8 @@ flightdeck/ │ │ └── load-yaml-matrix/ # Read a directory of YAML manifests into a workflow matrix │ └── workflows/ │ ├── deploy-shared.yml # Reusable deployment workflow -│ ├── renovate.yml # Prototype: thin workflow_dispatch trigger for renovate-shared.yml -│ ├── renovate-shared.yml # Prototype: computes its own target matrix, then fans out like deploy-shared.yml +│ ├── renovate.yml # Prototype: computes a target matrix and calls renovate-shared.yml per target, like deploy.yml +│ ├── renovate-shared.yml # Prototype: reusable single-target renovation workflow, like deploy-shared.yml │ └── release.yml # Release Please + publish Flightdeck assets │ ├── vaults/ # Encrypted env asset configurations, one per app @@ -369,27 +369,34 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl ### `renovate.yml` / `renovate-shared.yml` (prototype) -**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-file split as deploy (a thin `workflow_dispatch` trigger calling a reusable `workflow_call` workflow), and now the *same* matrix mechanics too — the difference that's actually being compared is narrower than it first looks: where the target matrix gets computed. +**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** At this point the two have converged onto the same shape: `renovate.yml` computes a matrix from `targets/` itself (plain `load-yaml-matrix`, unfiltered — deliberately not taught to filter by app, so that utility stays exactly as dumb and generic as it already is) and calls `renovate-shared.yml` once per target, with that target's secrets already picked out by name (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`: resolving each matrix cell's secret *value* by name happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`/`vars.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. -`deploy.yml` computes its matrix from `targets/` itself and calls `deploy-shared.yml` once per already-known target. `renovate.yml` can't do that — "find every target currently running this app" isn't answerable by the outer trigger before the job starts, since a matrix has to be fully resolved ahead of time and that's exactly the thing we don't know yet. So `renovate.yml` just passes `app`/`targets-directory` straight through with `secrets: inherit`, and `renovate-shared.yml` computes the matrix *itself*, in its own first job (`find-targets`, plain `load-yaml-matrix` over every manifest — deliberately not filtered by app, so that utility stays exactly as dumb and generic as it already is). Its second job (`renovate`) then fans out over that matrix exactly like `deploy-shared.yml` does — one job per target, with native per-matrix-cell secret/Tailscale resolution (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — except every target gets dispatched unconditionally, whether or not it actually runs the requested app. [`deploy/renovate.py`](deploy/renovate.py) itself makes that call: if `app` isn't a key in the target's own `apps` mapping, it just logs and no-ops instead of erroring, since a target-matrix fan-out has no other way to say "skip me." +The one real difference from deploy: not every target runs every app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: if `app` isn't a key in the target's own `apps` mapping, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just a checkout, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. -Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), [`deploy/renovate.py`](deploy/renovate.py) compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts` via `$GITHUB_OUTPUT`. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed, including every target-matrix cell that no-ops because it doesn't run the app at all. +Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), `deploy/renovate.py` compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts` via `$GITHUB_OUTPUT`. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. -**Known gaps:** - -- If `app` matches no target at all (a typo, say), every matrix job just no-ops and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. -- `find-targets` checks out its caller's own repository (to read `targets/`), while `renovate` checks out flightdeck's own ref (to get `deploy/renovate.py`, like `deploy-shared.yml` does) — so genuine cross-repo reuse (a private consumer repo calling `rubykatzen/flightdeck/.github/workflows/renovate-shared.yml@vX` for its own `targets/`) should already work, but hasn't been exercised outside this repo yet. - -Deferred until this contract shape is the chosen one. +**Known gap:** if `app` matches no target at all (a typo, say), every matrix job is just skipped and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. Deferred until this contract shape is the chosen one. ```yaml jobs: renovate: + needs: find-targets + if: needs.find-targets.outputs.count != '0' + strategy: + matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} uses: $/.github/workflows/renovate-shared.yml with: - app: beszel # required - targets-directory: targets # optional, default shown - secrets: inherit + app: ${{ inputs.app }} + target-name: ${{ matrix.name }} + hosts: ${{ toJson(matrix.hosts) }} + apps: ${{ toJson(matrix.apps) }} + path: ${{ matrix.path || '~/flightdeck' }} + tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} + secrets: + ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} + tailscale-oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} ``` ## License From c871b2fad3349d4e421e06f1bc836114e2316b9f Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Thu, 27 Aug 2026 22:29:52 +0200 Subject: [PATCH 06/22] refactor: pass renovate a target manifest path instead of flattened fields renovate.yml no longer exposes targets-directory as a dispatch input (hardcoded to "targets", matching deploy.yml's own convention) and its find-targets step id is now load-targets instead of the generic "matrix". The bigger change: instead of flattening a target's hosts/apps/path into separate with: inputs (each re-serialized via toJson(matrix.X)), renovate.yml now just passes target-manifest: ${{ matrix.manifest }} - the file path load-yaml-matrix already puts in every matrix item - and renovate-shared.yml reads that file itself, the same way encrypt-env takes a vault manifest path instead of flattened env fields. This drops the with: block from 6 fields to 3 (app, target-manifest, tailscale client id); the secrets: block is unchanged since secret values can only ever come from an explicit named pass-through, never a re-read file. renovate-shared.yml gets a second checkout (the caller's own repo, to read that manifest) alongside the existing override checkout (flightdeck's own code, now under .flightdeck/). deploy/renovate.py now takes {app, target_manifest} on stdin and parses the YAML itself instead of {app, hosts, path, apps} - it also derives and outputs target_name from the manifest's filename, so the Telegram message no longer needs a separate target-name input either. --- .github/workflows/renovate-shared.yml | 32 ++++--------- .github/workflows/renovate.yml | 19 +++----- README.md | 15 +++--- deploy/renovate.py | 60 +++++++++++++---------- deploy/tests/test_renovate.py | 69 +++++++++++++++++---------- 5 files changed, 103 insertions(+), 92 deletions(-) diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml index bb85653..f67908c 100644 --- a/.github/workflows/renovate-shared.yml +++ b/.github/workflows/renovate-shared.yml @@ -6,22 +6,10 @@ on: description: App name to renovate (re-pull its image and recreate its containers). type: string required: true - target-name: - description: Target name, used only to label the Telegram notification. + target-manifest: + description: Path to this target's manifest file in the calling repository's own targets/ directory, e.g. targets/heimdall.yml. type: string required: true - hosts: - description: JSON array of user@host SSH destinations to renovate. - type: string - required: true - apps: - description: JSON object of this target's own apps mapping, as declared in its targets/*.yml manifest. - type: string - required: true - path: - description: Base path on the target host for the current symlink. - type: string - default: "~/flightdeck" tailscale-oauth-client-id: description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). type: string @@ -47,13 +35,15 @@ jobs: renovate: runs-on: ubuntu-latest steps: + - uses: actions/checkout@v7 - uses: actions/checkout@v7 with: repository: ${{ job.workflow_repository }} ref: ${{ job.workflow_sha }} + path: .flightdeck - name: Install deploy dependencies shell: bash - run: pip install --user --break-system-packages -r deploy/requirements.txt + run: pip install --user --break-system-packages -r .flightdeck/deploy/requirements.txt - uses: tailscale/github-action@v4 if: inputs.tailscale-oauth-client-id != '' with: @@ -72,17 +62,15 @@ jobs: shell: bash env: APP: ${{ inputs.app }} - HOSTS: ${{ inputs.hosts }} - TARGET_PATH: ${{ inputs.path }} - TARGET_APPS: ${{ inputs.apps }} + TARGET_MANIFEST: ${{ inputs.target-manifest }} run: | - jq -n --arg app "$APP" --argjson hosts "$HOSTS" --arg path "$TARGET_PATH" --argjson apps "$TARGET_APPS" \ - '{app: $app, hosts: $hosts, path: $path, apps: $apps}' \ - | python3 deploy/renovate.py + jq -n --arg app "$APP" --arg target_manifest "$TARGET_MANIFEST" \ + '{app: $app, target_manifest: $target_manifest}' \ + | python3 .flightdeck/deploy/renovate.py - name: Notify Telegram if: steps.run.outputs.updated == 'true' uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 with: - message: "Renovate: ${{ inputs.app }} updated on ${{ inputs.target-name }} (${{ steps.run.outputs.updated_hosts }})" + message: "Renovate: ${{ inputs.app }} updated on ${{ steps.run.outputs.target_name }} (${{ steps.run.outputs.updated_hosts }})" telegram-bot-token: ${{ secrets.telegram-bot-token }} telegram-chat-id: ${{ secrets.telegram-chat-id }} diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index f582ed5..5dba28e 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -6,22 +6,18 @@ on: description: App name to renovate (re-pull its image and recreate its containers) wherever it's currently deployed. type: string required: true - targets-directory: - description: Directory of target manifests to search for the app. - type: string - default: targets jobs: find-targets: runs-on: ubuntu-latest outputs: - matrix: ${{ steps.matrix.outputs.matrix }} - count: ${{ steps.matrix.outputs.count }} + matrix: ${{ steps.load-targets.outputs.matrix }} + count: ${{ steps.load-targets.outputs.count }} steps: - uses: actions/checkout@v7 - uses: $/.github/actions/load-yaml-matrix - id: matrix + id: load-targets with: - directory: ${{ inputs.targets-directory }} + directory: targets renovate: needs: find-targets if: needs.find-targets.outputs.count != '0' @@ -30,13 +26,12 @@ jobs: uses: $/.github/workflows/renovate-shared.yml with: app: ${{ inputs.app }} - target-name: ${{ matrix.name }} - hosts: ${{ toJson(matrix.hosts) }} - apps: ${{ toJson(matrix.apps) }} - path: ${{ matrix.path || '~/flightdeck' }} + target-manifest: ${{ matrix.manifest }} tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} secrets: + # Per-target, resolved by name from this target's own manifest. ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} tailscale-oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} + # Fixed - the same for every target, not resolved from the manifest. telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/README.md b/README.md index d6480de..98de8c4 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ flightdeck/ │ ├── deploy/ │ ├── deploy.py # Push-based deploy entrypoint (runs on the CI runner) -│ ├── renovate.py # Prototype: re-pull/recreate one app's containers on one already-resolved target +│ ├── renovate.py # Prototype: re-pull/recreate one app's containers, reading its target manifest directly │ ├── resolve.py # owner/repo@tag[:asset] release ref resolution/download │ ├── collisions.py # Ciphertext-based env key collision detection │ ├── vault.py # SOPS decryption @@ -369,11 +369,13 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl ### `renovate.yml` / `renovate-shared.yml` (prototype) -**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** At this point the two have converged onto the same shape: `renovate.yml` computes a matrix from `targets/` itself (plain `load-yaml-matrix`, unfiltered — deliberately not taught to filter by app, so that utility stays exactly as dumb and generic as it already is) and calls `renovate-shared.yml` once per target, with that target's secrets already picked out by name (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`: resolving each matrix cell's secret *value* by name happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`/`vars.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. +**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-job matrix shape as deploy: `renovate.yml` computes a matrix from `targets/` itself (plain `load-yaml-matrix`, unfiltered — deliberately not taught to filter by app, so that utility stays exactly as dumb and generic as it already is) and calls `renovate-shared.yml` once per target, with that target's secrets already picked out by name (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`, and deliberately so: it [only works within the same organization or enterprise as the reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows), which would silently break for exactly the external, unrelated callers this is meant to support (confirmed real case: `dupmachine/flightdeck`). Resolving each matrix cell's secret *value* by name instead happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. -The one real difference from deploy: not every target runs every app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: if `app` isn't a key in the target's own `apps` mapping, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just a checkout, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. +Unlike `deploy-shared.yml`, which receives a target's `hosts`/`apps`/`path` already flattened into separate inputs, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-yaml-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields. This keeps the wired interface down to `app` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in). -Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), `deploy/renovate.py` compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts` via `$GITHUB_OUTPUT`. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. +Not every target runs every app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: if `app` isn't a key in the target manifest's own `apps` mapping, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. + +Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), `deploy/renovate.py` compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts`/`target_name` (derived from the manifest's own filename) via `$GITHUB_OUTPUT`. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. **Known gap:** if `app` matches no target at all (a typo, say), every matrix job is just skipped and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. Deferred until this contract shape is the chosen one. @@ -387,10 +389,7 @@ jobs: uses: $/.github/workflows/renovate-shared.yml with: app: ${{ inputs.app }} - target-name: ${{ matrix.name }} - hosts: ${{ toJson(matrix.hosts) }} - apps: ${{ toJson(matrix.apps) }} - path: ${{ matrix.path || '~/flightdeck' }} + target-manifest: ${{ matrix.manifest }} tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} secrets: ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} diff --git a/deploy/renovate.py b/deploy/renovate.py index fce41a3..2d8109d 100644 --- a/deploy/renovate.py +++ b/deploy/renovate.py @@ -1,33 +1,36 @@ #!/usr/bin/env python3 -"""Renovate: re-pull and recreate one app's containers on one already- -resolved target, without touching versions - no new app bundle, no new -vault-sourced env, no rebuilt release tree. Just `docker compose pull && -docker compose up -d` against that app's already-current release on each -of the target's hosts. - -One invocation is scoped to one target, same as deploy/deploy.py - the -matrix fan-out across every target under a directory, and per-target -secret/Tailscale resolution, live in renovate-shared.yml's own two-job -matrix (see README's "Renovate" section), not here. This script doesn't -know about targets/ or GitHub Secrets at all. - -Reads a JSON config from stdin: {"app": "", "hosts": [...], "path": -"...", "apps": {...}}. `apps` is this one target's own `apps` mapping, as -declared in its targets/*.yml manifest - since the matrix fans out to -every target regardless of whether it actually runs the requested app, -`app` not being a key in it just means this target is a clean no-op, not -an error. - -Writes `updated`/`updated_hosts` to $GITHUB_OUTPUT so the calling -workflow can notify only when a host's image actually changed, rather -than on every run. +"""Renovate: re-pull and recreate one app's containers on one target, read +directly from that target's own targets/*.yml manifest, without touching +versions - no new app bundle, no new vault-sourced env, no rebuilt release +tree. Just `docker compose pull && docker compose up -d` against that +app's already-current release on each of the target's hosts. + +Unlike deploy/deploy.py, which receives an already-flattened config, this +takes a manifest *path* and parses it itself - the same shape encrypt-env's +render-env.py already uses for vault manifests, rather than flattening a +target's hosts/apps/path into separate inputs the caller has to build. +renovate-shared.yml's own two-job matrix (see README's "Renovate" section) +is what finds every target and resolves its secrets; this script never +reads a directory or touches GitHub Secrets itself. + +Reads a JSON config from stdin: {"app": "", "target_manifest": +"targets/heimdall.yml"}. If `app` isn't a key in that manifest's own +`apps` mapping, this is a clean no-op, not an error - a target-matrix +fan-out dispatches to every target regardless of whether it actually +runs the requested app. + +Writes `updated`/`updated_hosts`/`target_name` to $GITHUB_OUTPUT so the +calling workflow can notify only when a host's image actually changed, +rather than on every run. """ import json import os import shlex import sys +from pathlib import Path import paramiko +import yaml from fabric import Connection @@ -59,15 +62,20 @@ def write_github_output(name, value): def main(): config = json.load(sys.stdin) app = config["app"] - apps = config.get("apps") or {} + manifest_path = Path(config["target_manifest"]) + target_name = manifest_path.stem + write_github_output("target_name", target_name) + + target = yaml.safe_load(manifest_path.read_text()) + apps = target.get("apps") or {} if app not in apps: - print(f"{app!r} is not deployed on this target, skipping") + print(f"{app!r} is not deployed on target {target_name!r}, skipping") write_github_output("updated", "false") return - base_path = config.get("path", "~/flightdeck") + base_path = target.get("path", "~/flightdeck") updated_hosts = [] - for host in config["hosts"]: + for host in target["hosts"]: print(f"Renovating {app} on {host}") if renovate_host(host, base_path, app): updated_hosts.append(host) diff --git a/deploy/tests/test_renovate.py b/deploy/tests/test_renovate.py index 4f00dd0..ecf7d1f 100644 --- a/deploy/tests/test_renovate.py +++ b/deploy/tests/test_renovate.py @@ -68,53 +68,74 @@ def test_returns_false_when_the_image_was_already_current(self): class MainTest(unittest.TestCase): - def _run_main(self, config): - with tempfile.TemporaryDirectory() as directory: - output_path = Path(directory) / "outputs" - with ( - patch.object(sys, "stdin", io.StringIO(json.dumps(config))), - patch.dict(os.environ, {"GITHUB_OUTPUT": str(output_path)}), - ): - renovate.main() - return output_path.read_text() if output_path.exists() else "" + def _run_main(self, directory, app, manifest_text, manifest_name="heimdall"): + manifest_path = Path(directory) / f"{manifest_name}.yml" + manifest_path.write_text(manifest_text) + output_path = Path(directory) / "outputs" + + config = {"app": app, "target_manifest": str(manifest_path)} + with ( + patch.object(sys, "stdin", io.StringIO(json.dumps(config))), + patch.dict(os.environ, {"GITHUB_OUTPUT": str(output_path)}), + ): + renovate.main() + return output_path.read_text() if output_path.exists() else "" def test_skips_a_target_that_does_not_run_the_app(self): - with patch.object(renovate, "renovate_host") as fake_renovate_host: - outputs = self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"traefik": {}}}) + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host") as fake_renovate_host, + ): + outputs = self._run_main(directory, "beszel", "apps:\n traefik: {}\nhosts: [deploy@host]\n") fake_renovate_host.assert_not_called() self.assertIn("updated=false\n", outputs) + self.assertIn("target_name=heimdall\n", outputs) def test_reports_updated_hosts_when_the_image_changed(self): def fake_renovate_host(host, base_path, app): return host == "deploy@app1.example.com" - with patch.object(renovate, "renovate_host", side_effect=fake_renovate_host): - outputs = self._run_main( - { - "app": "beszel", - "hosts": ["deploy@app1.example.com", "deploy@app2.example.com"], - "path": "~/flightdeck", - "apps": {"beszel": {}}, - } - ) + manifest = "apps:\n beszel: {}\nhosts: [deploy@app1.example.com, deploy@app2.example.com]\npath: ~/flightdeck\n" + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host", side_effect=fake_renovate_host), + ): + outputs = self._run_main(directory, "beszel", manifest) self.assertIn("updated=true\n", outputs) self.assertIn("updated_hosts=deploy@app1.example.com\n", outputs) def test_reports_not_updated_when_every_host_was_already_current(self): - with patch.object(renovate, "renovate_host", return_value=False): - outputs = self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"beszel": {}}}) + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host", return_value=False), + ): + outputs = self._run_main(directory, "beszel", "apps:\n beszel: {}\nhosts: [deploy@host]\n") self.assertIn("updated=false\n", outputs) self.assertIn("updated_hosts=\n", outputs) def test_defaults_path_when_omitted(self): - with patch.object(renovate, "renovate_host", return_value=False) as fake_renovate_host: - self._run_main({"app": "beszel", "hosts": ["deploy@host"], "apps": {"beszel": {}}}) + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host", return_value=False) as fake_renovate_host, + ): + self._run_main(directory, "beszel", "apps:\n beszel: {}\nhosts: [deploy@host]\n") fake_renovate_host.assert_called_once_with("deploy@host", "~/flightdeck", "beszel") + def test_derives_target_name_from_manifest_filename(self): + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host", return_value=False), + ): + outputs = self._run_main( + directory, "beszel", "apps:\n beszel: {}\nhosts: [deploy@host]\n", manifest_name="mainframe" + ) + + self.assertIn("target_name=mainframe\n", outputs) + if __name__ == "__main__": unittest.main() From aeb7957c32c856bb2e6fd2a188119396f0fa5cff Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Thu, 27 Aug 2026 23:42:13 +0200 Subject: [PATCH 07/22] refactor!: pass deploy a target manifest path instead of flattened fields Apply the same simplification renovate-shared.yml just adopted, now that the only real external caller (dupmachine/flightdeck) is also ours to update: deploy-shared.yml's public contract changes from flattened hosts/app-refs/apps/path/keep-releases inputs to a single target-manifest path, read and parsed by the workflow itself - the same shape encrypt-env already uses for vault manifests. Adds a second checkout (the caller's own repo, to read that file) alongside the existing override checkout of flightdeck's own code, now under .flightdeck/. Also adds target-manifest-ref: release.yml pins deploy-targets' own checkout to the just-published tag, so deploy-shared.yml's separate read of the same manifest file needs to agree on that exact ref rather than silently reading whatever the default branch's tip is by the time it runs. release.yml's deploy job now also lists release in its own needs:, since referencing needs.release.outputs requires a direct (not just transitive) dependency. deploy.py's main() now reads {target_manifest, sops_age_key} from stdin and parses the YAML itself via a new load_target() - validate_config and everything downstream is unchanged, since they already just consumed a plain dict shaped this way regardless of where it came from. deploy.yml also drops the "redeploy every target" (target: all) mode - target is now a required input, no default - and renames its find-targets step id from the generic "matrix" to "load-targets", matching renovate.yml's own naming. --- .github/workflows/deploy-shared.yml | 46 ++++++------------- .github/workflows/deploy.yml | 18 +++----- .github/workflows/release.yml | 15 +++---- README.md | 21 +++++---- deploy/deploy.py | 18 ++++++-- deploy/tests/test_deploy.py | 69 +++++++++++++++++++++++++++++ 6 files changed, 121 insertions(+), 66 deletions(-) diff --git a/.github/workflows/deploy-shared.yml b/.github/workflows/deploy-shared.yml index c2e84a9..a959180 100644 --- a/.github/workflows/deploy-shared.yml +++ b/.github/workflows/deploy-shared.yml @@ -2,26 +2,14 @@ name: Deploy (shared) on: workflow_call: inputs: - hosts: - description: JSON array of user@host SSH destinations to deploy to. + target-manifest: + description: Path to this target's manifest file in the calling repository's own targets/ directory, e.g. targets/heimdall.yml. type: string required: true - app-refs: - description: JSON array of release refs for app bundles to merge into the release, in owner/repo@tag[:asset] format. + target-manifest-ref: + description: Git ref to check out this repository at when reading target-manifest. Leave unset to use the default branch (e.g. set this to a release tag when the caller itself pinned its own checkout to one, so both reads agree). type: string - required: true - apps: - description: JSON object mapping each app name to run on this target to its own env_refs list, e.g. {"traefik":{"env_refs":["owner/repo@latest:traefik.sops.env"]}}. - type: string - required: true - path: - description: Base path on the target host for releases, shared files, and the current symlink. - type: string - default: "~/flightdeck" - keep-releases: - description: Number of past releases to keep on the target host. - type: number - default: 5 + default: "" tailscale-oauth-client-id: description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). type: string @@ -44,13 +32,17 @@ jobs: deploy: runs-on: ubuntu-latest steps: + - uses: actions/checkout@v7 + with: + ref: ${{ inputs.target-manifest-ref }} - uses: actions/checkout@v7 with: repository: ${{ job.workflow_repository }} ref: ${{ job.workflow_sha }} + path: .flightdeck - name: Install deploy dependencies shell: bash - run: pip install --user --break-system-packages -r deploy/requirements.txt + run: pip install --user --break-system-packages -r .flightdeck/deploy/requirements.txt - name: Install sops shell: bash env: @@ -76,20 +68,10 @@ jobs: - name: Run deploy shell: bash env: - HOSTS: ${{ inputs.hosts }} - APP_REFS: ${{ inputs.app-refs }} - APPS: ${{ inputs.apps }} - DEPLOY_PATH: ${{ inputs.path }} - KEEP_RELEASES: ${{ inputs.keep-releases }} + TARGET_MANIFEST: ${{ inputs.target-manifest }} SOPS_AGE_KEY: ${{ secrets.sops-age-key }} GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | - jq -n \ - --argjson hosts "$HOSTS" \ - --argjson app_refs "$APP_REFS" \ - --argjson apps "$APPS" \ - --arg path "$DEPLOY_PATH" \ - --argjson keep_releases "$KEEP_RELEASES" \ - --arg sops_age_key "$SOPS_AGE_KEY" \ - '{hosts: $hosts, app_refs: $app_refs, apps: $apps, path: $path, keep_releases: $keep_releases, sops_age_key: $sops_age_key}' \ - | python3 deploy/deploy.py + jq -n --arg target_manifest "$TARGET_MANIFEST" --arg sops_age_key "$SOPS_AGE_KEY" \ + '{target_manifest: $target_manifest, sops_age_key: $sops_age_key}' \ + | python3 .flightdeck/deploy/deploy.py diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index a605a96..ad63c83 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -3,22 +3,22 @@ on: workflow_dispatch: inputs: target: - description: Deploy target to redeploy. Use "all" to redeploy every target. + description: Deploy target to redeploy. type: string - default: all + required: true jobs: deploy-targets: runs-on: ubuntu-latest outputs: - matrix: ${{ steps.matrix.outputs.matrix }} - count: ${{ steps.matrix.outputs.count }} + matrix: ${{ steps.load-targets.outputs.matrix }} + count: ${{ steps.load-targets.outputs.count }} steps: - uses: actions/checkout@v7 - uses: $/.github/actions/load-yaml-matrix - id: matrix + id: load-targets with: directory: targets - name: ${{ inputs.target || 'all' }} + name: ${{ inputs.target }} deploy: needs: deploy-targets if: needs.deploy-targets.outputs.count != '0' @@ -26,11 +26,7 @@ jobs: matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} uses: $/.github/workflows/deploy-shared.yml with: - hosts: ${{ toJson(matrix.hosts) }} - app-refs: ${{ toJson(matrix.app_refs) }} - apps: ${{ toJson(matrix.apps) }} - path: ${{ matrix.path || '~/flightdeck' }} - keep-releases: ${{ matrix.keep_releases || 5 }} + target-manifest: ${{ matrix.manifest }} tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} secrets: ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9bf2817..989b7bb 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -67,28 +67,25 @@ jobs: if: needs.release.outputs.release_created == 'true' runs-on: ubuntu-latest outputs: - matrix: ${{ steps.matrix.outputs.matrix }} - count: ${{ steps.matrix.outputs.count }} + matrix: ${{ steps.load-targets.outputs.matrix }} + count: ${{ steps.load-targets.outputs.count }} steps: - uses: actions/checkout@v7 with: ref: ${{ needs.release.outputs.tag_name }} - uses: $/.github/actions/load-yaml-matrix - id: matrix + id: load-targets with: directory: targets deploy: - needs: [upload-apps, encrypt, deploy-targets] + needs: [release, upload-apps, encrypt, deploy-targets] if: needs.deploy-targets.outputs.count != '0' strategy: matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} uses: $/.github/workflows/deploy-shared.yml with: - hosts: ${{ toJson(matrix.hosts) }} - app-refs: ${{ toJson(matrix.app_refs) }} - apps: ${{ toJson(matrix.apps) }} - path: ${{ matrix.path || '~/flightdeck' }} - keep-releases: ${{ matrix.keep_releases || 5 }} + target-manifest: ${{ matrix.manifest }} + target-manifest-ref: ${{ needs.release.outputs.tag_name }} tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} secrets: ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} diff --git a/README.md b/README.md index 98de8c4..bf38960 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ A target server needs only: ## Automated Deploy -Deployment goes through [`deploy-shared.yml`](.github/workflows/deploy-shared.yml) (documented in the GitHub Actions section below), a reusable workflow wrapping [`deploy/deploy.py`](deploy/deploy.py) behind plain deploy vocabulary — `hosts`, `app-refs`, `apps`. +Deployment goes through [`deploy-shared.yml`](.github/workflows/deploy-shared.yml) (documented in the GitHub Actions section below), a reusable workflow wrapping [`deploy/deploy.py`](deploy/deploy.py) behind one input — `target-manifest`, a path to that target's own manifest file, which the workflow reads itself rather than receiving `hosts`/`app_refs`/`apps` already flattened. The deploy is push-based and runs entirely on the GitHub Actions runner: @@ -247,7 +247,7 @@ Credential fields contain GitHub Variable/Secret names, never credential values. A vault manifest's `env:` value is either `${NAME}` (a reference — look up the GitHub Secret/Variable named `NAME`) or a bare literal (any other value, used as-is with no lookup at all — see `DISABLE_SIGNUP: true` above). Use a literal for a value that's fixed for this target but isn't a secret and doesn't need a GitHub Secret/Variable to exist just to hold it. -`load-yaml-matrix` reads every file in `vaults/` or `targets/` into a matrix — it does not validate the manifest shape. Each manifest's fields are the responsibility of whatever consumes them: `encrypt-env` re-parses and validates its own manifest from `manifest`, and the workflows calling `deploy-shared.yml` apply `path`/`keep-releases` defaults and pull `credentials.secrets`/`credentials.variables` values directly from the matrix item. +`load-yaml-matrix` reads every file in `vaults/` or `targets/` into a matrix — it does not validate the manifest shape. Each manifest's fields are the responsibility of whatever consumes them: `encrypt-env` re-parses and validates its own manifest from `manifest`, and `deploy/deploy.py` re-parses and validates the target manifest itself (applying `path`/`keep_releases` defaults there) from the `target-manifest` path a matrix item's own `manifest` field already points to; the workflows calling `deploy-shared.yml` only pull `credentials.secrets`/`credentials.variables` values directly from the matrix item, to resolve actual secret/variable values by name. --- @@ -335,9 +335,11 @@ Requires `contents: write` permission on the calling job. `flightdeck-apps.zip` ### `deploy-shared.yml` -Runs [`deploy/deploy.py`](deploy/deploy.py) from this repository against the caller-supplied hosts. Intended to be called from a private consumer repository that owns both the config and secrets side (SSH key, encrypted `.sops.env` releases, the age private key, etc.) — this repository does not hold any deploy secrets itself. `apps..env_refs` entries typically reference that same calling repository via `${{ github.repository }}`, since it's both the config and secrets source. +Runs [`deploy/deploy.py`](deploy/deploy.py) from this repository against a target manifest owned by the caller. Intended to be called from a private consumer repository that owns both the config and secrets side (a `targets/*.yml` manifest shaped like the one above, the SSH key, encrypted `.sops.env` releases, the age private key, etc.) — this repository does not hold any deploy secrets itself. That manifest's `apps..env_refs` entries typically reference that same calling repository via `${{ github.repository }}`, since it's both the config and secrets source. -The interface is plain deploy vocabulary — callers never see `deploy.py`'s internals or hand-write its JSON config; the workflow builds that internally and pipes it to `python3 deploy/deploy.py` on stdin. The runner resolves and downloads every ref, decrypts and renders each app's env and config, merges the release, and pushes the finished result to each host over SSH — see "Automated Deploy" above for the full sequence. +The interface is a single path, not flattened deploy vocabulary — the caller never re-serializes its target's `hosts`/`app_refs`/`apps`/`path` through `toJson(...)`, and `deploy.py` never receives them as separate fields. `target-manifest` just points at the file (`targets/mainframe.yml` in the example below); the workflow checks out the caller's own repository to read it, parses and validates it itself, and pipes the result to `python3 deploy/deploy.py` on stdin alongside the one thing that genuinely can't live in that file - the decrypted `sops-age-key` secret value. The runner then resolves and downloads every ref, decrypts and renders each app's env and config, merges the release, and pushes the finished result to each host over SSH — see "Automated Deploy" above for the full sequence. + +`target-manifest-ref` matters only when the caller itself checked out something other than its default branch before computing the matrix this is called from (e.g. `release.yml` pins to the just-published release tag) - set it to that same ref so both reads agree on the manifest's exact content, instead of silently reading whatever the default branch's tip happens to be by the time this job runs. Tailscale is optional, not a dependency of this workflow: set `tailscale-oauth-client-id` (and the matching `tailscale-oauth-secret`) to have the runner join a tailnet as an ephemeral node before deploying. Leave both unset to skip that step entirely — e.g. when the job already runs on a self-hosted runner with network access to the hosts, or reaches them some other way. @@ -348,11 +350,8 @@ jobs: deploy: uses: rubykatzen/flightdeck/.github/workflows/deploy-shared.yml@v0.11.1 with: - hosts: '["deploy@app1.example.com", "deploy@app2.example.com"]' # required JSON array - app-refs: '["rubykatzen/flightdeck@latest"]' # required non-empty JSON array - apps: '{"traefik": {"env_refs": ["${{ github.repository }}@latest:mainframe-traefik.sops.env"]}}' # required non-empty JSON object - # path: ~/flightdeck # optional, default shown - # keep-releases: 5 # optional, default shown + target-manifest: targets/mainframe.yml # required, path in this repository + # target-manifest-ref: ${{ github.sha }} # optional, default: this repository's default branch tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) tailscale-tags: tag:ci # default: tag:ci secrets: @@ -363,7 +362,7 @@ jobs: -The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/deploy.py` itself. `app-refs` entries are separate and don't have to match the workflow pin. +The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/deploy.py` itself. `target-manifest`'s own `app_refs` entries are separate and don't have to match the workflow pin. --- @@ -371,7 +370,7 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl **Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-job matrix shape as deploy: `renovate.yml` computes a matrix from `targets/` itself (plain `load-yaml-matrix`, unfiltered — deliberately not taught to filter by app, so that utility stays exactly as dumb and generic as it already is) and calls `renovate-shared.yml` once per target, with that target's secrets already picked out by name (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`, and deliberately so: it [only works within the same organization or enterprise as the reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows), which would silently break for exactly the external, unrelated callers this is meant to support (confirmed real case: `dupmachine/flightdeck`). Resolving each matrix cell's secret *value* by name instead happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. -Unlike `deploy-shared.yml`, which receives a target's `hosts`/`apps`/`path` already flattened into separate inputs, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-yaml-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields. This keeps the wired interface down to `app` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in). +Like `deploy-shared.yml`, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-yaml-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields, and `deploy-shared.yml` now does the same for `hosts`/`apps`/`path`. This keeps the wired interface down to `app` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in) — same shape `deploy-shared.yml` uses. Not every target runs every app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: if `app` isn't a key in the target manifest's own `apps` mapping, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. diff --git a/deploy/deploy.py b/deploy/deploy.py index 4e62c8f..6dbaf96 100644 --- a/deploy/deploy.py +++ b/deploy/deploy.py @@ -6,8 +6,14 @@ nothing but Docker and Docker Compose: no sops, no age key, no gh, no flightdeck scripts of any kind. -Reads a JSON config from stdin (see README's "deploy-shared.yml" section -for the exact shape). +Reads a JSON config from stdin: {"target_manifest": "targets/heimdall.yml", +"sops_age_key": "..."}. `target_manifest` is a path to that target's own +manifest, parsed here rather than flattened into separate fields by the +caller - the same shape encrypt-env's render-env.py already uses for +vault manifests. `sops_age_key` can't come from that file, since it's a +secret value, not YAML-safe config - deploy-shared.yml passes it +separately (see README's "deploy-shared.yml" section for the exact +contract). """ import json import shlex @@ -192,6 +198,10 @@ def deploy_to_host(host, archive_path, apps, networks, config, release_name): connection.run("docker container prune -f && docker image prune -a -f") +def load_target(path): + return yaml.safe_load(Path(path).read_text()) + + def validate_config(config): if not config.get("hosts"): raise DeployError("Config must set hosts to a non-empty list") @@ -204,7 +214,9 @@ def validate_config(config): def main(): - config = json.load(sys.stdin) + stdin_config = json.load(sys.stdin) + config = load_target(stdin_config["target_manifest"]) + config["sops_age_key"] = stdin_config.get("sops_age_key") validate_config(config) with tempfile.TemporaryDirectory(prefix="flightdeck-deploy-") as raw_dir: work_dir = Path(raw_dir) diff --git a/deploy/tests/test_deploy.py b/deploy/tests/test_deploy.py index c129edc..5cfe501 100644 --- a/deploy/tests/test_deploy.py +++ b/deploy/tests/test_deploy.py @@ -1,4 +1,5 @@ import importlib.util +import io import json import sys import tarfile @@ -42,6 +43,22 @@ def make_zip(path, files): return path +class LoadTargetTest(unittest.TestCase): + def test_parses_a_target_manifest(self): + with tempfile.TemporaryDirectory() as directory: + manifest_path = Path(directory) / "heimdall.yml" + manifest_path.write_text( + "hosts: [user@host]\napp_refs: [owner/repo@latest]\n" + "apps:\n traefik:\n env_refs: [owner/repo@latest:a.sops.env]\n" + ) + + target = deploy.load_target(manifest_path) + + self.assertEqual(target["hosts"], ["user@host"]) + self.assertEqual(target["app_refs"], ["owner/repo@latest"]) + self.assertEqual(target["apps"]["traefik"]["env_refs"], ["owner/repo@latest:a.sops.env"]) + + class ValidateConfigTest(unittest.TestCase): VALID = { "hosts": ["user@host"], @@ -456,5 +473,57 @@ def test_no_previous_manifest_stops_nothing(self): self.assertNotIn("docker compose down", joined) +class MainTest(unittest.TestCase): + def test_reads_target_manifest_and_merges_sops_age_key(self): + with tempfile.TemporaryDirectory() as directory: + manifest_path = Path(directory) / "heimdall.yml" + manifest_path.write_text( + "hosts: [deploy@host]\n" + "app_refs: [owner/repo@latest]\n" + "apps:\n traefik:\n env_refs: [owner/repo@latest:a.sops.env]\n" + ) + stdin_config = {"target_manifest": str(manifest_path), "sops_age_key": "AGE-SECRET-KEY-1..."} + + with ( + patch.object(sys, "stdin", io.StringIO(json.dumps(stdin_config))), + patch.object( + deploy, "build_release", return_value=(Path(directory) / "release", ["owner/repo@v1.0.0"]) + ) as fake_build_release, + patch.object( + deploy, + "resolve_app_envs", + return_value={"traefik": ["owner/repo@v1.0.0:a.sops.env"]}, + ) as fake_resolve_app_envs, + patch.object(deploy, "write_release_manifest"), + patch.object(deploy, "archive_release", return_value=Path(directory) / "release.tar.gz"), + patch.object(deploy, "list_required_networks", return_value=[]), + patch.object(deploy, "deploy_to_host") as fake_deploy_to_host, + ): + deploy.main() + + config_arg = fake_build_release.call_args[0][0] + self.assertEqual(config_arg["hosts"], ["deploy@host"]) + self.assertEqual(config_arg["app_refs"], ["owner/repo@latest"]) + self.assertEqual(config_arg["sops_age_key"], "AGE-SECRET-KEY-1...") + + fake_resolve_app_envs.assert_called_once() + fake_deploy_to_host.assert_called_once() + host, archive_path, apps, networks, deploy_config = fake_deploy_to_host.call_args[0][:5] + expected = ("deploy@host", Path(directory) / "release.tar.gz", ["traefik"], [], config_arg) + self.assertEqual((host, archive_path, apps, networks, deploy_config), expected) + + def test_raises_when_sops_age_key_missing(self): + with tempfile.TemporaryDirectory() as directory: + manifest_path = Path(directory) / "heimdall.yml" + manifest_path.write_text("hosts: [deploy@host]\napp_refs: [owner/repo@latest]\napps:\n traefik: {}\n") + stdin_config = {"target_manifest": str(manifest_path), "sops_age_key": ""} + + with ( + patch.object(sys, "stdin", io.StringIO(json.dumps(stdin_config))), + self.assertRaises(deploy.DeployError), + ): + deploy.main() + + if __name__ == "__main__": unittest.main() From ce6863ca80b65b247d1e13579b54caa017efa1e2 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:04:08 +0200 Subject: [PATCH 08/22] refactor!: flatten target credentials, add validating load-targets-matrix Target manifests drop the credentials: wrapper entirely. Tailscale config moves out of targets/*.yml altogether - it's one shared tailnet, not per-target - so deploy.yml/release.yml/renovate.yml now reference vars.TAILSCALE_OAUTH_CLIENT_ID/secrets.TAILSCALE_OAUTH_SECRET directly instead of resolving a per-target name for the same fixed values. ssh_private_key/sops_age_key move to the manifest root, renamed to ssh_private_key_secret/sops_age_key_secret - the _secret suffix makes it unambiguous these are GitHub Secret names, not values, now that they're no longer nested under credentials.secrets. This also sidesteps a real naming collision in deploy.py, which already uses the bare key "sops_age_key" internally for the actual decrypted value. New .github/actions/load-targets-matrix, a specialized sibling to load-yaml-matrix used only for targets/: same directory+name matrix contract, but validates hosts/app_refs/apps/ssh_private_key_secret/ sops_age_key_secret before ever producing a matrix item, so a broken target manifest fails immediately with a clear error instead of surfacing deep inside deploy.py/renovate.py after a checkout and dependency install on a different job. load-yaml-matrix itself is untouched and keeps validating nothing - it's still the right tool for vaults/, where the caller never touches manifest fields directly and encrypt-env already validates the same content one job later anyway. deploy.yml/release.yml/renovate.yml switch their targets/ loading to load-targets-matrix (dropping directory: targets, now the default) and simplify their with:/secrets: blocks to the flat field names. --- .github/actions/load-targets-matrix/README.md | 19 +++ .../actions/load-targets-matrix/action.yml | 32 +++++ .../load-targets-matrix/requirements.txt | 1 + .../scripts/load-targets-matrix.py | 111 ++++++++++++++++ .../tests/test_load_targets_matrix.py | 122 ++++++++++++++++++ .github/workflows/deploy.yml | 11 +- .github/workflows/release.yml | 12 +- .github/workflows/renovate.yml | 10 +- README.md | 32 +++-- targets/heimdall.yml | 9 +- 10 files changed, 316 insertions(+), 43 deletions(-) create mode 100644 .github/actions/load-targets-matrix/README.md create mode 100644 .github/actions/load-targets-matrix/action.yml create mode 100644 .github/actions/load-targets-matrix/requirements.txt create mode 100644 .github/actions/load-targets-matrix/scripts/load-targets-matrix.py create mode 100644 .github/actions/load-targets-matrix/tests/test_load_targets_matrix.py diff --git a/.github/actions/load-targets-matrix/README.md b/.github/actions/load-targets-matrix/README.md new file mode 100644 index 0000000..6cce0c5 --- /dev/null +++ b/.github/actions/load-targets-matrix/README.md @@ -0,0 +1,19 @@ +# load-targets-matrix + +Composite GitHub Action that reads every target manifest in a directory into a GitHub Actions matrix, validating each one's required shape. Specialized counterpart to [`load-yaml-matrix`](../load-yaml-matrix) for `targets/*.yml` specifically — `load-yaml-matrix` stays generic and unvalidated for everything else (`vaults/*.yml`). + +## Usage + +```yaml +- uses: rubykatzen/flightdeck/.github/actions/load-targets-matrix@main + id: load-targets + with: + # directory: targets # optional, default shown + # name: all # optional; single target name to load, default: all +``` + +The action exposes `matrix`, containing `{ "include": [...] }`, and `count`. Each matrix item merges the manifest's own top-level YAML fields with `name` (the file's basename) and `manifest` (its path). + +Every manifest must set `hosts` (non-empty list), `app_refs` (non-empty list), `apps` (non-empty object), `ssh_private_key_secret` (non-empty string - the name of the GitHub Secret holding this target's SSH private key), and `sops_age_key_secret` (non-empty string - the name of the GitHub Secret holding this target's private SOPS age key). This fails the matrix build immediately, with an error naming the specific broken target, rather than letting a malformed manifest reach `deploy/deploy.py`/`deploy/renovate.py` much later, after a checkout and dependency install on a different job entirely. + +Files may use either the `.yml` or `.yaml` extension. Filenames must match `^[a-z0-9][a-z0-9-]*$` and be unique per directory; duplicate top-level YAML keys within a manifest are rejected. diff --git a/.github/actions/load-targets-matrix/action.yml b/.github/actions/load-targets-matrix/action.yml new file mode 100644 index 0000000..0e16bf5 --- /dev/null +++ b/.github/actions/load-targets-matrix/action.yml @@ -0,0 +1,32 @@ +name: Load targets matrix +description: Read every target manifest in a directory into a GitHub Actions matrix, validating each one's required shape. +inputs: + directory: + description: Directory containing target manifests. + required: false + default: targets + name: + description: Single target name to load, or all. + required: false + default: all +outputs: + matrix: + description: JSON strategy matrix, one item per target manifest, merging its parsed YAML fields with name and manifest. + value: ${{ steps.load.outputs.matrix }} + count: + description: Number of targets in the matrix. + value: ${{ steps.load.outputs.count }} +runs: + using: composite + steps: + - name: Install dependencies + shell: bash + run: python3 -m pip install --disable-pip-version-check --requirement "$GITHUB_ACTION_PATH/requirements.txt" + - name: Load matrix + id: load + shell: bash + env: + DIRECTORY: ${{ inputs.directory }} + NAME: ${{ inputs.name }} + run: | + python3 "$GITHUB_ACTION_PATH/scripts/load-targets-matrix.py" --directory "$DIRECTORY" --name "$NAME" diff --git a/.github/actions/load-targets-matrix/requirements.txt b/.github/actions/load-targets-matrix/requirements.txt new file mode 100644 index 0000000..8392d54 --- /dev/null +++ b/.github/actions/load-targets-matrix/requirements.txt @@ -0,0 +1 @@ +PyYAML==6.0.2 diff --git a/.github/actions/load-targets-matrix/scripts/load-targets-matrix.py b/.github/actions/load-targets-matrix/scripts/load-targets-matrix.py new file mode 100644 index 0000000..b89c63b --- /dev/null +++ b/.github/actions/load-targets-matrix/scripts/load-targets-matrix.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +import argparse +import json +import os +import re +import sys +from pathlib import Path + +import yaml + +NAME_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$") + + +class ManifestError(Exception): + pass + + +class UniqueKeyLoader(yaml.SafeLoader): + pass + + +def construct_mapping(loader, node, deep=False): + mapping = {} + for key_node, value_node in node.value: + key = loader.construct_object(key_node, deep=deep) + if key in mapping: + raise ManifestError(f"duplicate YAML key: {key}") + mapping[key] = loader.construct_object(value_node, deep=deep) + return mapping + + +UniqueKeyLoader.add_constructor( + yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, + construct_mapping, +) + + +def load_manifest(path): + try: + value = yaml.load(path.read_text(), Loader=UniqueKeyLoader) + except yaml.YAMLError as exc: + raise ManifestError(f"{path} is not valid YAML: {exc}") from exc + if not isinstance(value, dict): + raise ManifestError(f"{path} must contain a YAML mapping") + return value + + +def validate_target(name, manifest): + if not isinstance(manifest.get("hosts"), list) or not manifest["hosts"]: + raise ManifestError(f"target {name!r} must set hosts to a non-empty list") + if not isinstance(manifest.get("app_refs"), list) or not manifest["app_refs"]: + raise ManifestError(f"target {name!r} must set app_refs to a non-empty list") + if not isinstance(manifest.get("apps"), dict) or not manifest["apps"]: + raise ManifestError(f"target {name!r} must set apps to a non-empty object") + if not isinstance(manifest.get("ssh_private_key_secret"), str) or not manifest["ssh_private_key_secret"]: + raise ManifestError(f"target {name!r} must set ssh_private_key_secret to a non-empty string") + if not isinstance(manifest.get("sops_age_key_secret"), str) or not manifest["sops_age_key_secret"]: + raise ManifestError(f"target {name!r} must set sops_age_key_secret to a non-empty string") + + +def build_matrix(directory, selected="all"): + paths = sorted(directory.glob("*.yml")) + sorted(directory.glob("*.yaml")) + if not paths: + raise ManifestError(f"no manifests found in {directory}") + names = {path.stem for path in paths} + if selected != "all" and selected not in names: + raise ManifestError(f"unknown name: {selected}") + include = [] + seen_names = set() + for path in paths: + name = path.stem + if not NAME_RE.fullmatch(name): + raise ManifestError(f"invalid manifest filename: {path.name}") + if name in seen_names: + raise ManifestError(f"duplicate manifest name: {name}") + seen_names.add(name) + manifest = load_manifest(path) + validate_target(name, manifest) + if selected != "all" and name != selected: + continue + item = {"name": name, "manifest": str(path)} + item.update(manifest) + include.append(item) + return {"include": include} + + +def write_github_output(name, value): + output_path = os.environ.get("GITHUB_OUTPUT") + if output_path: + with open(output_path, "a", encoding="utf-8") as output: + output.write(f"{name}={value}\n") + + +def main(argv=None): + parser = argparse.ArgumentParser() + parser.add_argument("--directory", required=True, type=Path) + parser.add_argument("--name", default="all") + args = parser.parse_args(argv) + try: + matrix = build_matrix(args.directory, args.name) + encoded = json.dumps(matrix, separators=(",", ":")) + write_github_output("matrix", encoded) + write_github_output("count", len(matrix["include"])) + except ManifestError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/actions/load-targets-matrix/tests/test_load_targets_matrix.py b/.github/actions/load-targets-matrix/tests/test_load_targets_matrix.py new file mode 100644 index 0000000..c1b603c --- /dev/null +++ b/.github/actions/load-targets-matrix/tests/test_load_targets_matrix.py @@ -0,0 +1,122 @@ +import importlib.util +import tempfile +import unittest +from pathlib import Path + +MODULE_PATH = Path(__file__).resolve().parents[1] / "scripts" / "load-targets-matrix.py" +SPEC = importlib.util.spec_from_file_location("load_targets_matrix", MODULE_PATH) +load_targets_matrix = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(load_targets_matrix) + + +HEIMDALL = """\ +app_refs: [rubykatzen/flightdeck@latest] +apps: + traefik: {} +hosts: [root@100.75.53.13] +ssh_private_key_secret: DEPLOY_SSH_PRIVATE_KEY +sops_age_key_secret: HEIMDALL_AGE_PRIVATE_KEY +""" + +MAINFRAME = """\ +app_refs: [rubykatzen/flightdeck@latest, owner/extra-apps@latest] +apps: + rybbit: {} +hosts: [deploy@app1.example.com, deploy@app2.example.com] +path: ~/flightdeck +ssh_private_key_secret: DEPLOY_SSH_PRIVATE_KEY +sops_age_key_secret: MAINFRAME_AGE_PRIVATE_KEY +""" + + +class LoadTargetsMatrixTest(unittest.TestCase): + def setUp(self): + self.temporary_directory = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary_directory.cleanup) + self.directory = Path(self.temporary_directory.name) + (self.directory / "heimdall.yml").write_text(HEIMDALL) + (self.directory / "mainframe.yml").write_text(MAINFRAME) + + def test_builds_matrix_from_all_manifests(self): + matrix = load_targets_matrix.build_matrix(self.directory) + self.assertEqual( + sorted(item["name"] for item in matrix["include"]), + ["heimdall", "mainframe"], + ) + + def test_merges_manifest_fields_with_name_and_manifest(self): + item = load_targets_matrix.build_matrix(self.directory, "heimdall")["include"][0] + self.assertEqual(item["name"], "heimdall") + self.assertEqual(item["manifest"], str(self.directory / "heimdall.yml")) + self.assertEqual(item["hosts"], ["root@100.75.53.13"]) + self.assertEqual(item["ssh_private_key_secret"], "DEPLOY_SSH_PRIVATE_KEY") + self.assertEqual(item["sops_age_key_secret"], "HEIMDALL_AGE_PRIVATE_KEY") + + def test_filters_selected_manifest(self): + matrix = load_targets_matrix.build_matrix(self.directory, "heimdall") + self.assertEqual([item["name"] for item in matrix["include"]], ["heimdall"]) + + def test_rejects_unknown_name(self): + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "unknown name"): + load_targets_matrix.build_matrix(self.directory, "missing") + + def test_rejects_empty_directory(self): + empty = self.directory / "empty" + empty.mkdir() + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "no manifests found"): + load_targets_matrix.build_matrix(empty) + + def test_rejects_invalid_manifest_filename(self): + (self.directory / "Heimdall_Prod.yml").write_text(HEIMDALL) + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "invalid manifest filename"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_duplicate_manifest_name(self): + (self.directory / "heimdall.yaml").write_text(HEIMDALL) + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "duplicate manifest name"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_non_mapping_manifest(self): + (self.directory / "heimdall.yml").write_text("- one\n- two\n") + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "must contain a YAML mapping"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_duplicate_yaml_key(self): + (self.directory / "heimdall.yml").write_text("hosts: [one]\nhosts: [two]\n") + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "duplicate YAML key"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_missing_hosts(self): + (self.directory / "heimdall.yml").write_text(HEIMDALL.replace("hosts: [root@100.75.53.13]\n", "")) + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "must set hosts"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_missing_app_refs(self): + (self.directory / "heimdall.yml").write_text( + HEIMDALL.replace("app_refs: [rubykatzen/flightdeck@latest]\n", "") + ) + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "must set app_refs"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_missing_apps(self): + (self.directory / "heimdall.yml").write_text(HEIMDALL.replace("apps:\n traefik: {}\n", "")) + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "must set apps"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_missing_ssh_private_key_secret(self): + (self.directory / "heimdall.yml").write_text( + HEIMDALL.replace("ssh_private_key_secret: DEPLOY_SSH_PRIVATE_KEY\n", "") + ) + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "must set ssh_private_key_secret"): + load_targets_matrix.build_matrix(self.directory) + + def test_rejects_missing_sops_age_key_secret(self): + (self.directory / "heimdall.yml").write_text( + HEIMDALL.replace("sops_age_key_secret: HEIMDALL_AGE_PRIVATE_KEY\n", "") + ) + with self.assertRaisesRegex(load_targets_matrix.ManifestError, "must set sops_age_key_secret"): + load_targets_matrix.build_matrix(self.directory) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index ad63c83..c076af8 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -14,10 +14,9 @@ jobs: count: ${{ steps.load-targets.outputs.count }} steps: - uses: actions/checkout@v7 - - uses: $/.github/actions/load-yaml-matrix + - uses: $/.github/actions/load-targets-matrix id: load-targets with: - directory: targets name: ${{ inputs.target }} deploy: needs: deploy-targets @@ -27,8 +26,8 @@ jobs: uses: $/.github/workflows/deploy-shared.yml with: target-manifest: ${{ matrix.manifest }} - tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} + tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} secrets: - ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} - tailscale-oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} - sops-age-key: ${{ secrets[matrix.credentials.secrets.sops_age_key] }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} + sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 989b7bb..018568f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -73,10 +73,8 @@ jobs: - uses: actions/checkout@v7 with: ref: ${{ needs.release.outputs.tag_name }} - - uses: $/.github/actions/load-yaml-matrix + - uses: $/.github/actions/load-targets-matrix id: load-targets - with: - directory: targets deploy: needs: [release, upload-apps, encrypt, deploy-targets] if: needs.deploy-targets.outputs.count != '0' @@ -86,8 +84,8 @@ jobs: with: target-manifest: ${{ matrix.manifest }} target-manifest-ref: ${{ needs.release.outputs.tag_name }} - tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} + tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} secrets: - ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} - tailscale-oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} - sops-age-key: ${{ secrets[matrix.credentials.secrets.sops_age_key] }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} + sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index 5dba28e..2d79326 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -14,10 +14,8 @@ jobs: count: ${{ steps.load-targets.outputs.count }} steps: - uses: actions/checkout@v7 - - uses: $/.github/actions/load-yaml-matrix + - uses: $/.github/actions/load-targets-matrix id: load-targets - with: - directory: targets renovate: needs: find-targets if: needs.find-targets.outputs.count != '0' @@ -27,11 +25,11 @@ jobs: with: app: ${{ inputs.app }} target-manifest: ${{ matrix.manifest }} - tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} + tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} secrets: # Per-target, resolved by name from this target's own manifest. - ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} - tailscale-oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} # Fixed - the same for every target, not resolved from the manifest. + tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/README.md b/README.md index bf38960..66b7d8c 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ A target server needs only: - **Docker** >= 20.10 - **Docker Compose** >= 2.0 -- SSH access for the deploy key configured in that target's `credentials` +- SSH access for the deploy key configured in that target's `ssh_private_key_secret` ## Automated Deploy @@ -70,7 +70,8 @@ flightdeck/ │ │ ├── build-bundle/ # Build and upload a zip bundle from given paths │ │ ├── build-apps-bundle/ # Build and upload an apps/ catalog bundle │ │ ├── encrypt-env/ # Encrypt a target env and upload it to a release -│ │ └── load-yaml-matrix/ # Read a directory of YAML manifests into a workflow matrix +│ │ ├── load-yaml-matrix/ # Read a directory of YAML manifests into a workflow matrix (vaults/) +│ │ └── load-targets-matrix/ # Same, but for targets/ specifically - validates the required shape │ └── workflows/ │ ├── deploy-shared.yml # Reusable deployment workflow │ ├── renovate.yml # Prototype: computes a target matrix and calls renovate-shared.yml per target, like deploy.yml @@ -187,7 +188,7 @@ Useful as a source of ready-made Docker Compose definitions when adding a new ap ## GitHub Actions -This repository provides four composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, and `load-yaml-matrix`) and one reusable workflow, `deploy-shared.yml`. +This repository provides five composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, `load-yaml-matrix`, and `load-targets-matrix`) and one reusable workflow, `deploy-shared.yml`. --- @@ -234,20 +235,17 @@ hosts: - deploy@app1.example.com - deploy@app2.example.com path: ~/flightdeck # optional, default shown -credentials: - variables: - tailscale_oauth_client_id: TAILSCALE_OAUTH_CLIENT_ID - secrets: - ssh_private_key: DEPLOY_SSH_PRIVATE_KEY - tailscale_oauth_secret: TAILSCALE_OAUTH_SECRET - sops_age_key: MAINFRAME_AGE_PRIVATE_KEY +ssh_private_key_secret: DEPLOY_SSH_PRIVATE_KEY +sops_age_key_secret: MAINFRAME_AGE_PRIVATE_KEY ``` -Credential fields contain GitHub Variable/Secret names, never credential values. `app_refs` and `hosts` are YAML arrays; `apps` is a mapping from app name to that app's own `env_refs` array. Each host uses the SSH `user@host` format. `app_refs` must list at least one app bundle — flightdeck's own `apps/` catalog is just another entry, not implicit. `env_refs` is optional — omit it (or leave it `[]`) for an app that genuinely needs zero vault-sourced values (e.g. `beszel` above); it still gets a `.env` with `APP_NAME`/`DATA_DIR`, just no vault is fetched or decrypted for it. Don't create a vault manifest with an empty `env:` just to satisfy this field - there's nothing to encrypt, so there's nothing to gain from one. When `env_refs` is given, it must be non-empty; `deploy/deploy.py` decrypts and concatenates all of an app's sources into that app's own `.env` on the runner, failing loud on any key collision — but only within that one app's own sources. Two different apps' vaults sharing a key (e.g. both declaring `DOMAIN`) is expected, since each app gets a separate `.env`. `credentials.secrets.sops_age_key` names the GitHub Secret holding this target's *private* age key — the one used to decrypt its vaults, matching the public key in `keys/.pub` used to encrypt them. +`ssh_private_key_secret`/`sops_age_key_secret` are GitHub Secret *names*, never the credential values themselves - the `_secret` suffix says so explicitly, since a flat field like `sops_age_key` could otherwise read as the key material itself. `sops_age_key_secret` names the GitHub Secret holding this target's *private* age key - the one used to decrypt its vaults, matching the public key in `keys/.pub` used to encrypt them. Tailscale credentials live outside the target manifest entirely (`vars.TAILSCALE_OAUTH_CLIENT_ID`/`secrets.TAILSCALE_OAUTH_SECRET`, referenced directly by the workflows below) since the tailnet is shared infrastructure, not something that varies per target. + +`app_refs` and `hosts` are YAML arrays; `apps` is a mapping from app name to that app's own `env_refs` array. Each host uses the SSH `user@host` format. `app_refs` must list at least one app bundle — flightdeck's own `apps/` catalog is just another entry, not implicit. `env_refs` is optional — omit it (or leave it `[]`) for an app that genuinely needs zero vault-sourced values (e.g. `beszel` above); it still gets a `.env` with `APP_NAME`/`DATA_DIR`, just no vault is fetched or decrypted for it. Don't create a vault manifest with an empty `env:` just to satisfy this field - there's nothing to encrypt, so there's nothing to gain from one. When `env_refs` is given, it must be non-empty; `deploy/deploy.py` decrypts and concatenates all of an app's sources into that app's own `.env` on the runner, failing loud on any key collision — but only within that one app's own sources. Two different apps' vaults sharing a key (e.g. both declaring `DOMAIN`) is expected, since each app gets a separate `.env`. A vault manifest's `env:` value is either `${NAME}` (a reference — look up the GitHub Secret/Variable named `NAME`) or a bare literal (any other value, used as-is with no lookup at all — see `DISABLE_SIGNUP: true` above). Use a literal for a value that's fixed for this target but isn't a secret and doesn't need a GitHub Secret/Variable to exist just to hold it. -`load-yaml-matrix` reads every file in `vaults/` or `targets/` into a matrix — it does not validate the manifest shape. Each manifest's fields are the responsibility of whatever consumes them: `encrypt-env` re-parses and validates its own manifest from `manifest`, and `deploy/deploy.py` re-parses and validates the target manifest itself (applying `path`/`keep_releases` defaults there) from the `target-manifest` path a matrix item's own `manifest` field already points to; the workflows calling `deploy-shared.yml` only pull `credentials.secrets`/`credentials.variables` values directly from the matrix item, to resolve actual secret/variable values by name. +`load-yaml-matrix` reads every file in `vaults/` into a matrix — it does not validate the manifest shape; `encrypt-env` re-parses and validates its own manifest from `manifest` (see "`encrypt-env`" below). Targets go through the more specific [`load-targets-matrix`](.github/actions/load-targets-matrix) instead, which *does* validate the shape above (`hosts`, `app_refs`, `apps`, `ssh_private_key_secret`, `sops_age_key_secret` all required) before a broken manifest ever reaches a checkout+dependency-install on a different job entirely. The workflows calling `deploy-shared.yml`/`renovate-shared.yml` then pull `matrix.ssh_private_key_secret`/`matrix.sops_age_key_secret` directly, to resolve actual secret values by name. --- @@ -368,9 +366,9 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl ### `renovate.yml` / `renovate-shared.yml` (prototype) -**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-job matrix shape as deploy: `renovate.yml` computes a matrix from `targets/` itself (plain `load-yaml-matrix`, unfiltered — deliberately not taught to filter by app, so that utility stays exactly as dumb and generic as it already is) and calls `renovate-shared.yml` once per target, with that target's secrets already picked out by name (`secrets[matrix.credentials.secrets.ssh_private_key]`, etc.) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`, and deliberately so: it [only works within the same organization or enterprise as the reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows), which would silently break for exactly the external, unrelated callers this is meant to support (confirmed real case: `dupmachine/flightdeck`). Resolving each matrix cell's secret *value* by name instead happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. +**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-job matrix shape as deploy: `renovate.yml` computes a matrix from `targets/` itself (via [`load-targets-matrix`](.github/actions/load-targets-matrix)) and calls `renovate-shared.yml` once per target, with that target's SSH key already picked out by name (`secrets[matrix.ssh_private_key_secret]`) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`, and deliberately so: it [only works within the same organization or enterprise as the reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows), which would silently break for exactly the external, unrelated callers this is meant to support (confirmed real case: `dupmachine/flightdeck`). Resolving each matrix cell's secret *value* by name instead happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. Tailscale credentials aren't target-specific at all (one shared tailnet), so they're referenced directly (`vars.TAILSCALE_OAUTH_CLIENT_ID`/`secrets.TAILSCALE_OAUTH_SECRET`) rather than resolved per matrix cell. -Like `deploy-shared.yml`, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-yaml-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields, and `deploy-shared.yml` now does the same for `hosts`/`apps`/`path`. This keeps the wired interface down to `app` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in) — same shape `deploy-shared.yml` uses. +Like `deploy-shared.yml`, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-targets-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields, and `deploy-shared.yml` now does the same for `hosts`/`apps`/`path`. This keeps the wired interface down to `app` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in) — same shape `deploy-shared.yml` uses. Not every target runs every app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: if `app` isn't a key in the target manifest's own `apps` mapping, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. @@ -389,10 +387,10 @@ jobs: with: app: ${{ inputs.app }} target-manifest: ${{ matrix.manifest }} - tailscale-oauth-client-id: ${{ vars[matrix.credentials.variables.tailscale_oauth_client_id] }} + tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} secrets: - ssh-private-key: ${{ secrets[matrix.credentials.secrets.ssh_private_key] }} - tailscale-oauth-secret: ${{ secrets[matrix.credentials.secrets.tailscale_oauth_secret] }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} ``` diff --git a/targets/heimdall.yml b/targets/heimdall.yml index ab2a7a2..47cb477 100644 --- a/targets/heimdall.yml +++ b/targets/heimdall.yml @@ -12,10 +12,5 @@ apps: - rubykatzen/flightdeck@latest:cloudflared.sops.env hosts: - root@100.75.53.13 -credentials: - variables: - tailscale_oauth_client_id: TAILSCALE_OAUTH_CLIENT_ID - secrets: - ssh_private_key: DEPLOY_SSH_PRIVATE_KEY - tailscale_oauth_secret: TAILSCALE_OAUTH_SECRET - sops_age_key: HEIMDALL_AGE_PRIVATE_KEY +ssh_private_key_secret: DEPLOY_SSH_PRIVATE_KEY +sops_age_key_secret: HEIMDALL_AGE_PRIVATE_KEY From 59862075acfcb086cb39442f0faa7d3a3014e7c1 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:11:52 +0200 Subject: [PATCH 09/22] feat: renovate accepts a comma-separated list of apps, not just one Renamed app -> apps throughout (renovate.yml, renovate-shared.yml, deploy/renovate.py's stdin contract) so one run can target several apps at once - e.g. a future nightly cron renovating "traefik,rybbit" while leaving everything else alone - instead of one app per invocation. deploy/renovate.py splits the comma-separated list, renovates whichever requested apps are actually keys in the target's own apps mapping (still a clean no-op if none match), and loops hosts inside that. updated_hosts now reports app@host pairs instead of bare hosts, since more than one app can be renovated in the same run. --- .github/workflows/renovate-shared.yml | 12 +++---- .github/workflows/renovate.yml | 6 ++-- README.md | 10 +++--- deploy/renovate.py | 51 +++++++++++++++------------ deploy/tests/test_renovate.py | 32 ++++++++++++----- 5 files changed, 66 insertions(+), 45 deletions(-) diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml index f67908c..69aad5e 100644 --- a/.github/workflows/renovate-shared.yml +++ b/.github/workflows/renovate-shared.yml @@ -2,8 +2,8 @@ name: Renovate (shared) on: workflow_call: inputs: - app: - description: App name to renovate (re-pull its image and recreate its containers). + apps: + description: Comma-separated app name(s) to renovate (re-pull their image and recreate their containers). type: string required: true target-manifest: @@ -61,16 +61,16 @@ jobs: id: run shell: bash env: - APP: ${{ inputs.app }} + APPS: ${{ inputs.apps }} TARGET_MANIFEST: ${{ inputs.target-manifest }} run: | - jq -n --arg app "$APP" --arg target_manifest "$TARGET_MANIFEST" \ - '{app: $app, target_manifest: $target_manifest}' \ + jq -n --arg apps "$APPS" --arg target_manifest "$TARGET_MANIFEST" \ + '{apps: $apps, target_manifest: $target_manifest}' \ | python3 .flightdeck/deploy/renovate.py - name: Notify Telegram if: steps.run.outputs.updated == 'true' uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 with: - message: "Renovate: ${{ inputs.app }} updated on ${{ steps.run.outputs.target_name }} (${{ steps.run.outputs.updated_hosts }})" + message: "Renovate: updated on ${{ steps.run.outputs.target_name }} (${{ steps.run.outputs.updated_hosts }})" telegram-bot-token: ${{ secrets.telegram-bot-token }} telegram-chat-id: ${{ secrets.telegram-chat-id }} diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index 2d79326..e2910f2 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -2,8 +2,8 @@ name: Renovate on: workflow_dispatch: inputs: - app: - description: App name to renovate (re-pull its image and recreate its containers) wherever it's currently deployed. + apps: + description: Comma-separated app name(s) to renovate (re-pull their image and recreate their containers) wherever each is currently deployed. type: string required: true jobs: @@ -23,7 +23,7 @@ jobs: matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} uses: $/.github/workflows/renovate-shared.yml with: - app: ${{ inputs.app }} + apps: ${{ inputs.apps }} target-manifest: ${{ matrix.manifest }} tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} secrets: diff --git a/README.md b/README.md index 66b7d8c..07b357d 100644 --- a/README.md +++ b/README.md @@ -368,13 +368,13 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl **Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-job matrix shape as deploy: `renovate.yml` computes a matrix from `targets/` itself (via [`load-targets-matrix`](.github/actions/load-targets-matrix)) and calls `renovate-shared.yml` once per target, with that target's SSH key already picked out by name (`secrets[matrix.ssh_private_key_secret]`) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`, and deliberately so: it [only works within the same organization or enterprise as the reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows), which would silently break for exactly the external, unrelated callers this is meant to support (confirmed real case: `dupmachine/flightdeck`). Resolving each matrix cell's secret *value* by name instead happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. Tailscale credentials aren't target-specific at all (one shared tailnet), so they're referenced directly (`vars.TAILSCALE_OAUTH_CLIENT_ID`/`secrets.TAILSCALE_OAUTH_SECRET`) rather than resolved per matrix cell. -Like `deploy-shared.yml`, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-targets-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields, and `deploy-shared.yml` now does the same for `hosts`/`apps`/`path`. This keeps the wired interface down to `app` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in) — same shape `deploy-shared.yml` uses. +Like `deploy-shared.yml`, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-targets-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields, and `deploy-shared.yml` now does the same for `hosts`/`apps`/`path`. This keeps the wired interface down to `apps` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in) — same shape `deploy-shared.yml` uses. -Not every target runs every app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: if `app` isn't a key in the target manifest's own `apps` mapping, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. +`apps` is a comma-separated list, so one run can renovate several apps at once (e.g. a nightly cron renovating `traefik,rybbit` while leaving everything else alone) — pass a single name for the one-app case. Not every target runs every requested app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: it renovates whichever requested apps *are* keys in the target manifest's own `apps` mapping and skips the rest; if none match, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. -Renovating means `docker compose pull && docker compose up -d` against that app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), `deploy/renovate.py` compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts`/`target_name` (derived from the manifest's own filename) via `$GITHUB_OUTPUT`. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. +Renovating means `docker compose pull && docker compose up -d` against each matched app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), `deploy/renovate.py` compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts`/`target_name` (derived from the manifest's own filename) via `$GITHUB_OUTPUT` - `updated_hosts` lists `app@host` pairs, since more than one app may have been renovated in the same run. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. -**Known gap:** if `app` matches no target at all (a typo, say), every matrix job is just skipped and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. Deferred until this contract shape is the chosen one. +**Known gap:** if none of the requested `apps` match any target at all (a typo, say), every matrix job is just skipped and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. Deferred until this contract shape is the chosen one. ```yaml jobs: @@ -385,7 +385,7 @@ jobs: matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} uses: $/.github/workflows/renovate-shared.yml with: - app: ${{ inputs.app }} + apps: ${{ inputs.apps }} # comma-separated, e.g. "traefik,rybbit" target-manifest: ${{ matrix.manifest }} tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} secrets: diff --git a/deploy/renovate.py b/deploy/renovate.py index 2d8109d..a72c2cc 100644 --- a/deploy/renovate.py +++ b/deploy/renovate.py @@ -1,9 +1,10 @@ #!/usr/bin/env python3 -"""Renovate: re-pull and recreate one app's containers on one target, read -directly from that target's own targets/*.yml manifest, without touching -versions - no new app bundle, no new vault-sourced env, no rebuilt release -tree. Just `docker compose pull && docker compose up -d` against that -app's already-current release on each of the target's hosts. +"""Renovate: re-pull and recreate one-or-more apps' containers on one +target, read directly from that target's own targets/*.yml manifest, +without touching versions - no new app bundle, no new vault-sourced env, +no rebuilt release tree. Just `docker compose pull && docker compose +up -d` per requested app, against its already-current release, on each +of the target's hosts. Unlike deploy/deploy.py, which receives an already-flattened config, this takes a manifest *path* and parses it itself - the same shape encrypt-env's @@ -13,15 +14,17 @@ is what finds every target and resolves its secrets; this script never reads a directory or touches GitHub Secrets itself. -Reads a JSON config from stdin: {"app": "", "target_manifest": -"targets/heimdall.yml"}. If `app` isn't a key in that manifest's own -`apps` mapping, this is a clean no-op, not an error - a target-matrix -fan-out dispatches to every target regardless of whether it actually -runs the requested app. +Reads a JSON config from stdin: {"apps": "traefik,rybbit", "target_manifest": +"targets/heimdall.yml"}. `apps` is a comma-separated list of one or more +app names. Any requested app that isn't a key in that manifest's own +`apps` mapping is skipped, not an error - a target-matrix fan-out +dispatches to every target regardless of which of the requested apps it +actually runs; skipping all of them there is a clean no-op. Writes `updated`/`updated_hosts`/`target_name` to $GITHUB_OUTPUT so the calling workflow can notify only when a host's image actually changed, -rather than on every run. +rather than on every run. `updated_hosts` lists `app@host` pairs, since +more than one app may have been renovated in the same run. """ import json import os @@ -61,27 +64,29 @@ def write_github_output(name, value): def main(): config = json.load(sys.stdin) - app = config["app"] + requested_apps = [name.strip() for name in config["apps"].split(",") if name.strip()] manifest_path = Path(config["target_manifest"]) target_name = manifest_path.stem write_github_output("target_name", target_name) target = yaml.safe_load(manifest_path.read_text()) - apps = target.get("apps") or {} - if app not in apps: - print(f"{app!r} is not deployed on target {target_name!r}, skipping") + target_apps = target.get("apps") or {} + matching_apps = [app for app in requested_apps if app in target_apps] + if not matching_apps: + print(f"none of {requested_apps} are deployed on target {target_name!r}, skipping") write_github_output("updated", "false") return base_path = target.get("path", "~/flightdeck") - updated_hosts = [] - for host in target["hosts"]: - print(f"Renovating {app} on {host}") - if renovate_host(host, base_path, app): - updated_hosts.append(host) - - write_github_output("updated", "true" if updated_hosts else "false") - write_github_output("updated_hosts", ",".join(updated_hosts)) + updated = [] + for app in matching_apps: + for host in target["hosts"]: + print(f"Renovating {app} on {host}") + if renovate_host(host, base_path, app): + updated.append(f"{app}@{host}") + + write_github_output("updated", "true" if updated else "false") + write_github_output("updated_hosts", ",".join(updated)) if __name__ == "__main__": diff --git a/deploy/tests/test_renovate.py b/deploy/tests/test_renovate.py index ecf7d1f..9187f68 100644 --- a/deploy/tests/test_renovate.py +++ b/deploy/tests/test_renovate.py @@ -68,12 +68,12 @@ def test_returns_false_when_the_image_was_already_current(self): class MainTest(unittest.TestCase): - def _run_main(self, directory, app, manifest_text, manifest_name="heimdall"): + def _run_main(self, directory, apps, manifest_text, manifest_name="heimdall"): manifest_path = Path(directory) / f"{manifest_name}.yml" manifest_path.write_text(manifest_text) output_path = Path(directory) / "outputs" - config = {"app": app, "target_manifest": str(manifest_path)} + config = {"apps": apps, "target_manifest": str(manifest_path)} with ( patch.object(sys, "stdin", io.StringIO(json.dumps(config))), patch.dict(os.environ, {"GITHUB_OUTPUT": str(output_path)}), @@ -81,7 +81,7 @@ def _run_main(self, directory, app, manifest_text, manifest_name="heimdall"): renovate.main() return output_path.read_text() if output_path.exists() else "" - def test_skips_a_target_that_does_not_run_the_app(self): + def test_skips_a_target_that_runs_none_of_the_requested_apps(self): with ( tempfile.TemporaryDirectory() as directory, patch.object(renovate, "renovate_host") as fake_renovate_host, @@ -92,19 +92,35 @@ def test_skips_a_target_that_does_not_run_the_app(self): self.assertIn("updated=false\n", outputs) self.assertIn("target_name=heimdall\n", outputs) - def test_reports_updated_hosts_when_the_image_changed(self): + def test_renovates_only_the_requested_apps_present_on_the_target(self): + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host", return_value=False) as fake_renovate_host, + ): + self._run_main( + directory, + "traefik,rybbit,beszel", + "apps:\n traefik: {}\n gatus: {}\nhosts: [deploy@host]\n", + ) + + fake_renovate_host.assert_called_once_with("deploy@host", "~/flightdeck", "traefik") + + def test_reports_updated_app_host_pairs_when_the_image_changed(self): def fake_renovate_host(host, base_path, app): - return host == "deploy@app1.example.com" + return app == "beszel" and host == "deploy@app1.example.com" - manifest = "apps:\n beszel: {}\nhosts: [deploy@app1.example.com, deploy@app2.example.com]\npath: ~/flightdeck\n" + manifest = ( + "apps:\n beszel: {}\n traefik: {}\n" + "hosts: [deploy@app1.example.com, deploy@app2.example.com]\npath: ~/flightdeck\n" + ) with ( tempfile.TemporaryDirectory() as directory, patch.object(renovate, "renovate_host", side_effect=fake_renovate_host), ): - outputs = self._run_main(directory, "beszel", manifest) + outputs = self._run_main(directory, "beszel,traefik", manifest) self.assertIn("updated=true\n", outputs) - self.assertIn("updated_hosts=deploy@app1.example.com\n", outputs) + self.assertIn("updated_hosts=beszel@deploy@app1.example.com\n", outputs) def test_reports_not_updated_when_every_host_was_already_current(self): with ( From ee3b6d39a8c053ab2cedf1d8ae2bcb5d1412b57e Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:16:25 +0200 Subject: [PATCH 10/22] refactor: pass renovate's apps as a JSON array instead of comma-separated Matches the JSON convention used everywhere else in this codebase (hosts, app_refs, apps mappings) rather than inventing a one-off comma-separated format just for this input. renovate-shared.yml passes it through with --argjson instead of --arg, and deploy/renovate.py reads config["apps"] directly as the already-parsed list. --- .github/workflows/renovate-shared.yml | 4 ++-- .github/workflows/renovate.yml | 2 +- README.md | 4 ++-- deploy/renovate.py | 14 +++++++------- deploy/tests/test_renovate.py | 12 ++++++------ 5 files changed, 18 insertions(+), 18 deletions(-) diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml index 69aad5e..2d8e9dc 100644 --- a/.github/workflows/renovate-shared.yml +++ b/.github/workflows/renovate-shared.yml @@ -3,7 +3,7 @@ on: workflow_call: inputs: apps: - description: Comma-separated app name(s) to renovate (re-pull their image and recreate their containers). + description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers). type: string required: true target-manifest: @@ -64,7 +64,7 @@ jobs: APPS: ${{ inputs.apps }} TARGET_MANIFEST: ${{ inputs.target-manifest }} run: | - jq -n --arg apps "$APPS" --arg target_manifest "$TARGET_MANIFEST" \ + jq -n --argjson apps "$APPS" --arg target_manifest "$TARGET_MANIFEST" \ '{apps: $apps, target_manifest: $target_manifest}' \ | python3 .flightdeck/deploy/renovate.py - name: Notify Telegram diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index e2910f2..aec0c25 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -3,7 +3,7 @@ on: workflow_dispatch: inputs: apps: - description: Comma-separated app name(s) to renovate (re-pull their image and recreate their containers) wherever each is currently deployed. + description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers) wherever each is currently deployed, e.g. ["traefik","rybbit"]. type: string required: true jobs: diff --git a/README.md b/README.md index 07b357d..60b1a27 100644 --- a/README.md +++ b/README.md @@ -370,7 +370,7 @@ The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/depl Like `deploy-shared.yml`, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-targets-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields, and `deploy-shared.yml` now does the same for `hosts`/`apps`/`path`. This keeps the wired interface down to `apps` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in) — same shape `deploy-shared.yml` uses. -`apps` is a comma-separated list, so one run can renovate several apps at once (e.g. a nightly cron renovating `traefik,rybbit` while leaving everything else alone) — pass a single name for the one-app case. Not every target runs every requested app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: it renovates whichever requested apps *are* keys in the target manifest's own `apps` mapping and skips the rest; if none match, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. +`apps` is a JSON array, so one run can renovate several apps at once (e.g. a nightly cron renovating `["traefik","rybbit"]` while leaving everything else alone) — pass a single-element array for the one-app case. Not every target runs every requested app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: it renovates whichever requested apps *are* keys in the target manifest's own `apps` mapping and skips the rest; if none match, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. Renovating means `docker compose pull && docker compose up -d` against each matched app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), `deploy/renovate.py` compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts`/`target_name` (derived from the manifest's own filename) via `$GITHUB_OUTPUT` - `updated_hosts` lists `app@host` pairs, since more than one app may have been renovated in the same run. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. @@ -385,7 +385,7 @@ jobs: matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} uses: $/.github/workflows/renovate-shared.yml with: - apps: ${{ inputs.apps }} # comma-separated, e.g. "traefik,rybbit" + apps: ${{ inputs.apps }} # JSON array, e.g. '["traefik","rybbit"]' target-manifest: ${{ matrix.manifest }} tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} secrets: diff --git a/deploy/renovate.py b/deploy/renovate.py index a72c2cc..bcdc492 100644 --- a/deploy/renovate.py +++ b/deploy/renovate.py @@ -14,12 +14,12 @@ is what finds every target and resolves its secrets; this script never reads a directory or touches GitHub Secrets itself. -Reads a JSON config from stdin: {"apps": "traefik,rybbit", "target_manifest": -"targets/heimdall.yml"}. `apps` is a comma-separated list of one or more -app names. Any requested app that isn't a key in that manifest's own -`apps` mapping is skipped, not an error - a target-matrix fan-out -dispatches to every target regardless of which of the requested apps it -actually runs; skipping all of them there is a clean no-op. +Reads a JSON config from stdin: {"apps": ["traefik", "rybbit"], +"target_manifest": "targets/heimdall.yml"}. Any requested app that isn't +a key in that manifest's own `apps` mapping is skipped, not an error - a +target-matrix fan-out dispatches to every target regardless of which of +the requested apps it actually runs; skipping all of them there is a +clean no-op. Writes `updated`/`updated_hosts`/`target_name` to $GITHUB_OUTPUT so the calling workflow can notify only when a host's image actually changed, @@ -64,7 +64,7 @@ def write_github_output(name, value): def main(): config = json.load(sys.stdin) - requested_apps = [name.strip() for name in config["apps"].split(",") if name.strip()] + requested_apps = config["apps"] manifest_path = Path(config["target_manifest"]) target_name = manifest_path.stem write_github_output("target_name", target_name) diff --git a/deploy/tests/test_renovate.py b/deploy/tests/test_renovate.py index 9187f68..1719137 100644 --- a/deploy/tests/test_renovate.py +++ b/deploy/tests/test_renovate.py @@ -86,7 +86,7 @@ def test_skips_a_target_that_runs_none_of_the_requested_apps(self): tempfile.TemporaryDirectory() as directory, patch.object(renovate, "renovate_host") as fake_renovate_host, ): - outputs = self._run_main(directory, "beszel", "apps:\n traefik: {}\nhosts: [deploy@host]\n") + outputs = self._run_main(directory, ["beszel"], "apps:\n traefik: {}\nhosts: [deploy@host]\n") fake_renovate_host.assert_not_called() self.assertIn("updated=false\n", outputs) @@ -99,7 +99,7 @@ def test_renovates_only_the_requested_apps_present_on_the_target(self): ): self._run_main( directory, - "traefik,rybbit,beszel", + ["traefik", "rybbit", "beszel"], "apps:\n traefik: {}\n gatus: {}\nhosts: [deploy@host]\n", ) @@ -117,7 +117,7 @@ def fake_renovate_host(host, base_path, app): tempfile.TemporaryDirectory() as directory, patch.object(renovate, "renovate_host", side_effect=fake_renovate_host), ): - outputs = self._run_main(directory, "beszel,traefik", manifest) + outputs = self._run_main(directory, ["beszel", "traefik"], manifest) self.assertIn("updated=true\n", outputs) self.assertIn("updated_hosts=beszel@deploy@app1.example.com\n", outputs) @@ -127,7 +127,7 @@ def test_reports_not_updated_when_every_host_was_already_current(self): tempfile.TemporaryDirectory() as directory, patch.object(renovate, "renovate_host", return_value=False), ): - outputs = self._run_main(directory, "beszel", "apps:\n beszel: {}\nhosts: [deploy@host]\n") + outputs = self._run_main(directory, ["beszel"], "apps:\n beszel: {}\nhosts: [deploy@host]\n") self.assertIn("updated=false\n", outputs) self.assertIn("updated_hosts=\n", outputs) @@ -137,7 +137,7 @@ def test_defaults_path_when_omitted(self): tempfile.TemporaryDirectory() as directory, patch.object(renovate, "renovate_host", return_value=False) as fake_renovate_host, ): - self._run_main(directory, "beszel", "apps:\n beszel: {}\nhosts: [deploy@host]\n") + self._run_main(directory, ["beszel"], "apps:\n beszel: {}\nhosts: [deploy@host]\n") fake_renovate_host.assert_called_once_with("deploy@host", "~/flightdeck", "beszel") @@ -147,7 +147,7 @@ def test_derives_target_name_from_manifest_filename(self): patch.object(renovate, "renovate_host", return_value=False), ): outputs = self._run_main( - directory, "beszel", "apps:\n beszel: {}\nhosts: [deploy@host]\n", manifest_name="mainframe" + directory, ["beszel"], "apps:\n beszel: {}\nhosts: [deploy@host]\n", manifest_name="mainframe" ) self.assertIn("target_name=mainframe\n", outputs) From e111afda63d6849a6a18abe993cb7cc1fe842e9f Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:27:11 +0200 Subject: [PATCH 11/22] refactor!: rename load-yaml-matrix to load-vaults-matrix, default directory It's had exactly one consumer since load-targets-matrix took over targets/ (release.yml's encrypt-vaults job) - naming it for what it actually loads and defaulting directory to vaults lets that call site drop the with: block entirely. No behavior change: still no schema validation, encrypt-env re-parses and validates each manifest itself. Test fixtures switched from generic hawkeye/mainframe examples to vault-shaped ones (asset/keys/env) to match what this action actually reads now that it's no longer shared with targets/. --- .github/actions/load-targets-matrix/README.md | 2 +- .../README.md | 8 +- .../action.yml | 11 +-- .../requirements.txt | 0 .../scripts/load-vaults-matrix.py} | 0 .../tests/test_load_vaults_matrix.py | 87 +++++++++++++++++++ .../tests/test_load_yaml_matrix.py | 81 ----------------- .github/workflows/release.yml | 4 +- README.md | 6 +- 9 files changed, 102 insertions(+), 97 deletions(-) rename .github/actions/{load-yaml-matrix => load-vaults-matrix}/README.md (55%) rename .github/actions/{load-yaml-matrix => load-vaults-matrix}/action.yml (65%) rename .github/actions/{load-yaml-matrix => load-vaults-matrix}/requirements.txt (100%) rename .github/actions/{load-yaml-matrix/scripts/load-yaml-matrix.py => load-vaults-matrix/scripts/load-vaults-matrix.py} (100%) create mode 100644 .github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py delete mode 100644 .github/actions/load-yaml-matrix/tests/test_load_yaml_matrix.py diff --git a/.github/actions/load-targets-matrix/README.md b/.github/actions/load-targets-matrix/README.md index 6cce0c5..23af724 100644 --- a/.github/actions/load-targets-matrix/README.md +++ b/.github/actions/load-targets-matrix/README.md @@ -1,6 +1,6 @@ # load-targets-matrix -Composite GitHub Action that reads every target manifest in a directory into a GitHub Actions matrix, validating each one's required shape. Specialized counterpart to [`load-yaml-matrix`](../load-yaml-matrix) for `targets/*.yml` specifically — `load-yaml-matrix` stays generic and unvalidated for everything else (`vaults/*.yml`). +Composite GitHub Action that reads every target manifest in a directory into a GitHub Actions matrix, validating each one's required shape. Sibling to [`load-vaults-matrix`](../load-vaults-matrix), which stays generic and unvalidated for `vaults/*.yml`. ## Usage diff --git a/.github/actions/load-yaml-matrix/README.md b/.github/actions/load-vaults-matrix/README.md similarity index 55% rename from .github/actions/load-yaml-matrix/README.md rename to .github/actions/load-vaults-matrix/README.md index 7e86bd6..f819e14 100644 --- a/.github/actions/load-yaml-matrix/README.md +++ b/.github/actions/load-vaults-matrix/README.md @@ -1,14 +1,14 @@ -# load-yaml-matrix +# load-vaults-matrix -Composite GitHub Action that reads every YAML file in a directory into a GitHub Actions matrix. It does no schema validation — callers are responsible for the shape of their own manifests. +Composite GitHub Action that reads every vault manifest in `vaults/` into a GitHub Actions matrix. It does no schema validation — `encrypt-env` re-parses and validates each manifest itself when it actually encrypts it. Specialized counterpart to [`load-targets-matrix`](../load-targets-matrix), which *does* validate, for `targets/*.yml` specifically. ## Usage ```yaml -- uses: rubykatzen/flightdeck/.github/actions/load-yaml-matrix@main +- uses: rubykatzen/flightdeck/.github/actions/load-vaults-matrix@main id: matrix with: - directory: targets # required + # directory: vaults # optional, default shown # name: all # optional; single manifest name to load, default: all ``` diff --git a/.github/actions/load-yaml-matrix/action.yml b/.github/actions/load-vaults-matrix/action.yml similarity index 65% rename from .github/actions/load-yaml-matrix/action.yml rename to .github/actions/load-vaults-matrix/action.yml index 0b1f6a5..09d0363 100644 --- a/.github/actions/load-yaml-matrix/action.yml +++ b/.github/actions/load-vaults-matrix/action.yml @@ -1,9 +1,10 @@ -name: Load YAML matrix -description: Read every YAML file in a directory into a GitHub Actions matrix. No schema validation. +name: Load vaults matrix +description: Read every vault manifest in a directory into a GitHub Actions matrix. No schema validation - encrypt-env re-parses and validates each manifest itself. inputs: directory: - description: Directory containing YAML files. - required: true + description: Directory containing vault manifests. + required: false + default: vaults name: description: Single manifest name to load, or all. required: false @@ -28,4 +29,4 @@ runs: DIRECTORY: ${{ inputs.directory }} NAME: ${{ inputs.name }} run: | - python3 "$GITHUB_ACTION_PATH/scripts/load-yaml-matrix.py" --directory "$DIRECTORY" --name "$NAME" + python3 "$GITHUB_ACTION_PATH/scripts/load-vaults-matrix.py" --directory "$DIRECTORY" --name "$NAME" diff --git a/.github/actions/load-yaml-matrix/requirements.txt b/.github/actions/load-vaults-matrix/requirements.txt similarity index 100% rename from .github/actions/load-yaml-matrix/requirements.txt rename to .github/actions/load-vaults-matrix/requirements.txt diff --git a/.github/actions/load-yaml-matrix/scripts/load-yaml-matrix.py b/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py similarity index 100% rename from .github/actions/load-yaml-matrix/scripts/load-yaml-matrix.py rename to .github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py diff --git a/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py b/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py new file mode 100644 index 0000000..23811d8 --- /dev/null +++ b/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py @@ -0,0 +1,87 @@ +import importlib.util +import tempfile +import unittest +from pathlib import Path + +MODULE_PATH = Path(__file__).resolve().parents[1] / "scripts" / "load-vaults-matrix.py" +SPEC = importlib.util.spec_from_file_location("load_vaults_matrix", MODULE_PATH) +load_vaults_matrix = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(load_vaults_matrix) + + +TRAEFIK = """\ +asset: mainframe-traefik.sops.env +keys: + - mainframe +env: + HTTP_PORT: ${MAINFRAME_TRAEFIK_HTTP_PORT} +""" + +RYBBIT = """\ +asset: mainframe-rybbit.sops.env +keys: + - mainframe +env: + DOMAIN: ${MAINFRAME_DOMAIN} +""" + + +class LoadVaultsMatrixTest(unittest.TestCase): + def setUp(self): + self.temporary_directory = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary_directory.cleanup) + self.directory = Path(self.temporary_directory.name) + (self.directory / "traefik.yml").write_text(TRAEFIK) + (self.directory / "rybbit.yml").write_text(RYBBIT) + + def test_builds_matrix_from_all_manifests(self): + matrix = load_vaults_matrix.build_matrix(self.directory) + self.assertEqual( + sorted(item["name"] for item in matrix["include"]), + ["rybbit", "traefik"], + ) + + def test_merges_manifest_fields_with_name_and_manifest(self): + item = load_vaults_matrix.build_matrix(self.directory, "traefik")["include"][0] + self.assertEqual(item["name"], "traefik") + self.assertEqual(item["manifest"], str(self.directory / "traefik.yml")) + self.assertEqual(item["asset"], "mainframe-traefik.sops.env") + self.assertEqual(item["keys"], ["mainframe"]) + + def test_filters_selected_manifest(self): + matrix = load_vaults_matrix.build_matrix(self.directory, "traefik") + self.assertEqual([item["name"] for item in matrix["include"]], ["traefik"]) + + def test_rejects_unknown_name(self): + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "unknown name"): + load_vaults_matrix.build_matrix(self.directory, "missing") + + def test_rejects_empty_directory(self): + empty = self.directory / "empty" + empty.mkdir() + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "no manifests found"): + load_vaults_matrix.build_matrix(empty) + + def test_rejects_invalid_manifest_filename(self): + (self.directory / "Traefik_Prod.yml").write_text(TRAEFIK) + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "invalid manifest filename"): + load_vaults_matrix.build_matrix(self.directory) + + def test_rejects_duplicate_manifest_name(self): + (self.directory / "traefik.yaml").write_text(TRAEFIK) + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "duplicate manifest name"): + load_vaults_matrix.build_matrix(self.directory) + + def test_rejects_non_mapping_manifest(self): + (self.directory / "traefik.yml").write_text("- one\n- two\n") + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "must contain a YAML mapping"): + load_vaults_matrix.build_matrix(self.directory) + + def test_rejects_duplicate_yaml_key(self): + (self.directory / "traefik.yml").write_text("keys: [one]\nkeys: [two]\n") + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "duplicate YAML key"): + load_vaults_matrix.build_matrix(self.directory) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/actions/load-yaml-matrix/tests/test_load_yaml_matrix.py b/.github/actions/load-yaml-matrix/tests/test_load_yaml_matrix.py deleted file mode 100644 index 8d1cfb2..0000000 --- a/.github/actions/load-yaml-matrix/tests/test_load_yaml_matrix.py +++ /dev/null @@ -1,81 +0,0 @@ -import importlib.util -import tempfile -import unittest -from pathlib import Path - -MODULE_PATH = Path(__file__).resolve().parents[1] / "scripts" / "load-yaml-matrix.py" -SPEC = importlib.util.spec_from_file_location("load_yaml_matrix", MODULE_PATH) -load_yaml_matrix = importlib.util.module_from_spec(SPEC) -SPEC.loader.exec_module(load_yaml_matrix) - - -HAWKEYE = """\ -flightdeck_ref: rubykatzen/flightdeck@v1.2.3 -hosts: [rubykatzen-com@100.75.50.2] -""" - -MAINFRAME = """\ -flightdeck_ref: rubykatzen/flightdeck@v1.0.0 -hosts: [deploy@100.64.0.1] -""" - - -class LoadYamlMatrixTest(unittest.TestCase): - def setUp(self): - self.temporary_directory = tempfile.TemporaryDirectory() - self.addCleanup(self.temporary_directory.cleanup) - self.directory = Path(self.temporary_directory.name) - (self.directory / "hawkeye.yml").write_text(HAWKEYE) - (self.directory / "mainframe.yml").write_text(MAINFRAME) - - def test_builds_matrix_from_all_manifests(self): - matrix = load_yaml_matrix.build_matrix(self.directory) - self.assertEqual( - sorted(item["name"] for item in matrix["include"]), - ["hawkeye", "mainframe"], - ) - - def test_merges_manifest_fields_with_name_and_manifest(self): - item = load_yaml_matrix.build_matrix(self.directory, "hawkeye")["include"][0] - self.assertEqual(item["name"], "hawkeye") - self.assertEqual(item["manifest"], str(self.directory / "hawkeye.yml")) - self.assertEqual(item["flightdeck_ref"], "rubykatzen/flightdeck@v1.2.3") - self.assertEqual(item["hosts"], ["rubykatzen-com@100.75.50.2"]) - - def test_filters_selected_manifest(self): - matrix = load_yaml_matrix.build_matrix(self.directory, "hawkeye") - self.assertEqual([item["name"] for item in matrix["include"]], ["hawkeye"]) - - def test_rejects_unknown_name(self): - with self.assertRaisesRegex(load_yaml_matrix.ManifestError, "unknown name"): - load_yaml_matrix.build_matrix(self.directory, "missing") - - def test_rejects_empty_directory(self): - empty = self.directory / "empty" - empty.mkdir() - with self.assertRaisesRegex(load_yaml_matrix.ManifestError, "no manifests found"): - load_yaml_matrix.build_matrix(empty) - - def test_rejects_invalid_manifest_filename(self): - (self.directory / "Hawkeye_Prod.yml").write_text(HAWKEYE) - with self.assertRaisesRegex(load_yaml_matrix.ManifestError, "invalid manifest filename"): - load_yaml_matrix.build_matrix(self.directory) - - def test_rejects_duplicate_manifest_name(self): - (self.directory / "hawkeye.yaml").write_text(HAWKEYE) - with self.assertRaisesRegex(load_yaml_matrix.ManifestError, "duplicate manifest name"): - load_yaml_matrix.build_matrix(self.directory) - - def test_rejects_non_mapping_manifest(self): - (self.directory / "hawkeye.yml").write_text("- one\n- two\n") - with self.assertRaisesRegex(load_yaml_matrix.ManifestError, "must contain a YAML mapping"): - load_yaml_matrix.build_matrix(self.directory) - - def test_rejects_duplicate_yaml_key(self): - (self.directory / "hawkeye.yml").write_text("hosts: [one]\nhosts: [two]\n") - with self.assertRaisesRegex(load_yaml_matrix.ManifestError, "duplicate YAML key"): - load_yaml_matrix.build_matrix(self.directory) - - -if __name__ == "__main__": - unittest.main() diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 018568f..081967b 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -40,10 +40,8 @@ jobs: - uses: actions/checkout@v7 with: ref: ${{ needs.release.outputs.tag_name }} - - uses: $/.github/actions/load-yaml-matrix + - uses: $/.github/actions/load-vaults-matrix id: matrix - with: - directory: vaults encrypt: needs: [release, encrypt-vaults] if: needs.encrypt-vaults.outputs.count != '0' diff --git a/README.md b/README.md index 60b1a27..ba9313f 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ flightdeck/ │ │ ├── build-bundle/ # Build and upload a zip bundle from given paths │ │ ├── build-apps-bundle/ # Build and upload an apps/ catalog bundle │ │ ├── encrypt-env/ # Encrypt a target env and upload it to a release -│ │ ├── load-yaml-matrix/ # Read a directory of YAML manifests into a workflow matrix (vaults/) +│ │ ├── load-vaults-matrix/ # Read vaults/ into a workflow matrix, no schema validation │ │ └── load-targets-matrix/ # Same, but for targets/ specifically - validates the required shape │ └── workflows/ │ ├── deploy-shared.yml # Reusable deployment workflow @@ -188,7 +188,7 @@ Useful as a source of ready-made Docker Compose definitions when adding a new ap ## GitHub Actions -This repository provides five composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, `load-yaml-matrix`, and `load-targets-matrix`) and one reusable workflow, `deploy-shared.yml`. +This repository provides five composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, `load-vaults-matrix`, and `load-targets-matrix`) and one reusable workflow, `deploy-shared.yml`. --- @@ -245,7 +245,7 @@ sops_age_key_secret: MAINFRAME_AGE_PRIVATE_KEY A vault manifest's `env:` value is either `${NAME}` (a reference — look up the GitHub Secret/Variable named `NAME`) or a bare literal (any other value, used as-is with no lookup at all — see `DISABLE_SIGNUP: true` above). Use a literal for a value that's fixed for this target but isn't a secret and doesn't need a GitHub Secret/Variable to exist just to hold it. -`load-yaml-matrix` reads every file in `vaults/` into a matrix — it does not validate the manifest shape; `encrypt-env` re-parses and validates its own manifest from `manifest` (see "`encrypt-env`" below). Targets go through the more specific [`load-targets-matrix`](.github/actions/load-targets-matrix) instead, which *does* validate the shape above (`hosts`, `app_refs`, `apps`, `ssh_private_key_secret`, `sops_age_key_secret` all required) before a broken manifest ever reaches a checkout+dependency-install on a different job entirely. The workflows calling `deploy-shared.yml`/`renovate-shared.yml` then pull `matrix.ssh_private_key_secret`/`matrix.sops_age_key_secret` directly, to resolve actual secret values by name. +[`load-vaults-matrix`](.github/actions/load-vaults-matrix) reads every file in `vaults/` into a matrix — it does not validate the manifest shape; `encrypt-env` re-parses and validates its own manifest from `manifest` (see "`encrypt-env`" below). Targets go through the more specific [`load-targets-matrix`](.github/actions/load-targets-matrix) instead, which *does* validate the shape above (`hosts`, `app_refs`, `apps`, `ssh_private_key_secret`, `sops_age_key_secret` all required) before a broken manifest ever reaches a checkout+dependency-install on a different job entirely. The workflows calling `deploy-shared.yml`/`renovate-shared.yml` then pull `matrix.ssh_private_key_secret`/`matrix.sops_age_key_secret` directly, to resolve actual secret values by name. --- From 6f7293be20acc88054df2a4660c397be826f6bfd Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:47:35 +0200 Subject: [PATCH 12/22] refactor!: replace deploy-shared.yml/renovate-shared.yml with composite actions Both were single-job workflow_call workflows - exactly the shape GitHub's own guidance says a composite action fits better than a reusable workflow (which earns its keep with multiple jobs or job-level config, neither of which these ever used). This also directly eliminates two problems the workflow_call boundary forced on us: - The second checkout (repository: job.workflow_repository, ref: job.workflow_sha) needed purely to get this repo's own deploy.py/ renovate.py onto the runner is gone - a composite action's own files come along automatically via $GITHUB_ACTION_PATH whenever it's referenced as owner/repo/.github/actions/X@ref. - target-manifest-ref is gone with it - it only existed to keep that second checkout's ref in sync with whatever ref the caller had already resolved its matrix from. With one checkout instead of two, there's nothing to keep in sync. deploy.py and its sibling modules (collisions.py, render.py, resolve.py, vault.py) move from deploy/ into .github/actions/deploy/, since a composite action's script has to live inside the action's own directory tree to be reachable via $GITHUB_ACTION_PATH; deploy/tests/ moves the same way. renovate.py and its tests move into .github/actions/renovate/ - it already had zero dependency on deploy.py's sibling modules, so it becomes its own action rather than living alongside deploy's. deploy.yml/release.yml/renovate.yml's matrix jobs now do their own actions/checkout, resolve secrets themselves, and call the action directly as one step among their own - no more uses:/secrets: mapping into a separate workflow file. No script logic changed, only where the files live and how they're invoked; all 89 tests pass unchanged in their new locations. --- .github/actions/deploy/README.md | 28 +++++ .github/actions/deploy/action.yml | 65 +++++++++++ .../actions/deploy}/collisions.py | 0 {deploy => .github/actions/deploy}/deploy.py | 5 +- {deploy => .github/actions/deploy}/render.py | 0 .../actions/deploy}/requirements.txt | 0 {deploy => .github/actions/deploy}/resolve.py | 0 .../actions/deploy}/tests/test_collisions.py | 0 .../actions/deploy}/tests/test_deploy.py | 0 .../actions/deploy}/tests/test_render.py | 0 .../actions/deploy}/tests/test_resolve.py | 0 .../actions/deploy}/tests/test_vault.py | 0 {deploy => .github/actions/deploy}/vault.py | 0 .github/actions/load-targets-matrix/README.md | 2 +- .github/actions/renovate/README.md | 28 +++++ .github/actions/renovate/action.yml | 66 +++++++++++ .../actions/renovate}/renovate.py | 14 +-- .github/actions/renovate/requirements.txt | 2 + .../actions/renovate}/tests/test_renovate.py | 0 .github/workflows/deploy-shared.yml | 77 ------------- .github/workflows/deploy.yml | 19 ++-- .github/workflows/release.yml | 22 ++-- .github/workflows/renovate-shared.yml | 76 ------------- .github/workflows/renovate.yml | 24 ++-- AGENTS.md | 28 ++--- README.md | 103 +++++++++--------- 26 files changed, 301 insertions(+), 258 deletions(-) create mode 100644 .github/actions/deploy/README.md create mode 100644 .github/actions/deploy/action.yml rename {deploy => .github/actions/deploy}/collisions.py (100%) rename {deploy => .github/actions/deploy}/deploy.py (98%) rename {deploy => .github/actions/deploy}/render.py (100%) rename {deploy => .github/actions/deploy}/requirements.txt (100%) rename {deploy => .github/actions/deploy}/resolve.py (100%) rename {deploy => .github/actions/deploy}/tests/test_collisions.py (100%) rename {deploy => .github/actions/deploy}/tests/test_deploy.py (100%) rename {deploy => .github/actions/deploy}/tests/test_render.py (100%) rename {deploy => .github/actions/deploy}/tests/test_resolve.py (100%) rename {deploy => .github/actions/deploy}/tests/test_vault.py (100%) rename {deploy => .github/actions/deploy}/vault.py (100%) create mode 100644 .github/actions/renovate/README.md create mode 100644 .github/actions/renovate/action.yml rename {deploy => .github/actions/renovate}/renovate.py (86%) create mode 100644 .github/actions/renovate/requirements.txt rename {deploy => .github/actions/renovate}/tests/test_renovate.py (100%) delete mode 100644 .github/workflows/deploy-shared.yml delete mode 100644 .github/workflows/renovate-shared.yml diff --git a/.github/actions/deploy/README.md b/.github/actions/deploy/README.md new file mode 100644 index 0000000..98bea06 --- /dev/null +++ b/.github/actions/deploy/README.md @@ -0,0 +1,28 @@ +# deploy + +Composite GitHub Action that runs a push-based deploy against a target manifest already present in the caller's own checkout - resolves and downloads every release ref, decrypts and renders each app's env and config, merges the release, and pushes the finished result to each host over SSH. See the main [README](../../../README.md#automated-deploy) for the full sequence. + +## Usage + +```yaml +jobs: + deploy: + strategy: + matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: rubykatzen/flightdeck/.github/actions/deploy@main + with: + target-manifest: ${{ matrix.manifest }} # required, path in this repository + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} + github-token: ${{ secrets.GITHUB_TOKEN }} + # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) + # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} # required only if tailscale-oauth-client-id is set + # tailscale-tags: tag:ci # default: tag:ci +``` + +Unlike a `workflow_call` reusable workflow, this action doesn't check out anything itself - it reads `target-manifest` from whatever the caller's own preceding `actions/checkout` step already put on disk, and its own code (`deploy.py` and friends) comes along automatically via `$GITHUB_ACTION_PATH` whenever it's referenced as `owner/repo/.github/actions/deploy@ref`. This is also why there's no `target-manifest-ref` input here: if the caller needs a specific ref (e.g. a just-published release tag), it just checks out that ref itself before this step runs, the same way every other job in this repository already does. + +`sops-age-key`/`ssh-private-key` are secret *values*, resolved by the caller from the target manifest's own `sops_age_key_secret`/`ssh_private_key_secret` fields (GitHub Secret *names* - see the main README's "Vaults And Targets" section) - this action never reads the manifest's credential fields itself, since it never has access to `secrets.*` by name. diff --git a/.github/actions/deploy/action.yml b/.github/actions/deploy/action.yml new file mode 100644 index 0000000..de5a65c --- /dev/null +++ b/.github/actions/deploy/action.yml @@ -0,0 +1,65 @@ +name: Deploy +description: Push-based deploy against a target manifest already present in the caller's own checkout. +inputs: + target-manifest: + description: Path to the target manifest file, e.g. targets/heimdall.yml - already checked out by the caller. + required: true + ssh-private-key: + description: SSH private key used to connect to the hosts. + required: true + sops-age-key: + description: Private SOPS age key used to decrypt this target's vault-sourced env, on the runner. + required: true + github-token: + description: Token used to resolve/download release refs (app bundles, encrypted env sources). + required: true + tailscale-oauth-client-id: + description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). + required: false + default: "" + tailscale-oauth-secret: + description: Tailscale OAuth client secret used to join the tailnet. Required only when tailscale-oauth-client-id is set. + required: false + default: "" + tailscale-tags: + description: Comma-separated ACL tags for the ephemeral tailnet node. + required: false + default: tag:ci +runs: + using: composite + steps: + - name: Install deploy dependencies + shell: bash + run: pip install --user --break-system-packages --requirement "$GITHUB_ACTION_PATH/requirements.txt" + - name: Install sops + shell: bash + env: + SOPS_VERSION: "3.13.1" + run: | + curl --fail --location --silent --show-error \ + "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64" \ + --output /usr/local/bin/sops + chmod +x /usr/local/bin/sops + - uses: tailscale/github-action@v4 + if: inputs.tailscale-oauth-client-id != '' + with: + oauth-client-id: ${{ inputs.tailscale-oauth-client-id }} + oauth-secret: ${{ inputs.tailscale-oauth-secret }} + tags: ${{ inputs.tailscale-tags }} + - name: Load SSH key + shell: bash + run: | + eval "$(ssh-agent -s)" + echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" + echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" + ssh-add - <<< "${{ inputs.ssh-private-key }}" + - name: Run deploy + shell: bash + env: + TARGET_MANIFEST: ${{ inputs.target-manifest }} + SOPS_AGE_KEY: ${{ inputs.sops-age-key }} + GH_TOKEN: ${{ inputs.github-token }} + run: | + jq -n --arg target_manifest "$TARGET_MANIFEST" --arg sops_age_key "$SOPS_AGE_KEY" \ + '{target_manifest: $target_manifest, sops_age_key: $sops_age_key}' \ + | python3 "$GITHUB_ACTION_PATH/deploy.py" diff --git a/deploy/collisions.py b/.github/actions/deploy/collisions.py similarity index 100% rename from deploy/collisions.py rename to .github/actions/deploy/collisions.py diff --git a/deploy/deploy.py b/.github/actions/deploy/deploy.py similarity index 98% rename from deploy/deploy.py rename to .github/actions/deploy/deploy.py index 6dbaf96..3f7024d 100644 --- a/deploy/deploy.py +++ b/.github/actions/deploy/deploy.py @@ -11,9 +11,8 @@ manifest, parsed here rather than flattened into separate fields by the caller - the same shape encrypt-env's render-env.py already uses for vault manifests. `sops_age_key` can't come from that file, since it's a -secret value, not YAML-safe config - deploy-shared.yml passes it -separately (see README's "deploy-shared.yml" section for the exact -contract). +secret value, not YAML-safe config - the calling `deploy` action passes +it separately (see README's "deploy" section for the exact contract). """ import json import shlex diff --git a/deploy/render.py b/.github/actions/deploy/render.py similarity index 100% rename from deploy/render.py rename to .github/actions/deploy/render.py diff --git a/deploy/requirements.txt b/.github/actions/deploy/requirements.txt similarity index 100% rename from deploy/requirements.txt rename to .github/actions/deploy/requirements.txt diff --git a/deploy/resolve.py b/.github/actions/deploy/resolve.py similarity index 100% rename from deploy/resolve.py rename to .github/actions/deploy/resolve.py diff --git a/deploy/tests/test_collisions.py b/.github/actions/deploy/tests/test_collisions.py similarity index 100% rename from deploy/tests/test_collisions.py rename to .github/actions/deploy/tests/test_collisions.py diff --git a/deploy/tests/test_deploy.py b/.github/actions/deploy/tests/test_deploy.py similarity index 100% rename from deploy/tests/test_deploy.py rename to .github/actions/deploy/tests/test_deploy.py diff --git a/deploy/tests/test_render.py b/.github/actions/deploy/tests/test_render.py similarity index 100% rename from deploy/tests/test_render.py rename to .github/actions/deploy/tests/test_render.py diff --git a/deploy/tests/test_resolve.py b/.github/actions/deploy/tests/test_resolve.py similarity index 100% rename from deploy/tests/test_resolve.py rename to .github/actions/deploy/tests/test_resolve.py diff --git a/deploy/tests/test_vault.py b/.github/actions/deploy/tests/test_vault.py similarity index 100% rename from deploy/tests/test_vault.py rename to .github/actions/deploy/tests/test_vault.py diff --git a/deploy/vault.py b/.github/actions/deploy/vault.py similarity index 100% rename from deploy/vault.py rename to .github/actions/deploy/vault.py diff --git a/.github/actions/load-targets-matrix/README.md b/.github/actions/load-targets-matrix/README.md index 23af724..6d24b7c 100644 --- a/.github/actions/load-targets-matrix/README.md +++ b/.github/actions/load-targets-matrix/README.md @@ -14,6 +14,6 @@ Composite GitHub Action that reads every target manifest in a directory into a G The action exposes `matrix`, containing `{ "include": [...] }`, and `count`. Each matrix item merges the manifest's own top-level YAML fields with `name` (the file's basename) and `manifest` (its path). -Every manifest must set `hosts` (non-empty list), `app_refs` (non-empty list), `apps` (non-empty object), `ssh_private_key_secret` (non-empty string - the name of the GitHub Secret holding this target's SSH private key), and `sops_age_key_secret` (non-empty string - the name of the GitHub Secret holding this target's private SOPS age key). This fails the matrix build immediately, with an error naming the specific broken target, rather than letting a malformed manifest reach `deploy/deploy.py`/`deploy/renovate.py` much later, after a checkout and dependency install on a different job entirely. +Every manifest must set `hosts` (non-empty list), `app_refs` (non-empty list), `apps` (non-empty object), `ssh_private_key_secret` (non-empty string - the name of the GitHub Secret holding this target's SSH private key), and `sops_age_key_secret` (non-empty string - the name of the GitHub Secret holding this target's private SOPS age key). This fails the matrix build immediately, with an error naming the specific broken target, rather than letting a malformed manifest reach the [`deploy`](../deploy)/[`renovate`](../renovate) actions much later, after a checkout and dependency install on a different job entirely. Files may use either the `.yml` or `.yaml` extension. Filenames must match `^[a-z0-9][a-z0-9-]*$` and be unique per directory; duplicate top-level YAML keys within a manifest are rejected. diff --git a/.github/actions/renovate/README.md b/.github/actions/renovate/README.md new file mode 100644 index 0000000..856f5b9 --- /dev/null +++ b/.github/actions/renovate/README.md @@ -0,0 +1,28 @@ +# renovate + +Composite GitHub Action that re-pulls and recreates one-or-more apps' containers on a target already present in the caller's own checkout, without touching versions - no new app bundle, no new vault-sourced env, no rebuilt release tree. Just `docker compose pull && docker compose up -d` per requested app, against its already-current release, on each of the target's hosts. Sibling to [`deploy`](../deploy), same shape, deliberately narrower job. + +## Usage + +```yaml +jobs: + renovate: + strategy: + matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: rubykatzen/flightdeck/.github/actions/renovate@main + with: + apps: ${{ inputs.apps }} # required, JSON array e.g. '["traefik","rybbit"]' + target-manifest: ${{ matrix.manifest }} # required, path in this repository + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} + # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) + # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} # required only if tailscale-oauth-client-id is set +``` + +Not every target runs every requested app - `renovate.py` decides that itself, from the target manifest's own `apps` mapping, and simply does nothing (never opening an SSH connection) if none of the requested apps are present. To tell whether a host's image actually changed (rather than the pull being a no-op), it compares `docker compose images -q` output before and after the pull. Only sends the Telegram notification when at least one app actually changed - silent otherwise. + +Same checkout model as [`deploy`](../deploy): this action never checks out anything itself, it just reads `target-manifest` from whatever the caller's own preceding `actions/checkout` step already put on disk. diff --git a/.github/actions/renovate/action.yml b/.github/actions/renovate/action.yml new file mode 100644 index 0000000..7d3a438 --- /dev/null +++ b/.github/actions/renovate/action.yml @@ -0,0 +1,66 @@ +name: Renovate +description: Re-pull and recreate one-or-more apps' containers on a target already present in the caller's own checkout, without touching versions. +inputs: + apps: + description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers). + required: true + target-manifest: + description: Path to the target manifest file, e.g. targets/heimdall.yml - already checked out by the caller. + required: true + ssh-private-key: + description: SSH private key used to connect to the hosts. + required: true + telegram-bot-token: + description: Telegram bot token used to notify when an update is found. + required: true + telegram-chat-id: + description: Telegram chat ID used to notify when an update is found. + required: true + tailscale-oauth-client-id: + description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). + required: false + default: "" + tailscale-oauth-secret: + description: Tailscale OAuth client secret used to join the tailnet. Required only when tailscale-oauth-client-id is set. + required: false + default: "" + tailscale-tags: + description: Comma-separated ACL tags for the ephemeral tailnet node. + required: false + default: tag:ci +runs: + using: composite + steps: + - name: Install deploy dependencies + shell: bash + run: pip install --user --break-system-packages --requirement "$GITHUB_ACTION_PATH/requirements.txt" + - uses: tailscale/github-action@v4 + if: inputs.tailscale-oauth-client-id != '' + with: + oauth-client-id: ${{ inputs.tailscale-oauth-client-id }} + oauth-secret: ${{ inputs.tailscale-oauth-secret }} + tags: ${{ inputs.tailscale-tags }} + - name: Load SSH key + shell: bash + run: | + eval "$(ssh-agent -s)" + echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" + echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" + ssh-add - <<< "${{ inputs.ssh-private-key }}" + - name: Run renovate + id: run + shell: bash + env: + APPS: ${{ inputs.apps }} + TARGET_MANIFEST: ${{ inputs.target-manifest }} + run: | + jq -n --argjson apps "$APPS" --arg target_manifest "$TARGET_MANIFEST" \ + '{apps: $apps, target_manifest: $target_manifest}' \ + | python3 "$GITHUB_ACTION_PATH/renovate.py" + - name: Notify Telegram + if: steps.run.outputs.updated == 'true' + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Renovate: updated on ${{ steps.run.outputs.target_name }} (${{ steps.run.outputs.updated_hosts }})" + telegram-bot-token: ${{ inputs.telegram-bot-token }} + telegram-chat-id: ${{ inputs.telegram-chat-id }} diff --git a/deploy/renovate.py b/.github/actions/renovate/renovate.py similarity index 86% rename from deploy/renovate.py rename to .github/actions/renovate/renovate.py index bcdc492..47d7b06 100644 --- a/deploy/renovate.py +++ b/.github/actions/renovate/renovate.py @@ -6,13 +6,13 @@ up -d` per requested app, against its already-current release, on each of the target's hosts. -Unlike deploy/deploy.py, which receives an already-flattened config, this -takes a manifest *path* and parses it itself - the same shape encrypt-env's -render-env.py already uses for vault manifests, rather than flattening a -target's hosts/apps/path into separate inputs the caller has to build. -renovate-shared.yml's own two-job matrix (see README's "Renovate" section) -is what finds every target and resolves its secrets; this script never -reads a directory or touches GitHub Secrets itself. +Like deploy.py, this takes a manifest *path* and parses it itself - the +same shape encrypt-env's render-env.py already uses for vault manifests, +rather than flattening a target's hosts/apps/path into separate inputs +the caller has to build. Finding every target and resolving its secrets +happens in the calling workflow's own matrix (see README's "renovate" +section), not here - this script never reads a directory or touches +GitHub Secrets itself. Reads a JSON config from stdin: {"apps": ["traefik", "rybbit"], "target_manifest": "targets/heimdall.yml"}. Any requested app that isn't diff --git a/.github/actions/renovate/requirements.txt b/.github/actions/renovate/requirements.txt new file mode 100644 index 0000000..f38c8dc --- /dev/null +++ b/.github/actions/renovate/requirements.txt @@ -0,0 +1,2 @@ +fabric +PyYAML==6.0.3 diff --git a/deploy/tests/test_renovate.py b/.github/actions/renovate/tests/test_renovate.py similarity index 100% rename from deploy/tests/test_renovate.py rename to .github/actions/renovate/tests/test_renovate.py diff --git a/.github/workflows/deploy-shared.yml b/.github/workflows/deploy-shared.yml deleted file mode 100644 index a959180..0000000 --- a/.github/workflows/deploy-shared.yml +++ /dev/null @@ -1,77 +0,0 @@ -name: Deploy (shared) -on: - workflow_call: - inputs: - target-manifest: - description: Path to this target's manifest file in the calling repository's own targets/ directory, e.g. targets/heimdall.yml. - type: string - required: true - target-manifest-ref: - description: Git ref to check out this repository at when reading target-manifest. Leave unset to use the default branch (e.g. set this to a release tag when the caller itself pinned its own checkout to one, so both reads agree). - type: string - default: "" - tailscale-oauth-client-id: - description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). - type: string - default: "" - tailscale-tags: - description: Comma-separated ACL tags for the ephemeral tailnet node. - type: string - default: tag:ci - secrets: - ssh-private-key: - description: SSH private key used to connect to the hosts. - required: true - sops-age-key: - description: Private SOPS age key used to decrypt this target's vault-sourced env, on the runner. - required: true - tailscale-oauth-secret: - description: Tailscale OAuth client secret used to join the tailnet. Required only when tailscale-oauth-client-id is set. - required: false -jobs: - deploy: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v7 - with: - ref: ${{ inputs.target-manifest-ref }} - - uses: actions/checkout@v7 - with: - repository: ${{ job.workflow_repository }} - ref: ${{ job.workflow_sha }} - path: .flightdeck - - name: Install deploy dependencies - shell: bash - run: pip install --user --break-system-packages -r .flightdeck/deploy/requirements.txt - - name: Install sops - shell: bash - env: - SOPS_VERSION: "3.13.1" - run: | - curl --fail --location --silent --show-error \ - "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64" \ - --output /usr/local/bin/sops - chmod +x /usr/local/bin/sops - - uses: tailscale/github-action@v4 - if: inputs.tailscale-oauth-client-id != '' - with: - oauth-client-id: ${{ inputs.tailscale-oauth-client-id }} - oauth-secret: ${{ secrets.tailscale-oauth-secret }} - tags: ${{ inputs.tailscale-tags }} - - name: Load SSH key - shell: bash - run: | - eval "$(ssh-agent -s)" - echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" - echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" - ssh-add - <<< "${{ secrets.ssh-private-key }}" - - name: Run deploy - shell: bash - env: - TARGET_MANIFEST: ${{ inputs.target-manifest }} - SOPS_AGE_KEY: ${{ secrets.sops-age-key }} - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - jq -n --arg target_manifest "$TARGET_MANIFEST" --arg sops_age_key "$SOPS_AGE_KEY" \ - '{target_manifest: $target_manifest, sops_age_key: $sops_age_key}' \ - | python3 .flightdeck/deploy/deploy.py diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index c076af8..0b6d515 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -23,11 +23,14 @@ jobs: if: needs.deploy-targets.outputs.count != '0' strategy: matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} - uses: $/.github/workflows/deploy-shared.yml - with: - target-manifest: ${{ matrix.manifest }} - tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} - secrets: - ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} - tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} - sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: $/.github/actions/deploy + with: + target-manifest: ${{ matrix.manifest }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} + github-token: ${{ secrets.GITHUB_TOKEN }} + tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} + tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 081967b..d9c6f9d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -78,12 +78,16 @@ jobs: if: needs.deploy-targets.outputs.count != '0' strategy: matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} - uses: $/.github/workflows/deploy-shared.yml - with: - target-manifest: ${{ matrix.manifest }} - target-manifest-ref: ${{ needs.release.outputs.tag_name }} - tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} - secrets: - ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} - tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} - sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + with: + ref: ${{ needs.release.outputs.tag_name }} + - uses: $/.github/actions/deploy + with: + target-manifest: ${{ matrix.manifest }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} + github-token: ${{ secrets.GITHUB_TOKEN }} + tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} + tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} diff --git a/.github/workflows/renovate-shared.yml b/.github/workflows/renovate-shared.yml deleted file mode 100644 index 2d8e9dc..0000000 --- a/.github/workflows/renovate-shared.yml +++ /dev/null @@ -1,76 +0,0 @@ -name: Renovate (shared) -on: - workflow_call: - inputs: - apps: - description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers). - type: string - required: true - target-manifest: - description: Path to this target's manifest file in the calling repository's own targets/ directory, e.g. targets/heimdall.yml. - type: string - required: true - tailscale-oauth-client-id: - description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). - type: string - default: "" - tailscale-tags: - description: Comma-separated ACL tags for the ephemeral tailnet node. - type: string - default: tag:ci - secrets: - ssh-private-key: - description: SSH private key used to connect to the hosts. - required: true - tailscale-oauth-secret: - description: Tailscale OAuth client secret used to join the tailnet. Required only when tailscale-oauth-client-id is set. - required: false - telegram-bot-token: - description: Telegram bot token used to notify when an update is found. - required: true - telegram-chat-id: - description: Telegram chat ID used to notify when an update is found. - required: true -jobs: - renovate: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v7 - - uses: actions/checkout@v7 - with: - repository: ${{ job.workflow_repository }} - ref: ${{ job.workflow_sha }} - path: .flightdeck - - name: Install deploy dependencies - shell: bash - run: pip install --user --break-system-packages -r .flightdeck/deploy/requirements.txt - - uses: tailscale/github-action@v4 - if: inputs.tailscale-oauth-client-id != '' - with: - oauth-client-id: ${{ inputs.tailscale-oauth-client-id }} - oauth-secret: ${{ secrets.tailscale-oauth-secret }} - tags: ${{ inputs.tailscale-tags }} - - name: Load SSH key - shell: bash - run: | - eval "$(ssh-agent -s)" - echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" - echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV" - ssh-add - <<< "${{ secrets.ssh-private-key }}" - - name: Run renovate - id: run - shell: bash - env: - APPS: ${{ inputs.apps }} - TARGET_MANIFEST: ${{ inputs.target-manifest }} - run: | - jq -n --argjson apps "$APPS" --arg target_manifest "$TARGET_MANIFEST" \ - '{apps: $apps, target_manifest: $target_manifest}' \ - | python3 .flightdeck/deploy/renovate.py - - name: Notify Telegram - if: steps.run.outputs.updated == 'true' - uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 - with: - message: "Renovate: updated on ${{ steps.run.outputs.target_name }} (${{ steps.run.outputs.updated_hosts }})" - telegram-bot-token: ${{ secrets.telegram-bot-token }} - telegram-chat-id: ${{ secrets.telegram-chat-id }} diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index aec0c25..ad96edb 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -21,15 +21,15 @@ jobs: if: needs.find-targets.outputs.count != '0' strategy: matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} - uses: $/.github/workflows/renovate-shared.yml - with: - apps: ${{ inputs.apps }} - target-manifest: ${{ matrix.manifest }} - tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} - secrets: - # Per-target, resolved by name from this target's own manifest. - ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} - # Fixed - the same for every target, not resolved from the manifest. - tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} - telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} - telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: $/.github/actions/renovate + with: + apps: ${{ inputs.apps }} + target-manifest: ${{ matrix.manifest }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} + tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/AGENTS.md b/AGENTS.md index 193c314..860ea2e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,7 @@ input. ## Repository Overview -This is a Docker-based deployment system (flightdeck) that manages core services via docker-compose. The architecture uses Traefik as a reverse proxy with automatic SSL certificate management, and can merge optional extra application catalogs during deploy. There is no manual administration flow at all - no server-side console access, no local quick-start. Every deploy goes through `targets/`/`vaults/` manifests and GitHub Actions (`deploy/deploy.py`); see the "Environment Variable System" and "CI/CD" sections below. +This is a Docker-based deployment system (flightdeck) that manages core services via docker-compose. The architecture uses Traefik as a reverse proxy with automatic SSL certificate management, and can merge optional extra application catalogs during deploy. There is no manual administration flow at all - no server-side console access, no local quick-start. Every deploy goes through `targets/`/`vaults/` manifests and GitHub Actions (`.github/actions/deploy/deploy.py`); see the "Environment Variable System" and "CI/CD" sections below. ## Core Architecture @@ -60,7 +60,7 @@ This is a Docker-based deployment system (flightdeck) that manages core services - `apps/` - Contains core docker-compose configurations and shared compose templates - `apps-data/` - Persistent data storage on the target host (not in this repo): only state that must survive across releases and isn't regenerated by a deploy (database volumes, `traefik/acme.json`) -- `deploy/` - The push-based deploy entrypoint and its supporting modules (ref resolution, collision detection, decryption, config rendering), run on the GitHub Actions runner +- `.github/actions/deploy/` - The push-based deploy entrypoint and its supporting modules (ref resolution, collision detection, decryption, config rendering), run on the GitHub Actions runner; `.github/actions/renovate/` is its narrower sibling (re-pull/recreate one-or-more apps' containers without touching versions) ### Docker Compose Architecture @@ -90,7 +90,7 @@ The repository uses a modular docker-compose structure with reusable components: Each app in `apps/` has: - `docker-compose.yml` extending common services - Optional `*.tpl` config files sitting directly next to `docker-compose.yml` (see "Config Templates" below) - - A `.env` on the target host only, decrypted and placed there by `deploy/deploy.py` (never checked into this repo, never present until a real deploy runs) + - A `.env` on the target host only, decrypted and placed there by `.github/actions/deploy/deploy.py` (never checked into this repo, never present until a real deploy runs) ### Healthchecks @@ -105,11 +105,11 @@ Standard shape used throughout the catalog: `start_period: 30s`, `interval: 30s` ### Environment Variable System -There is no root `.env` anywhere - not on a target host, not locally. Each app's env comes entirely from that app's own vault(s), declared in `targets/{target}.yml`'s `apps..env_refs` (see README's "Vaults And Targets"). `env_refs` is optional - an app with no vault-sourced values at all (e.g. `apps/beszel/docker-compose.yml`, which reads nothing but `APP_NAME`/`DATA_DIR`) just omits it, rather than pointing at a vault manifest with an empty `env:`. `deploy/deploy.py` runs on the GitHub Actions runner: it downloads each app's still-encrypted vault assets, checks their key names for collisions from the ciphertext directly (no decryption needed for that check), decrypts them with the target's private SOPS age key, concatenates the plaintext, and writes it straight into that app's `.env` in the release tree before pushing. `deploy/render.py`'s `render_template` then does the same substitution `envsubst` would, also on the runner, for that app's `*.tpl` files, using the just-decrypted values (see "Config Templates" below). +There is no root `.env` anywhere - not on a target host, not locally. Each app's env comes entirely from that app's own vault(s), declared in `targets/{target}.yml`'s `apps..env_refs` (see README's "Vaults And Targets"). `env_refs` is optional - an app with no vault-sourced values at all (e.g. `apps/beszel/docker-compose.yml`, which reads nothing but `APP_NAME`/`DATA_DIR`) just omits it, rather than pointing at a vault manifest with an empty `env:`. `.github/actions/deploy/deploy.py` runs on the GitHub Actions runner: it downloads each app's still-encrypted vault assets, checks their key names for collisions from the ciphertext directly (no decryption needed for that check), decrypts them with the target's private SOPS age key, concatenates the plaintext, and writes it straight into that app's `.env` in the release tree before pushing. `.github/actions/deploy/render.py`'s `render_template` then does the same substitution `envsubst` would, also on the runner, for that app's `*.tpl` files, using the just-decrypted values (see "Config Templates" below). A vault declares the exact final variable name an app receives directly (e.g. `HTTP_PORT`, not `TRAEFIK_HTTP_PORT`) - there is no automatic prefix-stripping or filtering step anywhere. Variables for one app are never visible to another app, since each app's `.env` is built from that app's own vault(s) only. This allows running docker compose directly from the app folder without any `--env-file` flags while keeping app secrets scoped. -Two variables in every app's `.env` are never vault-sourced - `deploy/deploy.py` computes and writes them itself: +Two variables in every app's `.env` are never vault-sourced - `.github/actions/deploy/deploy.py` computes and writes them itself: - `APP_NAME` - just the app's own directory name (`traefik`, `rybbit`, ...). Written by `resolve_app_envs` on the runner, identical for every host a target deploys to, so it lives in the release tree's static `.env` like anything else. - `DATA_DIR` - the absolute, persistent data path for that app on the *target host* (`{base_path}/apps-data/{app}`). This can't be computed on the runner: `base_path` comes from a target's `path` (default `~/flightdeck`), and `~` only resolves once connected to a specific host (`deploy_to_host` reads `$HOME` over SSH) - and a target's `hosts:` can list more than one, potentially with different home directories. So `DATA_DIR` is appended to each app's already-pushed `.env` on the host itself, inside `deploy_to_host`'s per-app loop, after the release is extracted but before `docker compose` ever runs - never baked into the release tree the way `APP_NAME` is. @@ -122,7 +122,7 @@ Compose files reference `${DATA_DIR}` directly (`${DATA_DIR}/postgres:/var/lib/p The mechanism is pure naming convention, no manifest or registration needed - the same `.tpl` marker Terraform's `templatefile()` uses, as a terminal suffix (`codecov.yml.tpl`, same placement as Terraform's `user_data.tpl`). Note this means editors and GitHub's diff view won't apply YAML syntax highlighting to the template out of the box (they pick a language by the last extension, and `.tpl` isn't a registered one anywhere by default) - the rendered output (`codecov.yml`) isn't affected, only the template source. Configure a file association per editor if that matters to you (e.g. Zed's `file_types` setting). -Any file directly inside `apps/{app}/` (next to `docker-compose.yml`, no special subdirectory) matching `*.tpl` is a template. `deploy/deploy.py`'s `render_app_configs` finds them with a plain glob, substitutes with `deploy/render.py` (an `envsubst`-equivalent - `$VAR`/`${VAR}` only, no bash `${VAR:-default}` fallback syntax, missing variable becomes an empty string), and writes the result as a sibling file in the same directory with `.tpl` stripped (`codecov.yml.tpl` → `codecov.yml`), `chmod 600` since rendered output can carry secrets. This happens on the runner, before the release is archived, so the rendered file rides inside the release tar next to `.env` and is versioned with that release like everything else - never written directly onto the target host outside the atomic release/symlink-switch step. +Any file directly inside `apps/{app}/` (next to `docker-compose.yml`, no special subdirectory) matching `*.tpl` is a template. `.github/actions/deploy/deploy.py`'s `render_app_configs` finds them with a plain glob, substitutes with `.github/actions/deploy/render.py` (an `envsubst`-equivalent - `$VAR`/`${VAR}` only, no bash `${VAR:-default}` fallback syntax, missing variable becomes an empty string), and writes the result as a sibling file in the same directory with `.tpl` stripped (`codecov.yml.tpl` → `codecov.yml`), `chmod 600` since rendered output can carry secrets. This happens on the runner, before the release is archived, so the rendered file rides inside the release tar next to `.env` and is versioned with that release like everything else - never written directly onto the target host outside the atomic release/symlink-switch step. Compose files mount the rendered file by its plain relative path (`./codecov.yml:/config/codecov.yml`), one line per file - not a whole-directory mount - so it's obvious from the compose file alone which container path each config file lands at. `apps-data/{app}/` stays reserved for the opposite case: state a deploy must never regenerate (`acme.json`, database data directories) - never templated output. @@ -197,7 +197,7 @@ HTTP-01 (`httpChallenge`) is always configured and needs nothing from the vault ## Operations -There are no wrapper scripts and nothing runs them - starting, stopping, and restarting apps all happen by deploying (`deploy/deploy.py`, see "CI/CD" below). The external Docker networks `traefik` and `databases` are created idempotently by `deploy/deploy.py` on every deploy (derived from `apps/networks.yml`'s `external: true` entries), not by a separate first-run step. `apps-data/traefik/acme.json` isn't created by `deploy/deploy.py` at all - traefik's `docker-compose.yml` mounts `apps-data/{app}/` as a directory (not the file directly, which would make Docker create a directory in its place if the file doesn't exist yet), and Traefik creates `acme.json` inside it itself on first start, with the permissions it requires. +There are no wrapper scripts and nothing runs them - starting, stopping, and restarting apps all happen by deploying (`.github/actions/deploy/deploy.py`, see "CI/CD" below). The external Docker networks `traefik` and `databases` are created idempotently by `.github/actions/deploy/deploy.py` on every deploy (derived from `apps/networks.yml`'s `external: true` entries), not by a separate first-run step. `apps-data/traefik/acme.json` isn't created by `.github/actions/deploy/deploy.py` at all - traefik's `docker-compose.yml` mounts `apps-data/{app}/` as a directory (not the file directly, which would make Docker create a directory in its place if the file doesn't exist yet), and Traefik creates `acme.json` inside it itself on first start, with the permissions it requires. Debugging an already-deployed app means SSHing into the target host directly and using Docker Compose itself - no wrapper needed, since each app's directory is already a complete, ready-to-run Compose project (real `.env` sitting next to the compose file): @@ -389,14 +389,14 @@ GitHub Actions workflow (`.github/workflows/release.yml`) manages releases via [ 1. On every push to `main`, Release Please opens/updates a `chore: release X.Y.Z` PR with the computed version and generated `CHANGELOG.md` entry 2. Merging that PR tags the release and publishes a GitHub Release -3. A job then builds and uploads `flightdeck-apps.zip` from the `apps/` catalog. Deploy refs may use `@latest` as an alias resolved through GitHub's latest release API (`deploy/resolve.py`); no mutable `latest` release/tag is created. +3. A job then builds and uploads `flightdeck-apps.zip` from the `apps/` catalog. Deploy refs may use `@latest` as an alias resolved through GitHub's latest release API (`.github/actions/deploy/resolve.py`); no mutable `latest` release/tag is created. -Deployment helpers live in this repository, entirely under `deploy/`, run only on the GitHub Actions runner - the target host never runs any of this: +Deployment helpers live in this repository, entirely under `.github/actions/deploy/` (and `.github/actions/renovate/` for its narrower sibling), run only on the GitHub Actions runner - the target host never runs any of this: -- `deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `*.tpl` files in place with the decrypted values (see "Config Templates" above). It writes a `manifest.json` into the release tree (`schema_version`, `release` timestamp, resolved `app_refs`/`env_refs` - the actual tag `resolve.py` pulled, never `@latest` - and the desired `apps` list; no secrets, no target identifier, since whoever's reading it is already on that specific host). It then opens an SSH connection per host, pushes the finished release as one tarball (real `.env`, already-rendered config, the manifest, all versioned together), bootstraps networks/directories idempotently, reads the *previous* release's `manifest.json` off `current` and `docker compose down`s any app present there but no longer in the desired set (using `current` before it moves, so that app's last-known compose file/`.env` are still intact - safe even though every release tree already contains every app's compose file regardless of whether the target wants it, since `build_release` copies the whole catalog every time), switches `current` to the new timestamped release, and runs `docker compose pull && docker compose up -d` per desired app directly (no wrapper script on the host at all). -- `deploy/resolve.py`, `deploy/collisions.py`, `deploy/vault.py`, and `deploy/render.py` hold, respectively, the ref-resolution, ciphertext collision-detection, decryption, and template-rendering logic - each with real `unittest` coverage in `deploy/tests/`. +- `.github/actions/deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `*.tpl` files in place with the decrypted values (see "Config Templates" above). It writes a `manifest.json` into the release tree (`schema_version`, `release` timestamp, resolved `app_refs`/`env_refs` - the actual tag `resolve.py` pulled, never `@latest` - and the desired `apps` list; no secrets, no target identifier, since whoever's reading it is already on that specific host). It then opens an SSH connection per host, pushes the finished release as one tarball (real `.env`, already-rendered config, the manifest, all versioned together), bootstraps networks/directories idempotently, reads the *previous* release's `manifest.json` off `current` and `docker compose down`s any app present there but no longer in the desired set (using `current` before it moves, so that app's last-known compose file/`.env` are still intact - safe even though every release tree already contains every app's compose file regardless of whether the target wants it, since `build_release` copies the whole catalog every time), switches `current` to the new timestamped release, and runs `docker compose pull && docker compose up -d` per desired app directly (no wrapper script on the host at all). +- `.github/actions/deploy/resolve.py`, `.github/actions/deploy/collisions.py`, `.github/actions/deploy/vault.py`, and `.github/actions/deploy/render.py` hold, respectively, the ref-resolution, ciphertext collision-detection, decryption, and template-rendering logic - each with real `unittest` coverage in `.github/actions/deploy/tests/`. - `.github/actions/encrypt-env/` is a local composite action for rendering `vaults/` manifests from GitHub Secrets/Variables, encrypting them for age recipients, and publishing `.sops.env` as a GitHub Release asset — vault manifests hold only env/secrets, not app selection -- `.github/workflows/deploy-shared.yml` is a reusable workflow consumer repos call to run `deploy/deploy.py` from GitHub Actions over an optional Tailscale connection, without holding any deploy secrets in this repository +- `.github/actions/deploy/` is a composite action consumer repos call to run `deploy.py` from GitHub Actions over an optional Tailscale connection, without holding any deploy secrets in this repository. It doesn't check anything out itself - `target-manifest` is read from whatever the caller's own preceding `actions/checkout` step already put on disk, and `deploy.py` itself comes along automatically via `$GITHUB_ACTION_PATH`. `.github/actions/renovate/` is the same shape, scoped to `renovate.py` instead. The `releases/{timestamp}`/`current` symlink pattern exists for atomicity, not for rollback: a deploy either fully lands and only then switches the symlink as its last step, or fails partway and leaves `current` untouched — never a partially-applied app. Rendered config templates are part of this same guarantee - they're written into the release tree and travel inside the release tarball, not pushed separately or in place. There is deliberately no automated rollback, and manual rollback (point `current` at an old release directory by hand) is not a supported/maintained path — it wouldn't restore a floating-tag image's historical version, and in practice fixing forward through the normal deploy path is simpler and safer than reasoning about what a partial rollback actually restores. @@ -409,10 +409,10 @@ Each app in a target's `apps` mapping may list its own `env_refs` — release re - **traefik**: Entry point, uses external network. Configured entirely via `TRAEFIK_*` env vars, no config file - see "Traefik Integration" above. - **cloudflared**: Optional Cloudflare Tunnel client, joins only the `traefik` network - see "Traefik Integration" above. - Apps with databases include a versioned template (e.g. `postgres-18.yml`) and create app-specific database named `${APP_NAME}` -- Config templates use `envsubst`-equivalent substitution (`deploy/render.py`) - variables must be shell-compatible (`${VAR}` syntax) +- Config templates use `envsubst`-equivalent substitution (`.github/actions/deploy/render.py`) - variables must be shell-compatible (`${VAR}` syntax) Watchtower was dropped entirely (there is no `apps/watchtower/`, and no app -carries its label) — `deploy/deploy.py` already runs `docker compose pull && +carries its label) — `.github/actions/deploy/deploy.py` already runs `docker compose pull && up -d` for every app in a target's `apps` mapping on every deploy, which made Watchtower's own polling redundant. See `RETIRED.md`. diff --git a/README.md b/README.md index ba9313f..6c2cb6c 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ A target server needs only: ## Automated Deploy -Deployment goes through [`deploy-shared.yml`](.github/workflows/deploy-shared.yml) (documented in the GitHub Actions section below), a reusable workflow wrapping [`deploy/deploy.py`](deploy/deploy.py) behind one input — `target-manifest`, a path to that target's own manifest file, which the workflow reads itself rather than receiving `hosts`/`app_refs`/`apps` already flattened. +Deployment goes through the [`deploy`](.github/actions/deploy) composite action (documented in the GitHub Actions section below) wrapping [`deploy.py`](.github/actions/deploy/deploy.py) behind one input — `target-manifest`, a path to that target's own manifest file, which the action reads itself rather than receiving `hosts`/`app_refs`/`apps` already flattened. It runs as a step inside a job the caller already checked its own repository out for, so the action needs no checkout logic of its own. The deploy is push-based and runs entirely on the GitHub Actions runner: @@ -58,25 +58,24 @@ flightdeck/ │ ├── postgres/ # PostgreSQL data │ └── {app-name}/ # Each app's data that must survive across releases │ -├── deploy/ -│ ├── deploy.py # Push-based deploy entrypoint (runs on the CI runner) -│ ├── renovate.py # Prototype: re-pull/recreate one app's containers, reading its target manifest directly -│ ├── resolve.py # owner/repo@tag[:asset] release ref resolution/download -│ ├── collisions.py # Ciphertext-based env key collision detection -│ ├── vault.py # SOPS decryption -│ └── render.py # envsubst-equivalent config template rendering ├── .github/ │ ├── actions/ │ │ ├── build-bundle/ # Build and upload a zip bundle from given paths │ │ ├── build-apps-bundle/ # Build and upload an apps/ catalog bundle │ │ ├── encrypt-env/ # Encrypt a target env and upload it to a release │ │ ├── load-vaults-matrix/ # Read vaults/ into a workflow matrix, no schema validation -│ │ └── load-targets-matrix/ # Same, but for targets/ specifically - validates the required shape +│ │ ├── load-targets-matrix/ # Same, but for targets/ specifically - validates the required shape +│ │ ├── deploy/ # Push-based deploy against a target manifest (deploy.py + friends) +│ │ │ ├── deploy.py # Deploy entrypoint (runs on the CI runner) +│ │ │ ├── resolve.py # owner/repo@tag[:asset] release ref resolution/download +│ │ │ ├── collisions.py # Ciphertext-based env key collision detection +│ │ │ ├── vault.py # SOPS decryption +│ │ │ └── render.py # envsubst-equivalent config template rendering +│ │ └── renovate/ # Re-pull/recreate one-or-more apps' containers (renovate.py) │ └── workflows/ -│ ├── deploy-shared.yml # Reusable deployment workflow -│ ├── renovate.yml # Prototype: computes a target matrix and calls renovate-shared.yml per target, like deploy.yml -│ ├── renovate-shared.yml # Prototype: reusable single-target renovation workflow, like deploy-shared.yml -│ └── release.yml # Release Please + publish Flightdeck assets +│ ├── deploy.yml # Manual redeploy of one target +│ ├── renovate.yml # Manual renovate of one-or-more apps across every matching target +│ └── release.yml # Release Please + publish Flightdeck assets + auto-deploy │ ├── vaults/ # Encrypted env asset configurations, one per app └── targets/ # Deployment targets @@ -113,7 +112,7 @@ Variable names inside a compose file are always bare, never prefixed with the ap - **internal** - Isolated network for app-to-app communication - **databases** - Dedicated network for database services (PostgreSQL, Redis, MongoDB) -`traefik` and `databases` are created on the target host by `deploy/deploy.py` (derived from `apps/networks.yml`'s `external: true` entries); `internal` is created by Docker Compose itself. +`traefik` and `databases` are created on the target host by [`deploy.py`](.github/actions/deploy/deploy.py) (derived from `apps/networks.yml`'s `external: true` entries); `internal` is created by Docker Compose itself. ## Adding a New Application @@ -188,7 +187,7 @@ Useful as a source of ready-made Docker Compose definitions when adding a new ap ## GitHub Actions -This repository provides five composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, `load-vaults-matrix`, and `load-targets-matrix`) and one reusable workflow, `deploy-shared.yml`. +This repository provides seven composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, `load-vaults-matrix`, `load-targets-matrix`, `deploy`, and `renovate`) — no reusable workflows. A reusable workflow only pays for itself when the reused thing genuinely needs its own multiple jobs or job-level config (`permissions`, `concurrency`, etc.); `deploy`/`renovate` are each a single job's worth of steps, and a composite action gets that for free without a `workflow_call` boundary's checkout/secrets ceremony (see [`deploy`](#deploy) below for what that ceremony would otherwise cost). --- @@ -241,11 +240,11 @@ sops_age_key_secret: MAINFRAME_AGE_PRIVATE_KEY `ssh_private_key_secret`/`sops_age_key_secret` are GitHub Secret *names*, never the credential values themselves - the `_secret` suffix says so explicitly, since a flat field like `sops_age_key` could otherwise read as the key material itself. `sops_age_key_secret` names the GitHub Secret holding this target's *private* age key - the one used to decrypt its vaults, matching the public key in `keys/.pub` used to encrypt them. Tailscale credentials live outside the target manifest entirely (`vars.TAILSCALE_OAUTH_CLIENT_ID`/`secrets.TAILSCALE_OAUTH_SECRET`, referenced directly by the workflows below) since the tailnet is shared infrastructure, not something that varies per target. -`app_refs` and `hosts` are YAML arrays; `apps` is a mapping from app name to that app's own `env_refs` array. Each host uses the SSH `user@host` format. `app_refs` must list at least one app bundle — flightdeck's own `apps/` catalog is just another entry, not implicit. `env_refs` is optional — omit it (or leave it `[]`) for an app that genuinely needs zero vault-sourced values (e.g. `beszel` above); it still gets a `.env` with `APP_NAME`/`DATA_DIR`, just no vault is fetched or decrypted for it. Don't create a vault manifest with an empty `env:` just to satisfy this field - there's nothing to encrypt, so there's nothing to gain from one. When `env_refs` is given, it must be non-empty; `deploy/deploy.py` decrypts and concatenates all of an app's sources into that app's own `.env` on the runner, failing loud on any key collision — but only within that one app's own sources. Two different apps' vaults sharing a key (e.g. both declaring `DOMAIN`) is expected, since each app gets a separate `.env`. +`app_refs` and `hosts` are YAML arrays; `apps` is a mapping from app name to that app's own `env_refs` array. Each host uses the SSH `user@host` format. `app_refs` must list at least one app bundle — flightdeck's own `apps/` catalog is just another entry, not implicit. `env_refs` is optional — omit it (or leave it `[]`) for an app that genuinely needs zero vault-sourced values (e.g. `beszel` above); it still gets a `.env` with `APP_NAME`/`DATA_DIR`, just no vault is fetched or decrypted for it. Don't create a vault manifest with an empty `env:` just to satisfy this field - there's nothing to encrypt, so there's nothing to gain from one. When `env_refs` is given, it must be non-empty; [`deploy.py`](.github/actions/deploy/deploy.py) decrypts and concatenates all of an app's sources into that app's own `.env` on the runner, failing loud on any key collision — but only within that one app's own sources. Two different apps' vaults sharing a key (e.g. both declaring `DOMAIN`) is expected, since each app gets a separate `.env`. A vault manifest's `env:` value is either `${NAME}` (a reference — look up the GitHub Secret/Variable named `NAME`) or a bare literal (any other value, used as-is with no lookup at all — see `DISABLE_SIGNUP: true` above). Use a literal for a value that's fixed for this target but isn't a secret and doesn't need a GitHub Secret/Variable to exist just to hold it. -[`load-vaults-matrix`](.github/actions/load-vaults-matrix) reads every file in `vaults/` into a matrix — it does not validate the manifest shape; `encrypt-env` re-parses and validates its own manifest from `manifest` (see "`encrypt-env`" below). Targets go through the more specific [`load-targets-matrix`](.github/actions/load-targets-matrix) instead, which *does* validate the shape above (`hosts`, `app_refs`, `apps`, `ssh_private_key_secret`, `sops_age_key_secret` all required) before a broken manifest ever reaches a checkout+dependency-install on a different job entirely. The workflows calling `deploy-shared.yml`/`renovate-shared.yml` then pull `matrix.ssh_private_key_secret`/`matrix.sops_age_key_secret` directly, to resolve actual secret values by name. +[`load-vaults-matrix`](.github/actions/load-vaults-matrix) reads every file in `vaults/` into a matrix — it does not validate the manifest shape; `encrypt-env` re-parses and validates its own manifest from `manifest` (see "`encrypt-env`" below). Targets go through the more specific [`load-targets-matrix`](.github/actions/load-targets-matrix) instead, which *does* validate the shape above (`hosts`, `app_refs`, `apps`, `ssh_private_key_secret`, `sops_age_key_secret` all required) before a broken manifest ever reaches a checkout+dependency-install on a different job entirely. The workflows calling the [`deploy`](#deploy)/[`renovate`](#renovate) actions then pull `matrix.ssh_private_key_secret`/`matrix.sops_age_key_secret` directly, to resolve actual secret values by name. --- @@ -331,50 +330,48 @@ Requires `contents: write` permission on the calling job. `flightdeck-apps.zip` --- -### `deploy-shared.yml` +### `deploy` -Runs [`deploy/deploy.py`](deploy/deploy.py) from this repository against a target manifest owned by the caller. Intended to be called from a private consumer repository that owns both the config and secrets side (a `targets/*.yml` manifest shaped like the one above, the SSH key, encrypted `.sops.env` releases, the age private key, etc.) — this repository does not hold any deploy secrets itself. That manifest's `apps..env_refs` entries typically reference that same calling repository via `${{ github.repository }}`, since it's both the config and secrets source. +Runs [`deploy.py`](.github/actions/deploy/deploy.py) against a target manifest already present in the caller's own checkout. Intended to be used from a private consumer repository that owns both the config and secrets side (a `targets/*.yml` manifest shaped like the one above, the SSH key, encrypted `.sops.env` releases, the age private key, etc.) — this repository does not hold any deploy secrets itself. That manifest's `apps..env_refs` entries typically reference that same calling repository via `${{ github.repository }}`, since it's both the config and secrets source. -The interface is a single path, not flattened deploy vocabulary — the caller never re-serializes its target's `hosts`/`app_refs`/`apps`/`path` through `toJson(...)`, and `deploy.py` never receives them as separate fields. `target-manifest` just points at the file (`targets/mainframe.yml` in the example below); the workflow checks out the caller's own repository to read it, parses and validates it itself, and pipes the result to `python3 deploy/deploy.py` on stdin alongside the one thing that genuinely can't live in that file - the decrypted `sops-age-key` secret value. The runner then resolves and downloads every ref, decrypts and renders each app's env and config, merges the release, and pushes the finished result to each host over SSH — see "Automated Deploy" above for the full sequence. +This used to be a `workflow_call` reusable workflow (`deploy-shared.yml`), which needed its own second `checkout` (`repository: job.workflow_repository, ref: job.workflow_sha`) just to get *this* repository's own `deploy.py` onto the runner, and a `target-manifest-ref` input to keep that checkout's ref in sync with whatever ref the caller had already resolved its matrix from. As a composite action, neither problem exists: `deploy.py` comes along automatically via `$GITHUB_ACTION_PATH` whenever this is referenced as `owner/repo/.github/actions/deploy@ref`, and `target-manifest` is just read from whatever the caller's own preceding `actions/checkout` step already put on disk — no second checkout, no ref to keep in sync, because there's only ever one checkout to begin with. -`target-manifest-ref` matters only when the caller itself checked out something other than its default branch before computing the matrix this is called from (e.g. `release.yml` pins to the just-published release tag) - set it to that same ref so both reads agree on the manifest's exact content, instead of silently reading whatever the default branch's tip happens to be by the time this job runs. +The interface is a single path, not flattened deploy vocabulary — the caller never re-serializes its target's `hosts`/`app_refs`/`apps`/`path` through `toJson(...)`, and `deploy.py` never receives them as separate fields; it parses and validates the manifest itself. `sops-age-key`/`ssh-private-key` are the two things that genuinely can't live in that file - secret *values*, resolved by the caller from the manifest's own `sops_age_key_secret`/`ssh_private_key_secret` fields (GitHub Secret *names*) and passed in directly. The action then resolves and downloads every ref, decrypts and renders each app's env and config, merges the release, and pushes the finished result to each host over SSH — see "Automated Deploy" above for the full sequence. -Tailscale is optional, not a dependency of this workflow: set `tailscale-oauth-client-id` (and the matching `tailscale-oauth-secret`) to have the runner join a tailnet as an ephemeral node before deploying. Leave both unset to skip that step entirely — e.g. when the job already runs on a self-hosted runner with network access to the hosts, or reaches them some other way. +Tailscale is optional, not a dependency of this action: set `tailscale-oauth-client-id` (and the matching `tailscale-oauth-secret`) to have the runner join a tailnet as an ephemeral node before deploying. Leave both unset to skip that step entirely — e.g. when the job already runs on a self-hosted runner with network access to the hosts, or reaches them some other way. ```yaml jobs: deploy: - uses: rubykatzen/flightdeck/.github/workflows/deploy-shared.yml@v0.11.1 - with: - target-manifest: targets/mainframe.yml # required, path in this repository - # target-manifest-ref: ${{ github.sha }} # optional, default: this repository's default branch - tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) - tailscale-tags: tag:ci # default: tag:ci - secrets: - ssh-private-key: ${{ secrets.DEPLOY_SSH_PRIVATE_KEY }} - sops-age-key: ${{ secrets.MAINFRAME_AGE_PRIVATE_KEY }} - tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} # optional, required only if tailscale-oauth-client-id is set + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: rubykatzen/flightdeck/.github/actions/deploy@v0.11.1 + with: + target-manifest: targets/mainframe.yml # required, path in this repository + ssh-private-key: ${{ secrets.DEPLOY_SSH_PRIVATE_KEY }} + sops-age-key: ${{ secrets.MAINFRAME_AGE_PRIVATE_KEY }} + github-token: ${{ secrets.GITHUB_TOKEN }} + # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) + # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} # required only if tailscale-oauth-client-id is set + # tailscale-tags: tag:ci # default: tag:ci ``` -The `@v0.11.1` pin on the `uses:` line only controls which ref runs `deploy/deploy.py` itself. `target-manifest`'s own `app_refs` entries are separate and don't have to match the workflow pin. +The `@v0.11.1` pin on the `uses:` line only controls which ref this action's own code runs at. `target-manifest`'s own `app_refs` entries are separate and don't have to match it. --- -### `renovate.yml` / `renovate-shared.yml` (prototype) - -**Experimental — kept around to compare against `deploy.yml`/`deploy-shared.yml`'s approach before settling on one contract for #120/#121.** Same two-job matrix shape as deploy: `renovate.yml` computes a matrix from `targets/` itself (via [`load-targets-matrix`](.github/actions/load-targets-matrix)) and calls `renovate-shared.yml` once per target, with that target's SSH key already picked out by name (`secrets[matrix.ssh_private_key_secret]`) — exactly like `deploy.yml`/`deploy-shared.yml` already do. No `secrets: inherit`, and deliberately so: it [only works within the same organization or enterprise as the reusable workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows), which would silently break for exactly the external, unrelated callers this is meant to support (confirmed real case: `dupmachine/flightdeck`). Resolving each matrix cell's secret *value* by name instead happens in `renovate.yml` itself (a plain triggered workflow, not a `workflow_call` boundary, so it has native access to `secrets.*`), and only that one resolved value ever crosses into `renovate-shared.yml`, via its own explicitly declared `on.workflow_call.secrets`. Tailscale credentials aren't target-specific at all (one shared tailnet), so they're referenced directly (`vars.TAILSCALE_OAUTH_CLIENT_ID`/`secrets.TAILSCALE_OAUTH_SECRET`) rather than resolved per matrix cell. - -Like `deploy-shared.yml`, `renovate.yml` passes only `target-manifest: ${{ matrix.manifest }}` — the file path `load-targets-matrix` already put in every matrix item — and `renovate-shared.yml` reads that file itself, the same way `encrypt-env` takes a vault manifest *path* and parses it rather than receiving flattened `env` fields, and `deploy-shared.yml` now does the same for `hosts`/`apps`/`path`. This keeps the wired interface down to `apps` + one path + Tailscale wiring, instead of re-serializing a target's whole shape through `toJson(matrix.X)` on every field. The trade-off: `renovate-shared.yml` needs a second `checkout` (the caller's own repository, to actually read that manifest file - the existing override checkout only ever fetches flightdeck's own code, into `.flightdeck/`, since that's the repo `deploy/renovate.py` itself lives in) — same shape `deploy-shared.yml` uses. +### `renovate` -`apps` is a JSON array, so one run can renovate several apps at once (e.g. a nightly cron renovating `["traefik","rybbit"]` while leaving everything else alone) — pass a single-element array for the one-app case. Not every target runs every requested app, so every target's job still gets dispatched (a matrix job calling a reusable workflow via `uses:` can't condition its `if:` on `matrix.*` - only `github`, `inputs`, `needs`, and `vars` are available there), each with that target's own resolved SSH key loaded. [`deploy/renovate.py`](deploy/renovate.py) is what actually decides: it renovates whichever requested apps *are* keys in the target manifest's own `apps` mapping and skips the rest; if none match, it exits before ever opening an SSH connection to a host - the "wasted" work per non-matching target is just two checkouts, a pip install, and loading a key into the runner's local SSH agent, never an actual connection anywhere. +Re-pulls and recreates one-or-more apps' containers on a target already present in the caller's own checkout, without touching versions — no new app bundle, no new vault-sourced env, no rebuilt release tree. Just `docker compose pull && docker compose up -d` per requested app, against its already-current release, on each of the target's hosts. Sibling to [`deploy`](#deploy), same checkout-free shape, deliberately narrower job. -Renovating means `docker compose pull && docker compose up -d` against each matched app's already-current release directory on the host — nothing else. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), `deploy/renovate.py` compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts`/`target_name` (derived from the manifest's own filename) via `$GITHUB_OUTPUT` - `updated_hosts` lists `app@host` pairs, since more than one app may have been renovated in the same run. When `updated` is `true`, the job's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. +`apps` is a JSON array, so one run can renovate several apps at once (e.g. a nightly cron renovating `["traefik","rybbit"]` while leaving everything else alone) — pass a single-element array for the one-app case. Not every target runs every requested app; [`renovate.py`](.github/actions/renovate/renovate.py) decides that itself from the target manifest's own `apps` mapping and simply does nothing — never opening an SSH connection — if none of the requested apps are present there. -**Known gap:** if none of the requested `apps` match any target at all (a typo, say), every matrix job is just skipped and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. Deferred until this contract shape is the chosen one. +It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), it compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts`/`target_name` (derived from the manifest's own filename) via `$GITHUB_OUTPUT` - `updated_hosts` lists `app@host` pairs, since more than one app may have been renovated in the same run. When `updated` is `true`, the action's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. ```yaml jobs: @@ -383,18 +380,22 @@ jobs: if: needs.find-targets.outputs.count != '0' strategy: matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} - uses: $/.github/workflows/renovate-shared.yml - with: - apps: ${{ inputs.apps }} # JSON array, e.g. '["traefik","rybbit"]' - target-manifest: ${{ matrix.manifest }} - tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} - secrets: - ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} - tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} - telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} - telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: rubykatzen/flightdeck/.github/actions/renovate@v0.11.0 + with: + apps: ${{ inputs.apps }} # JSON array, e.g. '["traefik","rybbit"]' + target-manifest: ${{ matrix.manifest }} + ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} + # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} + # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} ``` +**Known gap:** if none of the requested `apps` match any target at all (a typo, say), every matrix job just does nothing and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. `apps`' "all/single/list" selection and scheduled/allowlist-driven runs (see #168) also aren't implemented yet — this only ever renovates exactly the apps it's given. + ## License Flightdeck is released under the [MIT License](LICENSE). From 4ab075529d4e3b90c301d4f28f735c0a5268be84 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:55:51 +0200 Subject: [PATCH 13/22] refactor: rename renovate.yml's find-targets job to load-targets Matches its step id (already load-targets) and deploy.yml/release.yml's own job naming for the equivalent step (deploy-targets loads targets/ too, just paired with a different verb since it goes on to deploy them). --- .github/actions/renovate/README.md | 2 +- .github/workflows/renovate.yml | 8 ++++---- README.md | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/actions/renovate/README.md b/.github/actions/renovate/README.md index 856f5b9..cc81896 100644 --- a/.github/actions/renovate/README.md +++ b/.github/actions/renovate/README.md @@ -8,7 +8,7 @@ Composite GitHub Action that re-pulls and recreates one-or-more apps' containers jobs: renovate: strategy: - matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} + matrix: ${{ fromJson(needs.load-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index ad96edb..aa8cb91 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -7,7 +7,7 @@ on: type: string required: true jobs: - find-targets: + load-targets: runs-on: ubuntu-latest outputs: matrix: ${{ steps.load-targets.outputs.matrix }} @@ -17,10 +17,10 @@ jobs: - uses: $/.github/actions/load-targets-matrix id: load-targets renovate: - needs: find-targets - if: needs.find-targets.outputs.count != '0' + needs: load-targets + if: needs.load-targets.outputs.count != '0' strategy: - matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} + matrix: ${{ fromJson(needs.load-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 diff --git a/README.md b/README.md index 6c2cb6c..eeb4eb6 100644 --- a/README.md +++ b/README.md @@ -376,10 +376,10 @@ It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuild ```yaml jobs: renovate: - needs: find-targets - if: needs.find-targets.outputs.count != '0' + needs: load-targets + if: needs.load-targets.outputs.count != '0' strategy: - matrix: ${{ fromJson(needs.find-targets.outputs.matrix) }} + matrix: ${{ fromJson(needs.load-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 From 0b784e7cee68c41fe1b716f677c65eccfb0e97ee Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:57:12 +0200 Subject: [PATCH 14/22] refactor: rename deploy-targets job to load-targets in deploy.yml/release.yml Matches its step id (already load-targets) and renovate.yml's own equivalent job, now that "deploy" as a verb belongs to the job that actually deploys, not the one that just loads the target matrix. --- .github/actions/deploy/README.md | 2 +- .github/workflows/deploy.yml | 8 ++++---- .github/workflows/release.yml | 8 ++++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/actions/deploy/README.md b/.github/actions/deploy/README.md index 98bea06..995f163 100644 --- a/.github/actions/deploy/README.md +++ b/.github/actions/deploy/README.md @@ -8,7 +8,7 @@ Composite GitHub Action that runs a push-based deploy against a target manifest jobs: deploy: strategy: - matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} + matrix: ${{ fromJson(needs.load-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 0b6d515..ed0f01f 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -7,7 +7,7 @@ on: type: string required: true jobs: - deploy-targets: + load-targets: runs-on: ubuntu-latest outputs: matrix: ${{ steps.load-targets.outputs.matrix }} @@ -19,10 +19,10 @@ jobs: with: name: ${{ inputs.target }} deploy: - needs: deploy-targets - if: needs.deploy-targets.outputs.count != '0' + needs: load-targets + if: needs.load-targets.outputs.count != '0' strategy: - matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} + matrix: ${{ fromJson(needs.load-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d9c6f9d..2e6ef04 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -60,7 +60,7 @@ jobs: env: GITHUB_SECRETS_JSON: ${{ toJson(secrets) }} GITHUB_VARS_JSON: ${{ toJson(vars) }} - deploy-targets: + load-targets: needs: release if: needs.release.outputs.release_created == 'true' runs-on: ubuntu-latest @@ -74,10 +74,10 @@ jobs: - uses: $/.github/actions/load-targets-matrix id: load-targets deploy: - needs: [release, upload-apps, encrypt, deploy-targets] - if: needs.deploy-targets.outputs.count != '0' + needs: [release, upload-apps, encrypt, load-targets] + if: needs.load-targets.outputs.count != '0' strategy: - matrix: ${{ fromJson(needs.deploy-targets.outputs.matrix) }} + matrix: ${{ fromJson(needs.load-targets.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 From 739d80c4d765eff9c8420b6cdb1521f69373dc79 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 22:58:05 +0200 Subject: [PATCH 15/22] refactor: rename encrypt-vaults job to load-vaults in release.yml Matches load-targets: the job's own work is loading vaults/ into a matrix, not encrypting anything - that's the separate encrypt job that consumes it. Step id renamed from the generic "matrix" to load-vaults to match, same as load-targets-matrix's own step id. --- .github/workflows/release.yml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 2e6ef04..5692406 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -29,24 +29,24 @@ jobs: with: release-tag: ${{ needs.release.outputs.tag_name }} token: ${{ secrets.GITHUB_TOKEN }} - encrypt-vaults: + load-vaults: needs: release if: needs.release.outputs.release_created == 'true' runs-on: ubuntu-latest outputs: - matrix: ${{ steps.matrix.outputs.matrix }} - count: ${{ steps.matrix.outputs.count }} + matrix: ${{ steps.load-vaults.outputs.matrix }} + count: ${{ steps.load-vaults.outputs.count }} steps: - uses: actions/checkout@v7 with: ref: ${{ needs.release.outputs.tag_name }} - uses: $/.github/actions/load-vaults-matrix - id: matrix + id: load-vaults encrypt: - needs: [release, encrypt-vaults] - if: needs.encrypt-vaults.outputs.count != '0' + needs: [release, load-vaults] + if: needs.load-vaults.outputs.count != '0' strategy: - matrix: ${{ fromJson(needs.encrypt-vaults.outputs.matrix) }} + matrix: ${{ fromJson(needs.load-vaults.outputs.matrix) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 From 9c0d4a7c58bd0c8884986e546efb00f150c71dfc Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 23:06:19 +0200 Subject: [PATCH 16/22] refactor: drop --user --break-system-packages from deploy/renovate's pip install Leftover from an old ansible-core install step, copy-pasted forward unchanged ever since - the actions written later (encrypt-env, load-targets-matrix, load-vaults-matrix) already use the correct form, `python3 -m pip install`, which targets the exact interpreter that later runs the script rather than whatever bare `pip` resolves to. That's already proven to work on this runner without needing to override PEP 668's externally-managed-environment protection at all. --- .github/actions/deploy/action.yml | 2 +- .github/actions/renovate/action.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/actions/deploy/action.yml b/.github/actions/deploy/action.yml index de5a65c..760c524 100644 --- a/.github/actions/deploy/action.yml +++ b/.github/actions/deploy/action.yml @@ -30,7 +30,7 @@ runs: steps: - name: Install deploy dependencies shell: bash - run: pip install --user --break-system-packages --requirement "$GITHUB_ACTION_PATH/requirements.txt" + run: python3 -m pip install --disable-pip-version-check --requirement "$GITHUB_ACTION_PATH/requirements.txt" - name: Install sops shell: bash env: diff --git a/.github/actions/renovate/action.yml b/.github/actions/renovate/action.yml index 7d3a438..5bbe99c 100644 --- a/.github/actions/renovate/action.yml +++ b/.github/actions/renovate/action.yml @@ -33,7 +33,7 @@ runs: steps: - name: Install deploy dependencies shell: bash - run: pip install --user --break-system-packages --requirement "$GITHUB_ACTION_PATH/requirements.txt" + run: python3 -m pip install --disable-pip-version-check --requirement "$GITHUB_ACTION_PATH/requirements.txt" - uses: tailscale/github-action@v4 if: inputs.tailscale-oauth-client-id != '' with: From 035f3710f1c379e9c1226dc7a36865307d466dcf Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 23:13:28 +0200 Subject: [PATCH 17/22] refactor: move Telegram notification out of the renovate action The action now only reports what happened (updated/updated-hosts/ target-name as proper action outputs) instead of hard-requiring telegram-bot-token/telegram-chat-id and sending the message itself. Notifying is renovate.yml's own job now, as a separate step reading those outputs - matches deploy, which never had any notification logic of its own, and means a different caller can wire up a different channel (or none) without forking this action. --- .github/actions/renovate/README.md | 14 +++++++++++--- .github/actions/renovate/action.yml | 23 ++++++++++------------- .github/workflows/renovate.yml | 6 ++++++ README.md | 14 +++++++++++--- 4 files changed, 38 insertions(+), 19 deletions(-) diff --git a/.github/actions/renovate/README.md b/.github/actions/renovate/README.md index cc81896..48d2bd5 100644 --- a/.github/actions/renovate/README.md +++ b/.github/actions/renovate/README.md @@ -13,16 +13,24 @@ jobs: steps: - uses: actions/checkout@v7 - uses: rubykatzen/flightdeck/.github/actions/renovate@main + id: renovate with: apps: ${{ inputs.apps }} # required, JSON array e.g. '["traefik","rybbit"]' target-manifest: ${{ matrix.manifest }} # required, path in this repository ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} - telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} - telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} # required only if tailscale-oauth-client-id is set + - name: Notify Telegram + if: steps.renovate.outputs.updated == 'true' + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Renovate: updated on ${{ steps.renovate.outputs.target-name }} (${{ steps.renovate.outputs.updated-hosts }})" + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} ``` -Not every target runs every requested app - `renovate.py` decides that itself, from the target manifest's own `apps` mapping, and simply does nothing (never opening an SSH connection) if none of the requested apps are present. To tell whether a host's image actually changed (rather than the pull being a no-op), it compares `docker compose images -q` output before and after the pull. Only sends the Telegram notification when at least one app actually changed - silent otherwise. +Not every target runs every requested app - `renovate.py` decides that itself, from the target manifest's own `apps` mapping, and simply does nothing (never opening an SSH connection) if none of the requested apps are present. To tell whether a host's image actually changed (rather than the pull being a no-op), it compares `docker compose images -q` output before and after the pull, and exposes that as this action's own `updated`/`updated-hosts`/`target-name` outputs. + +Unlike [`deploy`](../deploy) - which has no notification logic at all - this action still has none of its own either, on purpose: it only ever reports whether something changed. Sending a Telegram message (or anything else) is the *caller's* job, as a separate step reading these outputs, same as shown above - a different caller is free to wire up a different channel, or none, without forking this action. Same checkout model as [`deploy`](../deploy): this action never checks out anything itself, it just reads `target-manifest` from whatever the caller's own preceding `actions/checkout` step already put on disk. diff --git a/.github/actions/renovate/action.yml b/.github/actions/renovate/action.yml index 5bbe99c..0b240c0 100644 --- a/.github/actions/renovate/action.yml +++ b/.github/actions/renovate/action.yml @@ -10,12 +10,6 @@ inputs: ssh-private-key: description: SSH private key used to connect to the hosts. required: true - telegram-bot-token: - description: Telegram bot token used to notify when an update is found. - required: true - telegram-chat-id: - description: Telegram chat ID used to notify when an update is found. - required: true tailscale-oauth-client-id: description: Tailscale OAuth client ID used to join the tailnet. Leave unset to skip joining a tailnet (e.g. when the runner already has network access to the hosts). required: false @@ -28,6 +22,16 @@ inputs: description: Comma-separated ACL tags for the ephemeral tailnet node. required: false default: tag:ci +outputs: + updated: + description: "'true' if at least one requested app's image actually changed on this target." + value: ${{ steps.run.outputs.updated }} + updated-hosts: + description: Comma-separated app@host pairs that were actually updated. + value: ${{ steps.run.outputs.updated_hosts }} + target-name: + description: This target's name, derived from the manifest's own filename. + value: ${{ steps.run.outputs.target_name }} runs: using: composite steps: @@ -57,10 +61,3 @@ runs: jq -n --argjson apps "$APPS" --arg target_manifest "$TARGET_MANIFEST" \ '{apps: $apps, target_manifest: $target_manifest}' \ | python3 "$GITHUB_ACTION_PATH/renovate.py" - - name: Notify Telegram - if: steps.run.outputs.updated == 'true' - uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 - with: - message: "Renovate: updated on ${{ steps.run.outputs.target_name }} (${{ steps.run.outputs.updated_hosts }})" - telegram-bot-token: ${{ inputs.telegram-bot-token }} - telegram-chat-id: ${{ inputs.telegram-chat-id }} diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index aa8cb91..85aabaf 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -25,11 +25,17 @@ jobs: steps: - uses: actions/checkout@v7 - uses: $/.github/actions/renovate + id: renovate with: apps: ${{ inputs.apps }} target-manifest: ${{ matrix.manifest }} ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} + - name: Notify Telegram + if: steps.renovate.outputs.updated == 'true' + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Renovate: updated on ${{ steps.renovate.outputs.target-name }} (${{ steps.renovate.outputs.updated-hosts }})" telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/README.md b/README.md index eeb4eb6..07175ef 100644 --- a/README.md +++ b/README.md @@ -371,7 +371,9 @@ Re-pulls and recreates one-or-more apps' containers on a target already present `apps` is a JSON array, so one run can renovate several apps at once (e.g. a nightly cron renovating `["traefik","rybbit"]` while leaving everything else alone) — pass a single-element array for the one-app case. Not every target runs every requested app; [`renovate.py`](.github/actions/renovate/renovate.py) decides that itself from the target manifest's own `apps` mapping and simply does nothing — never opening an SSH connection — if none of the requested apps are present there. -It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), it compares `docker compose images -q` output before and after the pull, and reports `updated`/`updated_hosts`/`target_name` (derived from the manifest's own filename) via `$GITHUB_OUTPUT` - `updated_hosts` lists `app@host` pairs, since more than one app may have been renovated in the same run. When `updated` is `true`, the action's last step sends a Telegram message via `rubykatzen/baseline`'s generic `send-telegram-message` action (the same one `notify-telegram-release.yml`/`notify-telegram-pr.yml` use under the hood) — silent when nothing actually changed. +It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), it compares `docker compose images -q` output before and after the pull, and exposes `updated`/`updated-hosts`/`target-name` (derived from the manifest's own filename) as action outputs - `updated-hosts` lists `app@host` pairs, since more than one app may have been renovated in the same run. + +Unlike `deploy`, this action has no notification logic of its own — it just reports whether anything changed. `renovate.yml` below is what actually decides to notify, as its own separate step reading this action's outputs; a different caller is free to wire up a different channel, or none at all, without forking this action. ```yaml jobs: @@ -384,14 +386,20 @@ jobs: steps: - uses: actions/checkout@v7 - uses: rubykatzen/flightdeck/.github/actions/renovate@v0.11.0 + id: renovate with: apps: ${{ inputs.apps }} # JSON array, e.g. '["traefik","rybbit"]' target-manifest: ${{ matrix.manifest }} ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} - telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} - telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} + - name: Notify Telegram + if: steps.renovate.outputs.updated == 'true' + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Renovate: updated on ${{ steps.renovate.outputs.target-name }} (${{ steps.renovate.outputs.updated-hosts }})" + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} ``` **Known gap:** if none of the requested `apps` match any target at all (a typo, say), every matrix job just does nothing and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. `apps`' "all/single/list" selection and scheduled/allowlist-driven runs (see #168) also aren't implemented yet — this only ever renovates exactly the apps it's given. From 07975d43ced5df86c37ff7970ed61d9584096050 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 23:15:49 +0200 Subject: [PATCH 18/22] feat: notify Telegram after a successful deploy Same pattern just established for renovate: notification lives in the caller's own job, not baked into the deploy action. Unlike renovate, there's no "did anything change" gate needed - a deploy either fully lands or the job fails outright, so the Notify Telegram step runs unconditionally right after the deploy step, same as any other step that simply never executes if the one before it failed. release.yml's version also includes the release tag that was just deployed, since it already has that on hand from the release job. --- .github/actions/deploy/README.md | 8 ++++++++ .github/workflows/deploy.yml | 6 ++++++ .github/workflows/release.yml | 6 ++++++ README.md | 8 ++++++++ 4 files changed, 28 insertions(+) diff --git a/.github/actions/deploy/README.md b/.github/actions/deploy/README.md index 995f163..2e7350f 100644 --- a/.github/actions/deploy/README.md +++ b/.github/actions/deploy/README.md @@ -21,8 +21,16 @@ jobs: # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} # required only if tailscale-oauth-client-id is set # tailscale-tags: tag:ci # default: tag:ci + - name: Notify Telegram + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Deploy: ${{ matrix.name }} updated" + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} ``` Unlike a `workflow_call` reusable workflow, this action doesn't check out anything itself - it reads `target-manifest` from whatever the caller's own preceding `actions/checkout` step already put on disk, and its own code (`deploy.py` and friends) comes along automatically via `$GITHUB_ACTION_PATH` whenever it's referenced as `owner/repo/.github/actions/deploy@ref`. This is also why there's no `target-manifest-ref` input here: if the caller needs a specific ref (e.g. a just-published release tag), it just checks out that ref itself before this step runs, the same way every other job in this repository already does. `sops-age-key`/`ssh-private-key` are secret *values*, resolved by the caller from the target manifest's own `sops_age_key_secret`/`ssh_private_key_secret` fields (GitHub Secret *names* - see the main README's "Vaults And Targets" section) - this action never reads the manifest's credential fields itself, since it never has access to `secrets.*` by name. + +This action has no notification logic of its own, same as [`renovate`](../renovate) - notifying is the caller's job, as a plain following step. Unlike `renovate`, there's no "did anything change" output to gate it on: reaching that step at all already means the `deploy` step above it succeeded, so it just runs unconditionally. diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index ed0f01f..4e76238 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -34,3 +34,9 @@ jobs: github-token: ${{ secrets.GITHUB_TOKEN }} tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} + - name: Notify Telegram + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Deploy: ${{ matrix.name }} updated" + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 5692406..d6a5537 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -91,3 +91,9 @@ jobs: github-token: ${{ secrets.GITHUB_TOKEN }} tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} + - name: Notify Telegram + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Deploy: ${{ matrix.name }} updated to ${{ needs.release.outputs.tag_name }}" + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} diff --git a/README.md b/README.md index 07175ef..f4518a2 100644 --- a/README.md +++ b/README.md @@ -340,6 +340,8 @@ The interface is a single path, not flattened deploy vocabulary — the caller n Tailscale is optional, not a dependency of this action: set `tailscale-oauth-client-id` (and the matching `tailscale-oauth-secret`) to have the runner join a tailnet as an ephemeral node before deploying. Leave both unset to skip that step entirely — e.g. when the job already runs on a self-hosted runner with network access to the hosts, or reaches them some other way. +Like [`renovate`](#renovate), this action has no notification logic of its own - unlike `renovate`, a deploy doesn't need a "did anything actually change" check to decide whether to notify: reaching this point at all already means a real deploy just landed, so the caller's own following step notifies unconditionally (it simply never runs if the `deploy` step above it failed, same as any other step in a job). + ```yaml @@ -357,6 +359,12 @@ jobs: # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} # optional, default: unset (skip joining a tailnet) # tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} # required only if tailscale-oauth-client-id is set # tailscale-tags: tag:ci # default: tag:ci + - name: Notify Telegram + uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 + with: + message: "Deploy: mainframe updated" + telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} + telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} ``` From 47970678cf292547de8ab870759fde0d3c9698bb Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Fri, 28 Aug 2026 23:51:10 +0200 Subject: [PATCH 19/22] refactor!: drop deploy.yml's target input and load-*-matrix's name filter Single-target selection had exactly one consumer (deploy.yml's manual redeploy) and this repo has exactly one target with no second one planned - there was nothing left to disambiguate. deploy.yml now just redeploys everything, matching what release.yml's own deploy path already does; load-targets-matrix/load-vaults-matrix both drop the name input and its --name/selected filtering logic entirely, since neither has any remaining caller passing it. All 85 tests across deploy/renovate/load-targets-matrix/load-vaults- matrix pass unchanged in behavior for the "load everything" path - only the removed filtering path and its tests are gone. --- .github/actions/load-targets-matrix/README.md | 1 - .github/actions/load-targets-matrix/action.yml | 7 +------ .../scripts/load-targets-matrix.py | 10 ++-------- .../tests/test_load_targets_matrix.py | 12 ++---------- .github/actions/load-vaults-matrix/README.md | 1 - .github/actions/load-vaults-matrix/action.yml | 7 +------ .../load-vaults-matrix/scripts/load-vaults-matrix.py | 10 ++-------- .../tests/test_load_vaults_matrix.py | 12 ++---------- .github/workflows/deploy.yml | 9 +-------- README.md | 2 +- 10 files changed, 12 insertions(+), 59 deletions(-) diff --git a/.github/actions/load-targets-matrix/README.md b/.github/actions/load-targets-matrix/README.md index 6d24b7c..bfed0e9 100644 --- a/.github/actions/load-targets-matrix/README.md +++ b/.github/actions/load-targets-matrix/README.md @@ -9,7 +9,6 @@ Composite GitHub Action that reads every target manifest in a directory into a G id: load-targets with: # directory: targets # optional, default shown - # name: all # optional; single target name to load, default: all ``` The action exposes `matrix`, containing `{ "include": [...] }`, and `count`. Each matrix item merges the manifest's own top-level YAML fields with `name` (the file's basename) and `manifest` (its path). diff --git a/.github/actions/load-targets-matrix/action.yml b/.github/actions/load-targets-matrix/action.yml index 0e16bf5..27072d0 100644 --- a/.github/actions/load-targets-matrix/action.yml +++ b/.github/actions/load-targets-matrix/action.yml @@ -5,10 +5,6 @@ inputs: description: Directory containing target manifests. required: false default: targets - name: - description: Single target name to load, or all. - required: false - default: all outputs: matrix: description: JSON strategy matrix, one item per target manifest, merging its parsed YAML fields with name and manifest. @@ -27,6 +23,5 @@ runs: shell: bash env: DIRECTORY: ${{ inputs.directory }} - NAME: ${{ inputs.name }} run: | - python3 "$GITHUB_ACTION_PATH/scripts/load-targets-matrix.py" --directory "$DIRECTORY" --name "$NAME" + python3 "$GITHUB_ACTION_PATH/scripts/load-targets-matrix.py" --directory "$DIRECTORY" diff --git a/.github/actions/load-targets-matrix/scripts/load-targets-matrix.py b/.github/actions/load-targets-matrix/scripts/load-targets-matrix.py index b89c63b..bbded8d 100644 --- a/.github/actions/load-targets-matrix/scripts/load-targets-matrix.py +++ b/.github/actions/load-targets-matrix/scripts/load-targets-matrix.py @@ -58,13 +58,10 @@ def validate_target(name, manifest): raise ManifestError(f"target {name!r} must set sops_age_key_secret to a non-empty string") -def build_matrix(directory, selected="all"): +def build_matrix(directory): paths = sorted(directory.glob("*.yml")) + sorted(directory.glob("*.yaml")) if not paths: raise ManifestError(f"no manifests found in {directory}") - names = {path.stem for path in paths} - if selected != "all" and selected not in names: - raise ManifestError(f"unknown name: {selected}") include = [] seen_names = set() for path in paths: @@ -76,8 +73,6 @@ def build_matrix(directory, selected="all"): seen_names.add(name) manifest = load_manifest(path) validate_target(name, manifest) - if selected != "all" and name != selected: - continue item = {"name": name, "manifest": str(path)} item.update(manifest) include.append(item) @@ -94,10 +89,9 @@ def write_github_output(name, value): def main(argv=None): parser = argparse.ArgumentParser() parser.add_argument("--directory", required=True, type=Path) - parser.add_argument("--name", default="all") args = parser.parse_args(argv) try: - matrix = build_matrix(args.directory, args.name) + matrix = build_matrix(args.directory) encoded = json.dumps(matrix, separators=(",", ":")) write_github_output("matrix", encoded) write_github_output("count", len(matrix["include"])) diff --git a/.github/actions/load-targets-matrix/tests/test_load_targets_matrix.py b/.github/actions/load-targets-matrix/tests/test_load_targets_matrix.py index c1b603c..90c91c6 100644 --- a/.github/actions/load-targets-matrix/tests/test_load_targets_matrix.py +++ b/.github/actions/load-targets-matrix/tests/test_load_targets_matrix.py @@ -45,21 +45,13 @@ def test_builds_matrix_from_all_manifests(self): ) def test_merges_manifest_fields_with_name_and_manifest(self): - item = load_targets_matrix.build_matrix(self.directory, "heimdall")["include"][0] - self.assertEqual(item["name"], "heimdall") + matrix = load_targets_matrix.build_matrix(self.directory) + item = next(item for item in matrix["include"] if item["name"] == "heimdall") self.assertEqual(item["manifest"], str(self.directory / "heimdall.yml")) self.assertEqual(item["hosts"], ["root@100.75.53.13"]) self.assertEqual(item["ssh_private_key_secret"], "DEPLOY_SSH_PRIVATE_KEY") self.assertEqual(item["sops_age_key_secret"], "HEIMDALL_AGE_PRIVATE_KEY") - def test_filters_selected_manifest(self): - matrix = load_targets_matrix.build_matrix(self.directory, "heimdall") - self.assertEqual([item["name"] for item in matrix["include"]], ["heimdall"]) - - def test_rejects_unknown_name(self): - with self.assertRaisesRegex(load_targets_matrix.ManifestError, "unknown name"): - load_targets_matrix.build_matrix(self.directory, "missing") - def test_rejects_empty_directory(self): empty = self.directory / "empty" empty.mkdir() diff --git a/.github/actions/load-vaults-matrix/README.md b/.github/actions/load-vaults-matrix/README.md index f819e14..55c07d7 100644 --- a/.github/actions/load-vaults-matrix/README.md +++ b/.github/actions/load-vaults-matrix/README.md @@ -9,7 +9,6 @@ Composite GitHub Action that reads every vault manifest in `vaults/` into a GitH id: matrix with: # directory: vaults # optional, default shown - # name: all # optional; single manifest name to load, default: all ``` The action exposes `matrix`, containing `{ "include": [...] }`, and `count`. Each matrix item merges the manifest's own top-level YAML fields with `name` (the file's basename) and `manifest` (its path). diff --git a/.github/actions/load-vaults-matrix/action.yml b/.github/actions/load-vaults-matrix/action.yml index 09d0363..3dcf3e2 100644 --- a/.github/actions/load-vaults-matrix/action.yml +++ b/.github/actions/load-vaults-matrix/action.yml @@ -5,10 +5,6 @@ inputs: description: Directory containing vault manifests. required: false default: vaults - name: - description: Single manifest name to load, or all. - required: false - default: all outputs: matrix: description: JSON strategy matrix, one item per manifest, merging its parsed YAML fields with name and manifest. @@ -27,6 +23,5 @@ runs: shell: bash env: DIRECTORY: ${{ inputs.directory }} - NAME: ${{ inputs.name }} run: | - python3 "$GITHUB_ACTION_PATH/scripts/load-vaults-matrix.py" --directory "$DIRECTORY" --name "$NAME" + python3 "$GITHUB_ACTION_PATH/scripts/load-vaults-matrix.py" --directory "$DIRECTORY" diff --git a/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py b/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py index de13c58..e30967c 100644 --- a/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py +++ b/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py @@ -45,13 +45,10 @@ def load_manifest(path): return value -def build_matrix(directory, selected="all"): +def build_matrix(directory): paths = sorted(directory.glob("*.yml")) + sorted(directory.glob("*.yaml")) if not paths: raise ManifestError(f"no manifests found in {directory}") - names = {path.stem for path in paths} - if selected != "all" and selected not in names: - raise ManifestError(f"unknown name: {selected}") include = [] seen_names = set() for path in paths: @@ -62,8 +59,6 @@ def build_matrix(directory, selected="all"): raise ManifestError(f"duplicate manifest name: {name}") seen_names.add(name) manifest = load_manifest(path) - if selected != "all" and name != selected: - continue item = {"name": name, "manifest": str(path)} item.update(manifest) include.append(item) @@ -80,10 +75,9 @@ def write_github_output(name, value): def main(argv=None): parser = argparse.ArgumentParser() parser.add_argument("--directory", required=True, type=Path) - parser.add_argument("--name", default="all") args = parser.parse_args(argv) try: - matrix = build_matrix(args.directory, args.name) + matrix = build_matrix(args.directory) encoded = json.dumps(matrix, separators=(",", ":")) write_github_output("matrix", encoded) write_github_output("count", len(matrix["include"])) diff --git a/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py b/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py index 23811d8..7002b7d 100644 --- a/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py +++ b/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py @@ -42,20 +42,12 @@ def test_builds_matrix_from_all_manifests(self): ) def test_merges_manifest_fields_with_name_and_manifest(self): - item = load_vaults_matrix.build_matrix(self.directory, "traefik")["include"][0] - self.assertEqual(item["name"], "traefik") + matrix = load_vaults_matrix.build_matrix(self.directory) + item = next(item for item in matrix["include"] if item["name"] == "traefik") self.assertEqual(item["manifest"], str(self.directory / "traefik.yml")) self.assertEqual(item["asset"], "mainframe-traefik.sops.env") self.assertEqual(item["keys"], ["mainframe"]) - def test_filters_selected_manifest(self): - matrix = load_vaults_matrix.build_matrix(self.directory, "traefik") - self.assertEqual([item["name"] for item in matrix["include"]], ["traefik"]) - - def test_rejects_unknown_name(self): - with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "unknown name"): - load_vaults_matrix.build_matrix(self.directory, "missing") - def test_rejects_empty_directory(self): empty = self.directory / "empty" empty.mkdir() diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 4e76238..3cb46a1 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -1,11 +1,6 @@ name: Deploy on: - workflow_dispatch: - inputs: - target: - description: Deploy target to redeploy. - type: string - required: true + workflow_dispatch: {} jobs: load-targets: runs-on: ubuntu-latest @@ -16,8 +11,6 @@ jobs: - uses: actions/checkout@v7 - uses: $/.github/actions/load-targets-matrix id: load-targets - with: - name: ${{ inputs.target }} deploy: needs: load-targets if: needs.load-targets.outputs.count != '0' diff --git a/README.md b/README.md index f4518a2..f223b55 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ flightdeck/ │ │ │ └── render.py # envsubst-equivalent config template rendering │ │ └── renovate/ # Re-pull/recreate one-or-more apps' containers (renovate.py) │ └── workflows/ -│ ├── deploy.yml # Manual redeploy of one target +│ ├── deploy.yml # Manual redeploy of every target │ ├── renovate.yml # Manual renovate of one-or-more apps across every matching target │ └── release.yml # Release Please + publish Flightdeck assets + auto-deploy │ From e7eb5f1e414429bddaff2f8823f5ef414773a506 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Sat, 29 Aug 2026 00:50:01 +0200 Subject: [PATCH 20/22] refactor: have release.yml call deploy.yml instead of duplicating it deploy.yml lost its target input in an earlier commit - it now always redeploys every target, exactly matching what release.yml's own load-targets+deploy job pair already did. That made the two genuinely identical operations, not just similar ones, so the duplication is worth removing: deploy.yml adds workflow_call (no inputs, same as its workflow_dispatch trigger) and release.yml calls it directly with secrets: inherit instead of running its own copy. secrets: inherit is fine here specifically because both workflows live in this same repository - deploy.yml already has native access to every one of this repo's own secrets when triggered directly, so inheriting them from release.yml doesn't broaden anything. No organization boundary is being crossed, unlike the renovate/dupmachine case that ruled out inherit there. Default (unpinned) checkouts inside deploy.yml's own jobs now resolve correctly even when called from release.yml's push-triggered run, because github.sha for that run already IS the commit release-please just tagged - no target-manifest-ref equivalent needed. Trade-off: release.yml's deploy no longer includes the release tag in its Telegram message, since deploy.yml's own generic message doesn't know it. Not worth an extra input and a conditional format(...) expression to preserve that one detail. --- .github/workflows/deploy.yml | 1 + .github/workflows/release.yml | 40 ++++------------------------------- README.md | 19 ++++++++++++++++- 3 files changed, 23 insertions(+), 37 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 3cb46a1..dc25444 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -1,6 +1,7 @@ name: Deploy on: workflow_dispatch: {} + workflow_call: {} jobs: load-targets: runs-on: ubuntu-latest diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d6a5537..14c5fd6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -60,40 +60,8 @@ jobs: env: GITHUB_SECRETS_JSON: ${{ toJson(secrets) }} GITHUB_VARS_JSON: ${{ toJson(vars) }} - load-targets: - needs: release - if: needs.release.outputs.release_created == 'true' - runs-on: ubuntu-latest - outputs: - matrix: ${{ steps.load-targets.outputs.matrix }} - count: ${{ steps.load-targets.outputs.count }} - steps: - - uses: actions/checkout@v7 - with: - ref: ${{ needs.release.outputs.tag_name }} - - uses: $/.github/actions/load-targets-matrix - id: load-targets deploy: - needs: [release, upload-apps, encrypt, load-targets] - if: needs.load-targets.outputs.count != '0' - strategy: - matrix: ${{ fromJson(needs.load-targets.outputs.matrix) }} - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v7 - with: - ref: ${{ needs.release.outputs.tag_name }} - - uses: $/.github/actions/deploy - with: - target-manifest: ${{ matrix.manifest }} - ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} - sops-age-key: ${{ secrets[matrix.sops_age_key_secret] }} - github-token: ${{ secrets.GITHUB_TOKEN }} - tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} - tailscale-oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }} - - name: Notify Telegram - uses: rubykatzen/baseline/.github/actions/send-telegram-message@v0.16.1 - with: - message: "Deploy: ${{ matrix.name }} updated to ${{ needs.release.outputs.tag_name }}" - telegram-bot-token: ${{ secrets.TELEGRAM_BOT_TOKEN }} - telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} + needs: [release, upload-apps, encrypt] + if: needs.release.outputs.release_created == 'true' + uses: $/.github/workflows/deploy.yml + secrets: inherit diff --git a/README.md b/README.md index f223b55..395cc47 100644 --- a/README.md +++ b/README.md @@ -187,7 +187,9 @@ Useful as a source of ready-made Docker Compose definitions when adding a new ap ## GitHub Actions -This repository provides seven composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, `load-vaults-matrix`, `load-targets-matrix`, `deploy`, and `renovate`) — no reusable workflows. A reusable workflow only pays for itself when the reused thing genuinely needs its own multiple jobs or job-level config (`permissions`, `concurrency`, etc.); `deploy`/`renovate` are each a single job's worth of steps, and a composite action gets that for free without a `workflow_call` boundary's checkout/secrets ceremony (see [`deploy`](#deploy) below for what that ceremony would otherwise cost). +This repository provides seven composite actions under `.github/actions/` (`build-bundle`, `build-apps-bundle`, `encrypt-env`, `load-vaults-matrix`, `load-targets-matrix`, `deploy`, and `renovate`). A reusable workflow only pays for itself when the reused thing genuinely needs its own multiple jobs or job-level config (`permissions`, `concurrency`, etc.); `deploy`/`renovate` are each a single job's worth of steps, and a composite action gets that for free without a `workflow_call` boundary's checkout/secrets ceremony (see [`deploy`](#deploy) below for what that ceremony would otherwise cost). + +The one exception is `deploy.yml` itself: it also declares `workflow_call`, purely so `release.yml` can call it directly instead of duplicating its `load-targets`+`deploy` job pair — see "`deploy.yml`" below. --- @@ -371,6 +373,21 @@ jobs: The `@v0.11.1` pin on the `uses:` line only controls which ref this action's own code runs at. `target-manifest`'s own `app_refs` entries are separate and don't have to match it. +#### `deploy.yml` + +This repository's own `deploy.yml` is the manual-redeploy entry point shown above (`workflow_dispatch`, no inputs — it always redeploys every target in `targets/`), but it also declares `on.workflow_call` with no inputs of its own, purely so `release.yml` can call it directly instead of duplicating its `load-targets`+`deploy` job pair: + +```yaml +jobs: + deploy: + needs: [release, upload-apps, encrypt] + if: needs.release.outputs.release_created == 'true' + uses: $/.github/workflows/deploy.yml + secrets: inherit +``` + +`secrets: inherit` is safe here specifically because both workflows live in this same repository — `deploy.yml` already has native access to every one of this repository's own secrets when triggered directly, so inheriting them from `release.yml` (also this repository) doesn't actually broaden anything. This is the one narrow case where `secrets: inherit`'s same-organization restriction (see [`renovate`](#renovate) below) isn't a concern at all, since there's no organization boundary being crossed in the first place. The trade-off: `release.yml`'s deploy no longer includes the release tag in its Telegram message, since `deploy.yml`'s own generic "`{target}` updated" text doesn't know it - not worth reintroducing an input and a conditional `format(...)` expression just to preserve that one detail. + --- ### `renovate` From e8773ec8cb703f23e6e84443aeecf83731427e46 Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Sat, 29 Aug 2026 01:24:33 +0200 Subject: [PATCH 21/22] feat: renovate everything nightly on a schedule renovate.yml now also triggers on `schedule: 0 3 * * *`. A cron trigger can't supply workflow_dispatch inputs, so `apps` becomes optional (default "[]") both on the workflow input and the renovate action's own input, and an empty apps array now means "every app this target runs" instead of "no apps requested" - renovate.py already skipped apps not present on a given target, so this just changes what an empty selection means rather than adding new matching logic. --- .github/actions/renovate/action.yml | 5 +++-- .github/actions/renovate/renovate.py | 11 +++++++--- .../actions/renovate/tests/test_renovate.py | 21 +++++++++++++++++++ .github/workflows/renovate.yml | 9 +++++--- README.md | 8 ++++--- 5 files changed, 43 insertions(+), 11 deletions(-) diff --git a/.github/actions/renovate/action.yml b/.github/actions/renovate/action.yml index 0b240c0..7d14b3e 100644 --- a/.github/actions/renovate/action.yml +++ b/.github/actions/renovate/action.yml @@ -2,8 +2,9 @@ name: Renovate description: Re-pull and recreate one-or-more apps' containers on a target already present in the caller's own checkout, without touching versions. inputs: apps: - description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers). - required: true + description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers). Empty array means every app this target runs. + required: false + default: "[]" target-manifest: description: Path to the target manifest file, e.g. targets/heimdall.yml - already checked out by the caller. required: true diff --git a/.github/actions/renovate/renovate.py b/.github/actions/renovate/renovate.py index 47d7b06..3702271 100644 --- a/.github/actions/renovate/renovate.py +++ b/.github/actions/renovate/renovate.py @@ -19,7 +19,9 @@ a key in that manifest's own `apps` mapping is skipped, not an error - a target-matrix fan-out dispatches to every target regardless of which of the requested apps it actually runs; skipping all of them there is a -clean no-op. +clean no-op. An empty `apps` list means "every app this target runs" - +the scheduled nightly run has no specific apps to name, so it renovates +everything. Writes `updated`/`updated_hosts`/`target_name` to $GITHUB_OUTPUT so the calling workflow can notify only when a host's image actually changed, @@ -71,9 +73,12 @@ def main(): target = yaml.safe_load(manifest_path.read_text()) target_apps = target.get("apps") or {} - matching_apps = [app for app in requested_apps if app in target_apps] + matching_apps = [app for app in requested_apps if app in target_apps] if requested_apps else list(target_apps) if not matching_apps: - print(f"none of {requested_apps} are deployed on target {target_name!r}, skipping") + if requested_apps: + print(f"none of {requested_apps} are deployed on target {target_name!r}, skipping") + else: + print(f"target {target_name!r} has no apps deployed, skipping") write_github_output("updated", "false") return diff --git a/.github/actions/renovate/tests/test_renovate.py b/.github/actions/renovate/tests/test_renovate.py index 1719137..af066d5 100644 --- a/.github/actions/renovate/tests/test_renovate.py +++ b/.github/actions/renovate/tests/test_renovate.py @@ -92,6 +92,27 @@ def test_skips_a_target_that_runs_none_of_the_requested_apps(self): self.assertIn("updated=false\n", outputs) self.assertIn("target_name=heimdall\n", outputs) + def test_renovates_every_app_when_apps_is_empty(self): + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host", return_value=False) as fake_renovate_host, + ): + self._run_main(directory, [], "apps:\n traefik: {}\n gatus: {}\nhosts: [deploy@host]\n") + + fake_renovate_host.assert_any_call("deploy@host", "~/flightdeck", "traefik") + fake_renovate_host.assert_any_call("deploy@host", "~/flightdeck", "gatus") + self.assertEqual(fake_renovate_host.call_count, 2) + + def test_skips_a_target_with_no_apps_when_apps_is_empty(self): + with ( + tempfile.TemporaryDirectory() as directory, + patch.object(renovate, "renovate_host") as fake_renovate_host, + ): + outputs = self._run_main(directory, [], "apps: {}\nhosts: [deploy@host]\n") + + fake_renovate_host.assert_not_called() + self.assertIn("updated=false\n", outputs) + def test_renovates_only_the_requested_apps_present_on_the_target(self): with ( tempfile.TemporaryDirectory() as directory, diff --git a/.github/workflows/renovate.yml b/.github/workflows/renovate.yml index 85aabaf..b5056c8 100644 --- a/.github/workflows/renovate.yml +++ b/.github/workflows/renovate.yml @@ -3,9 +3,12 @@ on: workflow_dispatch: inputs: apps: - description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers) wherever each is currently deployed, e.g. ["traefik","rybbit"]. + description: JSON array of app name(s) to renovate (re-pull their image and recreate their containers) wherever each is currently deployed, e.g. ["traefik","rybbit"]. Leave empty to renovate every app on every target. type: string - required: true + required: false + default: "[]" + schedule: + - cron: "0 3 * * *" jobs: load-targets: runs-on: ubuntu-latest @@ -27,7 +30,7 @@ jobs: - uses: $/.github/actions/renovate id: renovate with: - apps: ${{ inputs.apps }} + apps: ${{ inputs.apps || '[]' }} target-manifest: ${{ matrix.manifest }} ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} diff --git a/README.md b/README.md index 395cc47..d6aeb6d 100644 --- a/README.md +++ b/README.md @@ -394,7 +394,7 @@ jobs: Re-pulls and recreates one-or-more apps' containers on a target already present in the caller's own checkout, without touching versions — no new app bundle, no new vault-sourced env, no rebuilt release tree. Just `docker compose pull && docker compose up -d` per requested app, against its already-current release, on each of the target's hosts. Sibling to [`deploy`](#deploy), same checkout-free shape, deliberately narrower job. -`apps` is a JSON array, so one run can renovate several apps at once (e.g. a nightly cron renovating `["traefik","rybbit"]` while leaving everything else alone) — pass a single-element array for the one-app case. Not every target runs every requested app; [`renovate.py`](.github/actions/renovate/renovate.py) decides that itself from the target manifest's own `apps` mapping and simply does nothing — never opening an SSH connection — if none of the requested apps are present there. +`apps` is a JSON array, so one run can renovate several apps at once (e.g. `["traefik","rybbit"]`) — pass a single-element array for the one-app case, or an empty array for every app the target runs. Not every target runs every requested app; [`renovate.py`](.github/actions/renovate/renovate.py) decides that itself from the target manifest's own `apps` mapping and simply does nothing — never opening an SSH connection — if none of the requested apps are present there. It never touches `app_refs`/`env_refs`, never re-decrypts a vault, never rebuilds the release tree; it just picks up a new image behind an existing tag. To tell whether a host's image actually changed (rather than the pull being a no-op), it compares `docker compose images -q` output before and after the pull, and exposes `updated`/`updated-hosts`/`target-name` (derived from the manifest's own filename) as action outputs - `updated-hosts` lists `app@host` pairs, since more than one app may have been renovated in the same run. @@ -413,7 +413,7 @@ jobs: - uses: rubykatzen/flightdeck/.github/actions/renovate@v0.11.0 id: renovate with: - apps: ${{ inputs.apps }} # JSON array, e.g. '["traefik","rybbit"]' + apps: ${{ inputs.apps || '[]' }} # JSON array, e.g. '["traefik","rybbit"]'; '[]' on the nightly cron run target-manifest: ${{ matrix.manifest }} ssh-private-key: ${{ secrets[matrix.ssh_private_key_secret] }} # tailscale-oauth-client-id: ${{ vars.TAILSCALE_OAUTH_CLIENT_ID }} @@ -427,7 +427,9 @@ jobs: telegram-chat-id: ${{ secrets.TELEGRAM_CHAT_ID }} ``` -**Known gap:** if none of the requested `apps` match any target at all (a typo, say), every matrix job just does nothing and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. `apps`' "all/single/list" selection and scheduled/allowlist-driven runs (see #168) also aren't implemented yet — this only ever renovates exactly the apps it's given. +`renovate.yml` also runs on a nightly `schedule` (`0 3 * * *`), with `apps` defaulting to `[]` — a cron trigger can't supply `workflow_dispatch` inputs at all, so the empty-array-means-everything behavior above exists specifically to give the scheduled run something to pass. + +**Known gap:** if none of the requested `apps` match any target at all (a typo, say), every matrix job just does nothing and the whole run still reports success — there's no cheap way to fail loudly on "zero matches across the board" without a job that waits on the whole matrix and inspects its results. ## License From ea992e9f888493584b7a3cb1cf63cbe8a98eed8f Mon Sep 17 00:00:00 2001 From: Yuriy Kirillov Date: Sat, 29 Aug 2026 01:27:34 +0200 Subject: [PATCH 22/22] feat: minimally validate vault manifests in load-vaults-matrix Mirrors load-targets-matrix's own validate_target: just type + non-empty checks on asset/keys/env, so a structurally broken vault manifest fails the matrix build immediately instead of reaching a checkout+dependency install on a different job (encrypt-env) before erroring out. encrypt-env's render-env.py still does the real field-level validation (name patterns, source references) - this only catches missing/empty top-level sections early, same division of labor as targets already had. --- .github/actions/load-targets-matrix/README.md | 2 +- .github/actions/load-vaults-matrix/README.md | 6 ++++-- .../scripts/load-vaults-matrix.py | 10 ++++++++++ .../tests/test_load_vaults_matrix.py | 17 +++++++++++++++++ README.md | 4 ++-- 5 files changed, 34 insertions(+), 5 deletions(-) diff --git a/.github/actions/load-targets-matrix/README.md b/.github/actions/load-targets-matrix/README.md index bfed0e9..5ebd5cc 100644 --- a/.github/actions/load-targets-matrix/README.md +++ b/.github/actions/load-targets-matrix/README.md @@ -1,6 +1,6 @@ # load-targets-matrix -Composite GitHub Action that reads every target manifest in a directory into a GitHub Actions matrix, validating each one's required shape. Sibling to [`load-vaults-matrix`](../load-vaults-matrix), which stays generic and unvalidated for `vaults/*.yml`. +Composite GitHub Action that reads every target manifest in a directory into a GitHub Actions matrix, validating each one's required shape. Sibling to [`load-vaults-matrix`](../load-vaults-matrix), same shape, for `vaults/*.yml` instead. ## Usage diff --git a/.github/actions/load-vaults-matrix/README.md b/.github/actions/load-vaults-matrix/README.md index 55c07d7..2e3061e 100644 --- a/.github/actions/load-vaults-matrix/README.md +++ b/.github/actions/load-vaults-matrix/README.md @@ -1,6 +1,6 @@ # load-vaults-matrix -Composite GitHub Action that reads every vault manifest in `vaults/` into a GitHub Actions matrix. It does no schema validation — `encrypt-env` re-parses and validates each manifest itself when it actually encrypts it. Specialized counterpart to [`load-targets-matrix`](../load-targets-matrix), which *does* validate, for `targets/*.yml` specifically. +Composite GitHub Action that reads every vault manifest in `vaults/` into a GitHub Actions matrix, validating each one's minimal shape. `encrypt-env` still re-parses and validates each manifest itself, in full, when it actually encrypts it - the checks here only catch a structurally broken manifest before it reaches a checkout and dependency install on a different job entirely. Sibling to [`load-targets-matrix`](../load-targets-matrix), same shape, for `targets/*.yml` instead. ## Usage @@ -13,4 +13,6 @@ Composite GitHub Action that reads every vault manifest in `vaults/` into a GitH The action exposes `matrix`, containing `{ "include": [...] }`, and `count`. Each matrix item merges the manifest's own top-level YAML fields with `name` (the file's basename) and `manifest` (its path). -Files may use either the `.yml` or `.yaml` extension. Filenames must match `^[a-z0-9][a-z0-9-]*$` and be unique per directory; duplicate top-level YAML keys within a manifest are rejected. Beyond that, the parsed YAML mapping is passed through as-is — validate anything else downstream. +Every manifest must set `asset` (non-empty string), `keys` (non-empty list), and `env` (non-empty mapping) - just enough to catch a missing or empty section early. `encrypt-env` is what actually validates each field's shape (name patterns, source references, and so on). + +Files may use either the `.yml` or `.yaml` extension. Filenames must match `^[a-z0-9][a-z0-9-]*$` and be unique per directory; duplicate top-level YAML keys within a manifest are rejected. diff --git a/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py b/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py index e30967c..3ceb251 100644 --- a/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py +++ b/.github/actions/load-vaults-matrix/scripts/load-vaults-matrix.py @@ -45,6 +45,15 @@ def load_manifest(path): return value +def validate_vault(name, manifest): + if not isinstance(manifest.get("asset"), str) or not manifest["asset"]: + raise ManifestError(f"vault {name!r} must set asset to a non-empty string") + if not isinstance(manifest.get("keys"), list) or not manifest["keys"]: + raise ManifestError(f"vault {name!r} must set keys to a non-empty list") + if not isinstance(manifest.get("env"), dict) or not manifest["env"]: + raise ManifestError(f"vault {name!r} must set env to a non-empty mapping") + + def build_matrix(directory): paths = sorted(directory.glob("*.yml")) + sorted(directory.glob("*.yaml")) if not paths: @@ -59,6 +68,7 @@ def build_matrix(directory): raise ManifestError(f"duplicate manifest name: {name}") seen_names.add(name) manifest = load_manifest(path) + validate_vault(name, manifest) item = {"name": name, "manifest": str(path)} item.update(manifest) include.append(item) diff --git a/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py b/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py index 7002b7d..db63574 100644 --- a/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py +++ b/.github/actions/load-vaults-matrix/tests/test_load_vaults_matrix.py @@ -74,6 +74,23 @@ def test_rejects_duplicate_yaml_key(self): with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "duplicate YAML key"): load_vaults_matrix.build_matrix(self.directory) + def test_rejects_missing_asset(self): + (self.directory / "traefik.yml").write_text("keys: [mainframe]\nenv: {HTTP_PORT: '80'}\n") + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "must set asset"): + load_vaults_matrix.build_matrix(self.directory) + + def test_rejects_empty_keys(self): + (self.directory / "traefik.yml").write_text( + "asset: mainframe-traefik.sops.env\nkeys: []\nenv: {HTTP_PORT: '80'}\n" + ) + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "must set keys"): + load_vaults_matrix.build_matrix(self.directory) + + def test_rejects_missing_env(self): + (self.directory / "traefik.yml").write_text("asset: mainframe-traefik.sops.env\nkeys: [mainframe]\n") + with self.assertRaisesRegex(load_vaults_matrix.ManifestError, "must set env"): + load_vaults_matrix.build_matrix(self.directory) + if __name__ == "__main__": unittest.main() diff --git a/README.md b/README.md index d6aeb6d..6c13b19 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ flightdeck/ │ │ ├── build-bundle/ # Build and upload a zip bundle from given paths │ │ ├── build-apps-bundle/ # Build and upload an apps/ catalog bundle │ │ ├── encrypt-env/ # Encrypt a target env and upload it to a release -│ │ ├── load-vaults-matrix/ # Read vaults/ into a workflow matrix, no schema validation +│ │ ├── load-vaults-matrix/ # Read vaults/ into a workflow matrix, minimal shape validation │ │ ├── load-targets-matrix/ # Same, but for targets/ specifically - validates the required shape │ │ ├── deploy/ # Push-based deploy against a target manifest (deploy.py + friends) │ │ │ ├── deploy.py # Deploy entrypoint (runs on the CI runner) @@ -246,7 +246,7 @@ sops_age_key_secret: MAINFRAME_AGE_PRIVATE_KEY A vault manifest's `env:` value is either `${NAME}` (a reference — look up the GitHub Secret/Variable named `NAME`) or a bare literal (any other value, used as-is with no lookup at all — see `DISABLE_SIGNUP: true` above). Use a literal for a value that's fixed for this target but isn't a secret and doesn't need a GitHub Secret/Variable to exist just to hold it. -[`load-vaults-matrix`](.github/actions/load-vaults-matrix) reads every file in `vaults/` into a matrix — it does not validate the manifest shape; `encrypt-env` re-parses and validates its own manifest from `manifest` (see "`encrypt-env`" below). Targets go through the more specific [`load-targets-matrix`](.github/actions/load-targets-matrix) instead, which *does* validate the shape above (`hosts`, `app_refs`, `apps`, `ssh_private_key_secret`, `sops_age_key_secret` all required) before a broken manifest ever reaches a checkout+dependency-install on a different job entirely. The workflows calling the [`deploy`](#deploy)/[`renovate`](#renovate) actions then pull `matrix.ssh_private_key_secret`/`matrix.sops_age_key_secret` directly, to resolve actual secret values by name. +[`load-vaults-matrix`](.github/actions/load-vaults-matrix) reads every file in `vaults/` into a matrix, checking only that `asset`/`keys`/`env` are present and non-empty — `encrypt-env` still re-parses and fully validates its own manifest from `manifest` (field-level shape: name patterns, source references, and so on; see "`encrypt-env`" below). Targets go through the more specific [`load-targets-matrix`](.github/actions/load-targets-matrix) instead, which validates the shape above (`hosts`, `app_refs`, `apps`, `ssh_private_key_secret`, `sops_age_key_secret` all required) the same way. Either way, a structurally broken manifest fails here, before it ever reaches a checkout+dependency-install on a different job entirely. The workflows calling the [`deploy`](#deploy)/[`renovate`](#renovate) actions then pull `matrix.ssh_private_key_secret`/`matrix.sops_age_key_secret` directly, to resolve actual secret values by name. ---