From 10036e9a3b21e3a881770ee20e04ac72daf0c824 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Wed, 9 Sep 2026 14:52:25 -0700 Subject: [PATCH] feat(engine): the proton engine, and git in the proton image engines/proton.yml moves a staging tree into one Proton Drive folder through the official CLI: the session pulled from the bucket and sealed back after every call, the destination checked by UID, one upload whose summary confirms it, hooks for the mirror's stage and prune. The bucket holds only the session: the CLI skips identical content and create-new-revision handles the rest, so Proton's version history is the mirror's history. The proton image gains git and s3, a boto3 get/put, and tells boto3 to send checksums only when required, which is R2's one quirk. examples/proton consumes the engine and its offline verb is the engine's check: confirm over an accepting and a refusing summary, and the age verb's round trip. --- .github/workflows/ci.yml | 1 - CLAUDE.md | 30 ++- README.md | 61 ++++- docker/proton.Dockerfile | 19 +- docker/s3.py | 49 ++++ engines/proton.yml | 240 ++++++++++++++++++++ examples/proton/Taskfile.yml | 56 +++-- examples/proton/fixtures/upload-failed.json | 2 + examples/proton/fixtures/upload-ok.json | 2 + examples/proton/op.env | 6 +- examples/proton/render.txt | 103 ++++++++- toolbox.yml | 13 +- 12 files changed, 537 insertions(+), 45 deletions(-) create mode 100755 docker/s3.py create mode 100644 engines/proton.yml create mode 100644 examples/proton/fixtures/upload-failed.json create mode 100644 examples/proton/fixtures/upload-ok.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 884d396..275d5d4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,6 +30,5 @@ jobs: working-directory: examples/${{ matrix.variant }} run: task check - name: The engine's verbs that need no bucket, over the fixtures - if: matrix.variant == 'rsync' working-directory: examples/${{ matrix.variant }} run: task run -- task offline diff --git a/CLAUDE.md b/CLAUDE.md index b843e98..754d0f3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,10 +10,11 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont mirror's root value. - Verb names are reserved across host and container. `plan` is the host-side read-only run; an engine's batch planner is `split`. The hooks, `prepare`, `verify`, `index`, - `smoke`, `smoke-mirror`, `report-engine` in the engine and `report-mirror` in the - toolbox, are the only verbs a mirror redefines, each excluded on the include that - defines it; `pipeline` and `plan-pipeline` come from the engine and a mirror excludes - them only for another order. `report-engine` exists in both files, so an engine + `smoke`, `smoke-mirror`, `report-engine` in the rsync engine, `stage` and `prune` in + the proton engine, and `report-mirror` in the toolbox, are the only verbs a mirror + redefines, each excluded on the include that defines it; `pipeline` and + `plan-pipeline` come from the engine and a mirror excludes them only for another + order. `report-engine` exists in both files, so an engine consumer excludes it on the toolbox include. Extension is a hook, never a copy: a verb that needs more than the engine does gets a `-mirror` hook the engine calls last. - A mirror's root var shadows a command-line `KEY=value` inside an included verb, so an @@ -48,6 +49,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont so the container engine's own progress lines never reach `render.txt`. - Inside a `sh:` var, `printf -- '-e %s'` prints dashes: task's built-in shell takes the `--` as the format. Use `printf '%s %s ' -e "$v"`. +- Task's built-in shell has no `umask`. A file that must be born 0600 is + `install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does. +- The proton engine keeps no state of the mirror's in the bucket, only the CLI session, + because `filesystem upload` skips a file whose content Proton already holds and + `-f create-new-revision` handles one that changed. Every CLI call goes through `pd`, + which pushes the session back whatever the exit: the refresh token rotates, and a run + that kept a rotated token to itself leaves the next run unable to log in. Two mirrors + never share one session for the same reason. - Actions pinned to a full SHA with the version in a trailing comment. A mirror pins the two reusable workflows that way; each checks this repository out at its own commit (`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is @@ -58,11 +67,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont ```sh cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline -cd examples/proton && task image-build && task run -- task tools && task check +cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline ``` A verb change updates the `render.txt` files via `task render-update`; `offline` is -the engine's own check over `examples/rsync/fixtures/`: the list diff over `run-root` -and `run-empty`, `retry`'s exit codes, and `prepare` and `verify` over `tree/`, a signed -subtree whose tlpdb is signed by a throwaway key pinned in the example. Regenerate the -tree with a new key only to change its shape; the private half was never kept. +each engine's own check. The rsync one runs over `examples/rsync/fixtures/`: the list +diff over `run-root` and `run-empty`, `retry`'s exit codes, and `prepare` and `verify` +over `tree/`, a signed subtree whose tlpdb is signed by a throwaway key pinned in the +example. Regenerate the tree with a new key only to change its shape; the private half +was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an +accepting and a refusing upload summary, and the `age` verb round trip with a throwaway +identity. diff --git a/README.md b/README.md index ec7e2ee..6880fc2 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,9 @@ The toolbox every katoptra mirror includes by URL. The rule it enforces: the code that starts a run, contains it, resolves its secrets, checks it and reports it lives here, -once. The code that moves bytes for a transport lives here too, once per engine, and -`engines/rsync.yml` is the first. A mirror holds only its identity, the order of its +once. The code that moves bytes for a transport lives here too, once per engine: +`engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a +staging tree into Proton Drive. A mirror holds only its identity, the order of its pipeline, and the few verbs no other mirror shares. ## The layers @@ -266,6 +267,60 @@ so an hourly run costs one listing of upstream and none of the bucket. Two thing What a mirror cannot afford to lose is the bucket. Everything else, the state file and the staging tree included, is derived from it and from upstream. +### The proton engine + +`engines/proton.yml` moves a staging tree into one Proton Drive folder, through the +official `proton-drive` CLI, for a mirror whose upstream fits in a run: the mirror fills +`staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket +but the CLI session, because the CLI skips a file whose content Proton already holds and +`-f create-new-revision` makes a revision of one that changed; Proton's version history +is the history of the mirror. + +| Verb | Does | +|---|---| +| `pipeline`, `plan-pipeline` | `clock`, `session`, `destination`, `stage`, `upload`, `confirm`, `prune`, `report`, `ping`; and the read-only half, through `stage` | +| `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` | +| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit | +| `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` | +| `stage` | Hook. The mirror fills `staging/` with what Proton should hold | +| `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` | +| `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` | +| `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` | +| `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash | +| `session-seal -- ` | Host side: a laptop login's two files, encrypted into the bucket | +| `empty-trash` | Host side, asks first: everything in Proton's trash, permanently | +| `report-engine` | Hook. The engine's rows of the run summary | + +The engine reads no root var. Its inputs are the environment, by the names every Proton +mirror's `op.env` carries: `MIRROR_PROTON_DESTINATION` and its `_UID`, `MIRROR_R2_BUCKET`, +`MIRROR_AGE_IDENTITY`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and +`AWS_ENDPOINT_URL_S3`. The session is the one thing to set up by hand: the CLI can only be +seeded by a browser sign-in, so log in once on a laptop into a directory inside the repo +and run `task session-seal -- `. Two mirrors never share a session; its +refresh token rotates on every call, and the loser of a race needs a fresh login. + +A mirror of every repository under two GitHub owners, shaped like github: + +```yaml +version: '3' +vars: + OWNERS: jshvn katoptra +includes: + toolbox: + taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml + flatten: true + excludes: [report-engine, report-mirror] + vars: {NAME: github, DESC: a nightly mirror of every repository under jshvn and katoptra into Proton Drive, IMAGE: ghcr.io/katoptra/toolbox:proton-v2} + proton: + taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml + flatten: true + excludes: [stage, prune] +tasks: + stage: {cmds: ['# list the repositories, clone each as a mirror, bundle it under {{.STAGING}}//']} + prune: {cmds: ['# list-folder each owner in Proton; trash the bundles no repository has']} + report-mirror: {cmds: ['# the mirror rows']} +``` + ## Overriding a verb List it under `excludes:` on the include that defines it and define it in the mirror. @@ -314,7 +369,7 @@ two matrix entries: | Variant | Base | Tools | For | |---|---|---|---| | `rsync` | ubuntu 24.04 | rsync, gnupg, xz, curl, perl, go-task, AWS CLI v2 (s3, sts) | rsync upstreams to R2: ctan, tlnet, cran, cpan | -| `proton` | python 3.13 slim | proton-drive, age, go-task, boto3, requests, pytest, ruff | Python pipelines: dropbox, photos | +| `proton` | python 3.13 slim | proton-drive, age, git, go-task, boto3, requests, pytest, ruff, and `s3`, a boto3 get/put | Proton Drive sinks: github through the proton engine, dropbox through its own Python | An HTTPS engine would be a third row: the same base as `rsync`, curl and the AWS CLI, and a `list` that reads an index instead of `rsync --list-only`. diff --git a/docker/proton.Dockerfile b/docker/proton.Dockerfile index b2e74ec..877f95e 100644 --- a/docker/proton.Dockerfile +++ b/docker/proton.Dockerfile @@ -1,7 +1,8 @@ # syntax=docker/dockerfile:1.7 -# The proton toolbox: Python plus the Proton Drive CLI and age, for the mirrors whose -# engine is a Python package. The repo is bind-mounted at /work; PYTHONPATH finds its -# src/. Base pinned by digest; every tool from toolchain.lock.toml at the build context. +# The proton toolbox: Python plus the Proton Drive CLI, age and git, for the mirrors whose +# sink is Proton Drive, whether their engine is engines/proton.yml or a Python package of +# their own. The repo is bind-mounted at /work; PYTHONPATH finds its src/. Base pinned by +# digest; every tool from toolchain.lock.toml at the build context, git and curl from apt. FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS fetch RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \ @@ -26,10 +27,11 @@ RUN set -eu; \ FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS toolbox -RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \ +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl git \ && rm -rf /var/lib/apt/lists/* COPY --from=fetch /usr/local/bin/proton-drive /usr/local/bin/age /usr/local/bin/age-keygen /usr/local/bin/task /usr/local/bin/ COPY toolchain.lock.toml /etc/toolchain.lock.toml +COPY docker/s3.py /usr/local/bin/s3 RUN python - <<'PY' import subprocess, tomllib @@ -49,12 +51,17 @@ first = lambda argv: subprocess.check_output(argv, text=True).splitlines()[0] assert "@" + lock["proton_drive_cli"]["version"] in first(["proton-drive", "version"]) assert lock["age"]["version"] in first(["age", "--version"]) assert lock["task"]["version"] in subprocess.check_output(["task", "--version"], text=True) +assert first(["git", "--version"]).startswith("git version 2.") +assert subprocess.run(["s3"], capture_output=True).returncode == 2 # usage; boto3 imports PY # Inside a run there is no network for Taskfiles: the mirror's .task/remote cache rides -# in with the bind mount. R2 has one region; the value is a literal, not a secret. +# in with the bind mount. R2 has one region and rejects the SDK's default checksum +# headers; both values are literals, not secrets. ENV PYTHONUNBUFFERED=1 \ PYTHONPATH=/work/src \ TASK_REMOTE_OFFLINE=1 \ - AWS_REGION=auto + AWS_REGION=auto \ + AWS_REQUEST_CHECKSUM_CALCULATION=when_required \ + AWS_RESPONSE_CHECKSUM_VALIDATION=when_required WORKDIR /work diff --git a/docker/s3.py b/docker/s3.py new file mode 100755 index 0000000..af2176c --- /dev/null +++ b/docker/s3.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""s3 get KEY FILE | s3 put FILE KEY: one object of the bucket MIRROR_R2_BUCKET names. + +The proton image's S3 client, for the mirrors whose engine keeps a Proton session in the +bucket. Credentials and the endpoint come from the environment as boto3 reads them: +AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL_S3. `get` exits 3 when the +key is absent, so a caller can tell a missing object from a failed read. A failure prints +its class alone: the message may carry the endpoint, and the logs are public. +""" + +import os +import sys + +import boto3 +from botocore.config import Config +from botocore.exceptions import BotoCoreError, ClientError + +MISSING = {"404", "NoSuchKey"} + + +def main(argv: list[str]) -> int: + if len(argv) != 4 or argv[1] not in {"get", "put"}: + print(__doc__.strip(), file=sys.stderr) + return 2 + names = ("MIRROR_R2_BUCKET", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_ENDPOINT_URL_S3") + unset = [n for n in names if not os.environ.get(n)] + if unset: + print(f"s3: unset in the environment: {' '.join(unset)}", file=sys.stderr) + return 2 + bucket = os.environ["MIRROR_R2_BUCKET"] + client = boto3.client("s3", config=Config(retries={"max_attempts": 5, "mode": "standard"})) + try: + if argv[1] == "get": + client.download_file(bucket, argv[2], argv[3]) + else: + client.upload_file(argv[2], bucket, argv[3]) + except ClientError as exc: + if argv[1] == "get" and exc.response.get("Error", {}).get("Code") in MISSING: + return 3 + print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr) + return 1 + except BotoCoreError as exc: + print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/engines/proton.yml b/engines/proton.yml new file mode 100644 index 0000000..16de42a --- /dev/null +++ b/engines/proton.yml @@ -0,0 +1,240 @@ +# The proton engine: how bytes move from a staging tree into one Proton Drive folder, for +# every mirror that includes this file flattened beside the toolbox. A run restores the +# Proton CLI session from the bucket, checks that the destination is the folder the vault +# names, lets the mirror fill the staging tree, uploads it in one CLI call (the CLI skips a +# file whose content Proton already holds and makes a new revision of one that changed), +# confirms the upload from the CLI's own summary, lets the mirror prune what its upstream +# dropped, and reports. Proton's version history is the cloud-side history, so the bucket +# holds nothing of the mirror's but the session. +# +# Read from the environment, by the names every Proton mirror's vault uses: +# MIRROR_PROTON_DESTINATION (the CLI path of the destination folder, /my-files/GitHub +# say), MIRROR_PROTON_DESTINATION_UID (its UID, compared on every run before a write), +# MIRROR_R2_BUCKET, MIRROR_AGE_IDENTITY, and the three AWS_* names boto3 reads, with +# AWS_ENDPOINT_URL_S3 for the endpoint. Each is a line of op.env, so it crosses into the +# container by name. +# +# Hooks: stage fills STAGING and every mirror defines it; prune trashes what upstream no +# longer has, and a mirror that wants that defines it. A mirror lists each it defines under +# excludes: on this include. pipeline and plan-pipeline are the vocabulary in order. +# +# Rules this file keeps, as the toolbox does: no vars: default for anything a mirror +# owns, and the CLI session is written back to the bucket after every CLI call, whatever +# the exit, because its refresh token rotates and a run that kept a rotated token to +# itself leaves the next run unable to log in. +version: '3' + +vars: + STAGING: '{{.ROOT_DIR}}/staging' + SESSION: '{{.ROOT_DIR}}/.run/session' + SESSION_KEY: .state/session.tar.age + UPLOAD: '{{.ROOT_DIR}}/.run/upload.json' + # The CLI with its session as plain files under RUN/session: the image has no keyring. + PD: PROTON_DRIVE_CACHE_DIR={{.SESSION}} PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive + +tasks: + pipeline: + desc: clock -> session -> destination -> stage -> upload -> confirm -> prune -> report -> ping + cmds: + - {task: clock} + - {task: session} + - {task: destination} + - {task: stage} + - {task: upload} + - {task: confirm} + - {task: prune} + - {task: report} + - {task: ping} + + plan-pipeline: + desc: 'The read-only half: clock -> session -> destination -> stage, then what an upload would carry' + cmds: + - {task: clock} + - {task: session} + - {task: destination} + - {task: stage} + - | + find {{.STAGING}} -type f -printf '%s\n' | awk '{ n++; s += $1 } END { printf "plan: %d files, %.1f MB staged\n", n, s / 1e6 }' + + # ---- the session ---- + session: + desc: Restore the Proton CLI session from the bucket into RUN/session + cmds: + - rm -rf {{.SESSION}} && mkdir -p {{.SESSION}} && chmod 700 {{.SESSION}} + - {task: pull, vars: {KEY: '{{.SESSION_KEY}}', OUT: '{{.RUN}}/session.tar.age', MISSING: 'echo "no Proton session at {{.SESSION_KEY}}: log in on a laptop and run task session-seal" >&2; exit 1'}} + - {task: age, vars: {MODE: decrypt, IN: '{{.RUN}}/session.tar.age', OUT: '{{.RUN}}/session.tar'}} + # The two members the CLI needs, and nothing else the archive might hold. + - tar -xf {{.RUN}}/session.tar -C {{.SESSION}} auth-session.json clientUid.json && chmod 600 {{.SESSION}}/*.json + - rm -f {{.RUN}}/session.tar {{.RUN}}/session.tar.age + - sha256sum {{.SESSION}}/auth-session.json > {{.RUN}}/session.sha + + session-push: + desc: Seal RUN/session back into the bucket when the CLI rotated its token since the last push + internal: true + status: ['sha256sum --status -c {{.RUN}}/session.sha'] + cmds: + - 'echo "session: token rotated, sealing it back to the bucket"' + - {task: seal, vars: {DIR: '{{.SESSION}}'}} + - sha256sum {{.SESSION}}/auth-session.json > {{.RUN}}/session.sha + + seal: + desc: 'The two session files of DIR, tarred and age-encrypted, to the bucket: task seal DIR=.run/pd' + requires: {vars: [DIR]} + cmds: + - defer: rm -f {{.RUN}}/session.tar {{.RUN}}/session.tar.age + - mkdir -p {{.RUN}} && tar -cf {{.RUN}}/session.tar -C {{.DIR}} auth-session.json clientUid.json + - {task: age, vars: {MODE: encrypt, IN: '{{.RUN}}/session.tar', OUT: '{{.RUN}}/session.tar.age'}} + - {task: push, vars: {FILE: '{{.RUN}}/session.tar.age', KEY: '{{.SESSION_KEY}}'}} + + session-seal: + desc: 'Encrypt a laptop Proton CLI session into the bucket: task session-seal -- .run/pd' + cmds: + - {task: op, vars: {ARGS: 'task seal DIR={{.CLI_ARGS}}'}} + + age: + desc: age MODE (encrypt or decrypt) IN to OUT with MIRROR_AGE_IDENTITY, which touches disk for the call alone, mode 0600 + internal: true + requires: {vars: [MODE, IN, OUT]} + cmds: + - | + test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } + k={{.RUN}}/age.key; mkdir -p {{.RUN}}; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" + rc=0 + case {{.MODE}} in + encrypt) age -r "$(age-keygen -y "$k")" -o {{.OUT}} {{.IN}} || rc=$? ;; + decrypt) age -d -i "$k" -o {{.OUT}} {{.IN}} || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; + esac + rm -f "$k"; exit $rc + + # ---- the bucket ---- + pull: + desc: KEY from the bucket to OUT; MISSING runs when the bucket has no such key + internal: true + requires: {vars: [KEY, OUT, MISSING]} + cmds: + - | + rc=0; s3 get {{.KEY}} {{.OUT}} || rc=$? + case $rc in 0) ;; 3) {{.MISSING}} ;; *) exit $rc ;; esac + + push: + desc: FILE to the bucket's KEY as one PutObject + internal: true + requires: {vars: [FILE, KEY]} + cmds: + - s3 put {{.FILE}} {{.KEY}} + + # ---- the CLI ---- + # Every CLI call goes through here: stderr to RUN/pd.err rather than the public log, + # the session pushed back whatever the exit, then the exit judged against OK, the codes + # accepted besides 0. + pd: + desc: One CLI call, ARGS, with the session written back afterwards + internal: true + requires: {vars: [ARGS]} + cmds: + - rc=0; {{.PD}} {{.ARGS}} 2> {{.RUN}}/pd.err || rc=$?; echo $rc > {{.RUN}}/pd.rc + - {task: session-push} + - | + rc=$(cat {{.RUN}}/pd.rc) + case " {{.OK | default 0}} " in *" $rc "*) exit 0 ;; esac + echo "proton-drive exited $rc; its stderr, $(wc -l < {{.RUN}}/pd.err) lines, is in .run/pd.err" >&2; exit $rc + + destination: + desc: Refuse the run unless MIRROR_PROTON_DESTINATION resolves to exactly one folder whose UID is MIRROR_PROTON_DESTINATION_UID + cmds: + - defer: rm -f {{.RUN}}/parent.json + - {task: pd, vars: {ARGS: 'filesystem list -j "$(dirname "$MIRROR_PROTON_DESTINATION")" > {{.RUN}}/parent.json'}} + # ponytail: the folder is found by basename, so a destination whose own name holds a + # slash is unsupported; the CLI escapes one with a backslash and dirname does not. + - | + python3 - "$(basename "$MIRROR_PROTON_DESTINATION")" "$MIRROR_PROTON_DESTINATION_UID" {{.RUN}}/parent.json <<'PY' + import json, sys + name, uid, path = sys.argv[1:] + unwrap = lambda v: v.get("value") if isinstance(v, dict) else v + nodes = json.load(open(path)) + found = [n for n in nodes if isinstance(n, dict) and unwrap(n.get("name")) == name] + if len(found) != 1 or unwrap(found[0].get("type")) != "folder": + sys.exit(f"destination: {len(found)} folder(s) of that name among {len(nodes)} entries of the parent; expected exactly one") + if not uid or str(unwrap(found[0].get("uid")) or "") != uid: + sys.exit("destination: the folder's UID is not the one the vault names") + print("destination: the folder and its UID match the vault") + PY + + upload: + desc: STAGING into the destination in one CLI call; identical files are skipped, changed ones become new revisions; the CLI's summary goes to RUN/upload.json + cmds: + - 'test -n "$(ls -A {{.STAGING}} 2>/dev/null)" || { echo "upload: staging is empty; stage put nothing there" >&2; exit 1; }' + # Exit 1 is the CLI refusing some items and handling the rest; confirm reads the summary. + - {task: pd, vars: {ARGS: 'filesystem upload -f create-new-revision -d merge -t --json {{.STAGING}}/* "$MIRROR_PROTON_DESTINATION" > {{.UPLOAD}}', OK: '0 1'}} + + confirm: + desc: The CLI's summary must account for every staged file and folder, with no failures; the verdict goes to RUN/confirm.txt + cmds: + - | + files=$(find {{.STAGING}} -type f | wc -l); folders=$(find {{.STAGING}} -mindepth 1 -type d | wc -l) + python3 - "$files" "$folders" {{.UPLOAD}} {{.RUN}}/confirm.txt <<'PY' + import json, sys + files, folders = int(sys.argv[1]), int(sys.argv[2]) + # The CLI writes progress and its summary as one JSON object per line; the last + # line carrying transferredItems is the summary. + summary = None + for line in open(sys.argv[3]): + try: + candidate = json.loads(line) + except ValueError: + continue + if isinstance(candidate, dict) and "transferredItems" in candidate: + summary = candidate + if summary is None: + sys.exit("confirm: no upload summary in the CLI's output") + t, s, f = (int(summary.get(k, 0) or 0) for k in ("transferredItems", "skippedItems", "failedItems")) + verdict = f"{t} uploaded as new files or revisions, {s} skipped as identical, {f} failed, of {files} files in {folders} folders staged" + ok = f == 0 and t + s == files + folders + open(sys.argv[4], "w").write(verdict + ("" if ok else " (not confirmed)") + "\n") + print("confirm: " + verdict) + sys.exit(0 if ok else 1) + PY + + list-folder: + desc: The JSON listing of the Proton folder FOLDER to OUT; a folder the CLI cannot list gives an empty one, so a caller prunes nothing + internal: true + requires: {vars: [FOLDER, OUT]} + cmds: + - {task: pd, vars: {ARGS: 'filesystem list -j "{{.FOLDER}}" > {{.OUT}} || echo "[]" > {{.OUT}}'}} + + trash: + desc: Trash the Proton nodes at PATHS, CLI paths space separated; Proton's trash keeps them + status: ['test -z "{{.PATHS}}"'] + cmds: + - {task: pd, vars: {ARGS: 'filesystem trash {{.PATHS}}'}} + + empty-trash: + desc: Permanently delete everything in Proton Drive's trash, inside the toolbox; asks first, never scheduled + prompt: This permanently deletes everything in Proton Drive trash. Continue? + cmds: + - {task: op, vars: {ARGS: task empty-trash-pipeline}} + + empty-trash-pipeline: + internal: true + cmds: + - {task: clock} + - {task: session} + - {task: pd, vars: {ARGS: filesystem empty-trash}} + + # ---- hooks ---- + stage: {cmds: []} # the mirror's: fill STAGING with what Proton should hold + prune: {cmds: []} # the mirror's: trash what its upstream no longer has + + report-engine: + desc: The engine's rows of the run summary, appended after the toolbox's + silent: true + cmds: + - | + staged=$(find {{.STAGING}} -type f 2>/dev/null | wc -l) + mb=$(find {{.STAGING}} -type f -printf '%s\n' 2>/dev/null | awk '{ s += $1 } END { printf "%.1f", s / 1e6 }') + cat >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" </dev/null || echo "not confirmed this run") | + | Session | $(test -f {{.RUN}}/session.sha && echo "restored, and sealed back on rotation" || echo "not restored") | + EOF diff --git a/examples/proton/Taskfile.yml b/examples/proton/Taskfile.yml index ed0c08e..02341b8 100644 --- a/examples/proton/Taskfile.yml +++ b/examples/proton/Taskfile.yml @@ -1,34 +1,62 @@ -# A consumer of the toolbox with the proton image. No engine, so the mirror defines the -# two pipeline verbs itself, from the toolbox's clock, report and ping and its own steps. +# A consumer of the toolbox and the proton engine, with the proton image. Inside this repo +# the includes are paths; a mirror includes +# https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml and +# https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml. The pipeline is +# the engine's; render.txt, every command of it, is the library's own check of the engine, +# and `task run -- task offline` runs the verbs that need neither Proton nor a bucket. version: '3' includes: toolbox: taskfile: ../../toolbox.yml flatten: true + excludes: [report-engine] # the engine's vars: NAME: example-proton - DESC: the proton toolbox, exercised + DESC: the proton toolbox and engine, exercised IMAGE: ghcr.io/katoptra/toolbox:proton-dev LIB_DIR: ../.. + MENU: '"" "${c}engine -- offline, inside the image${r}" " task run -- task offline confirm over fixtures/, and an age round trip"' + proton: + taskfile: ../../engines/proton.yml + flatten: true + excludes: [stage] tasks: - pipeline: - desc: clock -> tools -> report -> ping - cmds: - - {task: clock} - - {task: tools} - - {task: report} - - {task: ping} - plan-pipeline: - desc: 'The read-only half: clock -> tools' + stage: + desc: One file in one folder under STAGING, so the pipeline has something to upload cmds: - - {task: clock} - - {task: tools} + - rm -rf {{.STAGING}} && mkdir -p {{.STAGING}}/example && printf 'hello\n' > {{.STAGING}}/example/hello.txt tools: desc: Every tool the proton image promises, by version cmds: - python --version - proton-drive version | head -1 - age --version + - git --version - python -c 'import boto3, requests; print(boto3.__version__, requests.__version__)' - ruff --version - pytest --version + - s3 2>&1 | head -1 + offline: + desc: The engine's verbs that need neither Proton nor a bucket, inside the image + cmds: + # confirm: one file in one folder staged; a summary that accounts for both with no + # failure confirms, one naming a failure does not. + - {task: stage} + - cp {{.ROOT_DIR}}/fixtures/upload-ok.json {{.UPLOAD}} + - {task: confirm} + - grep -q 'skipped as identical, 0 failed' {{.RUN}}/confirm.txt + - cp {{.ROOT_DIR}}/fixtures/upload-failed.json {{.UPLOAD}} + - if task confirm >/dev/null 2>&1; then echo "confirm accepted a summary with a failure" >&2; exit 1; fi + - grep -q '(not confirmed)' {{.RUN}}/confirm.txt + # age: a throwaway identity round-trips a file through the engine's verb, and the + # identity file is gone afterwards. + - MIRROR_AGE_IDENTITY="$(age-keygen 2>/dev/null)" task age-roundtrip + - test ! -e {{.RUN}}/age.key + - 'echo "offline: every check passed"' + age-roundtrip: + desc: encrypt then decrypt RUN/plain.txt through the engine's age verb, and compare + cmds: + - printf 'round trip\n' > {{.RUN}}/plain.txt + - {task: age, vars: {MODE: encrypt, IN: '{{.RUN}}/plain.txt', OUT: '{{.RUN}}/plain.age'}} + - {task: age, vars: {MODE: decrypt, IN: '{{.RUN}}/plain.age', OUT: '{{.RUN}}/plain.out'}} + - cmp {{.RUN}}/plain.txt {{.RUN}}/plain.out diff --git a/examples/proton/fixtures/upload-failed.json b/examples/proton/fixtures/upload-failed.json new file mode 100644 index 0000000..26902ec --- /dev/null +++ b/examples/proton/fixtures/upload-failed.json @@ -0,0 +1,2 @@ +{"type":"progress","name":"hello.txt","transferred":6} +{"transferredItems":1,"skippedItems":0,"failedItems":1,"failures":[{"name":"hello.txt","error":"upload failed"}]} diff --git a/examples/proton/fixtures/upload-ok.json b/examples/proton/fixtures/upload-ok.json new file mode 100644 index 0000000..9ea615c --- /dev/null +++ b/examples/proton/fixtures/upload-ok.json @@ -0,0 +1,2 @@ +{"type":"progress","name":"hello.txt","transferred":6} +{"transferredItems":1,"skippedItems":1,"failedItems":0,"failures":[]} diff --git a/examples/proton/op.env b/examples/proton/op.env index 77f81ab..119e979 100644 --- a/examples/proton/op.env +++ b/examples/proton/op.env @@ -1,5 +1,9 @@ # op:// references only; `op run --env-file=op.env` resolves them at run time. +MIRROR_PROTON_DESTINATION=op://VAULT/proton/destination +MIRROR_PROTON_DESTINATION_UID=op://VAULT/proton/destination_uid AWS_ACCESS_KEY_ID=op://VAULT/r2/access_key_id AWS_SECRET_ACCESS_KEY=op://VAULT/r2/secret_access_key -AWS_ENDPOINT_URL=op://VAULT/r2/endpoint +AWS_ENDPOINT_URL_S3=op://VAULT/r2/endpoint +MIRROR_R2_BUCKET=op://VAULT/r2/bucket +MIRROR_AGE_IDENTITY=op://VAULT/age/identity HEALTHCHECK_URL=op://VAULT/healthcheck/url diff --git a/examples/proton/render.txt b/examples/proton/render.txt index 6499d89..58dfe47 100644 --- a/examples/proton/render.txt +++ b/examples/proton/render.txt @@ -1,8 +1,99 @@ task: [clock] mkdir -p /work/examples/proton/.run && rm -f /work/examples/proton/.run/chain && date -u '+%s %H %u' > /work/examples/proton/.run/start.txt -task: [tools] python --version -task: [tools] proton-drive version | head -1 -task: [tools] age --version -task: [tools] python -c 'import boto3, requests; print(boto3.__version__, requests.__version__)' -task: [tools] ruff --version -task: [tools] pytest --version +task: [session] rm -rf /work/examples/proton/.run/session && mkdir -p /work/examples/proton/.run/session && chmod 700 /work/examples/proton/.run/session +task: [pull] rc=0; s3 get .state/session.tar.age /work/examples/proton/.run/session.tar.age || rc=$? +case $rc in 0) ;; 3) echo "no Proton session at .state/session.tar.age: log in on a laptop and run task session-seal" >&2; exit 1 ;; *) exit $rc ;; esac + +task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } +k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" +rc=0 +case decrypt in + encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age || rc=$? ;; + decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; +esac +rm -f "$k"; exit $rc + +task: [session] tar -xf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json && chmod 600 /work/examples/proton/.run/session/*.json +task: [session] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age +task: [session] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha +task: [pd] rc=0; PROTON_DRIVE_CACHE_DIR=/work/examples/proton/.run/session PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem list -j "$(dirname "$MIRROR_PROTON_DESTINATION")" > /work/examples/proton/.run/parent.json 2> /work/examples/proton/.run/pd.err || rc=$?; echo $rc > /work/examples/proton/.run/pd.rc +task: [session-push] echo "session: token rotated, sealing it back to the bucket" +task: [seal] mkdir -p /work/examples/proton/.run && tar -cf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json +task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } +k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" +rc=0 +case encrypt in + encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; +esac +rm -f "$k"; exit $rc + +task: [push] s3 put /work/examples/proton/.run/session.tar.age .state/session.tar.age +task: [seal] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age +task: [session-push] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha +task: [pd] rc=$(cat /work/examples/proton/.run/pd.rc) +case " 0 " in *" $rc "*) exit 0 ;; esac +echo "proton-drive exited $rc; its stderr, $(wc -l < /work/examples/proton/.run/pd.err) lines, is in .run/pd.err" >&2; exit $rc + +task: [destination] python3 - "$(basename "$MIRROR_PROTON_DESTINATION")" "$MIRROR_PROTON_DESTINATION_UID" /work/examples/proton/.run/parent.json <<'PY' +import json, sys +name, uid, path = sys.argv[1:] +unwrap = lambda v: v.get("value") if isinstance(v, dict) else v +nodes = json.load(open(path)) +found = [n for n in nodes if isinstance(n, dict) and unwrap(n.get("name")) == name] +if len(found) != 1 or unwrap(found[0].get("type")) != "folder": + sys.exit(f"destination: {len(found)} folder(s) of that name among {len(nodes)} entries of the parent; expected exactly one") +if not uid or str(unwrap(found[0].get("uid")) or "") != uid: + sys.exit("destination: the folder's UID is not the one the vault names") +print("destination: the folder and its UID match the vault") +PY + +task: [destination] rm -f /work/examples/proton/.run/parent.json +task: [stage] rm -rf /work/examples/proton/staging && mkdir -p /work/examples/proton/staging/example && printf 'hello\n' > /work/examples/proton/staging/example/hello.txt +task: [upload] test -n "$(ls -A /work/examples/proton/staging 2>/dev/null)" || { echo "upload: staging is empty; stage put nothing there" >&2; exit 1; } +task: [pd] rc=0; PROTON_DRIVE_CACHE_DIR=/work/examples/proton/.run/session PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem upload -f create-new-revision -d merge -t --json /work/examples/proton/staging/* "$MIRROR_PROTON_DESTINATION" > /work/examples/proton/.run/upload.json 2> /work/examples/proton/.run/pd.err || rc=$?; echo $rc > /work/examples/proton/.run/pd.rc +task: [session-push] echo "session: token rotated, sealing it back to the bucket" +task: [seal] mkdir -p /work/examples/proton/.run && tar -cf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json +task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } +k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" +rc=0 +case encrypt in + encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; +esac +rm -f "$k"; exit $rc + +task: [push] s3 put /work/examples/proton/.run/session.tar.age .state/session.tar.age +task: [seal] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age +task: [session-push] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha +task: [pd] rc=$(cat /work/examples/proton/.run/pd.rc) +case " 0 1 " in *" $rc "*) exit 0 ;; esac +echo "proton-drive exited $rc; its stderr, $(wc -l < /work/examples/proton/.run/pd.err) lines, is in .run/pd.err" >&2; exit $rc + +task: [confirm] files=$(find /work/examples/proton/staging -type f | wc -l); folders=$(find /work/examples/proton/staging -mindepth 1 -type d | wc -l) +python3 - "$files" "$folders" /work/examples/proton/.run/upload.json /work/examples/proton/.run/confirm.txt <<'PY' +import json, sys +files, folders = int(sys.argv[1]), int(sys.argv[2]) +# The CLI writes progress and its summary as one JSON object per line; the last +# line carrying transferredItems is the summary. +summary = None +for line in open(sys.argv[3]): + try: + candidate = json.loads(line) + except ValueError: + continue + if isinstance(candidate, dict) and "transferredItems" in candidate: + summary = candidate +if summary is None: + sys.exit("confirm: no upload summary in the CLI's output") +t, s, f = (int(summary.get(k, 0) or 0) for k in ("transferredItems", "skippedItems", "failedItems")) +verdict = f"{t} uploaded as new files or revisions, {s} skipped as identical, {f} failed, of {files} files in {folders} folders staged" +ok = f == 0 and t + s == files + folders +open(sys.argv[4], "w").write(verdict + ("" if ok else " (not confirmed)") + "\n") +print("confirm: " + verdict) +sys.exit(0 if ok else 1) +PY + task: [ping] test -z "$HEALTHCHECK_URL" || curl -fsS -m 10 --retry 3 -o /dev/null "$HEALTHCHECK_URL" diff --git a/toolbox.yml b/toolbox.yml index 6a63b3b..e1a03c1 100644 --- a/toolbox.yml +++ b/toolbox.yml @@ -8,11 +8,14 @@ # Rules this file keeps: no vars: default for anything a mirror owns (it would shadow # the mirror's value; defaults go inline), and host-side and in-container verbs share # one namespace, so the names below are reserved, and so is the engine vocabulary a -# second include adds beside them: pipeline, plan-pipeline, list, normalise, state, -# pull, push, rebuild, diff, split, batches, batch, fetch, publish, merge, checkpoint, -# remove, delete, reconcile, retry, and the hooks prepare, verify, index, smoke, -# smoke-mirror and report-engine. report-engine lives here as a no-op and in every -# engine, so an engine consumer's toolbox include lists it under excludes:. +# second include adds beside them. The rsync engine's: pipeline, plan-pipeline, list, +# normalise, state, pull, push, rebuild, diff, split, batches, batch, fetch, publish, +# merge, checkpoint, remove, delete, reconcile, retry, and the hooks prepare, verify, +# index, smoke and smoke-mirror. The proton engine's: session, session-push, seal, +# session-seal, age, pull, push, pd, destination, upload, confirm, list-folder, trash, +# empty-trash, empty-trash-pipeline, and the hooks stage and prune. report-engine lives +# here as a no-op and in every engine, so an engine consumer's toolbox include lists it +# under excludes:. version: '3' vars: