diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 884d396..275d5d4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,6 +30,5 @@ jobs: working-directory: examples/${{ matrix.variant }} run: task check - name: The engine's verbs that need no bucket, over the fixtures - if: matrix.variant == 'rsync' working-directory: examples/${{ matrix.variant }} run: task run -- task offline diff --git a/CLAUDE.md b/CLAUDE.md index b843e98..754d0f3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,10 +10,11 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont mirror's root value. - Verb names are reserved across host and container. `plan` is the host-side read-only run; an engine's batch planner is `split`. The hooks, `prepare`, `verify`, `index`, - `smoke`, `smoke-mirror`, `report-engine` in the engine and `report-mirror` in the - toolbox, are the only verbs a mirror redefines, each excluded on the include that - defines it; `pipeline` and `plan-pipeline` come from the engine and a mirror excludes - them only for another order. `report-engine` exists in both files, so an engine + `smoke`, `smoke-mirror`, `report-engine` in the rsync engine, `stage` and `prune` in + the proton engine, and `report-mirror` in the toolbox, are the only verbs a mirror + redefines, each excluded on the include that defines it; `pipeline` and + `plan-pipeline` come from the engine and a mirror excludes them only for another + order. `report-engine` exists in both files, so an engine consumer excludes it on the toolbox include. Extension is a hook, never a copy: a verb that needs more than the engine does gets a `-mirror` hook the engine calls last. - A mirror's root var shadows a command-line `KEY=value` inside an included verb, so an @@ -48,6 +49,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont so the container engine's own progress lines never reach `render.txt`. - Inside a `sh:` var, `printf -- '-e %s'` prints dashes: task's built-in shell takes the `--` as the format. Use `printf '%s %s ' -e "$v"`. +- Task's built-in shell has no `umask`. A file that must be born 0600 is + `install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does. +- The proton engine keeps no state of the mirror's in the bucket, only the CLI session, + because `filesystem upload` skips a file whose content Proton already holds and + `-f create-new-revision` handles one that changed. Every CLI call goes through `pd`, + which pushes the session back whatever the exit: the refresh token rotates, and a run + that kept a rotated token to itself leaves the next run unable to log in. Two mirrors + never share one session for the same reason. - Actions pinned to a full SHA with the version in a trailing comment. A mirror pins the two reusable workflows that way; each checks this repository out at its own commit (`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is @@ -58,11 +67,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont ```sh cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline -cd examples/proton && task image-build && task run -- task tools && task check +cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline ``` A verb change updates the `render.txt` files via `task render-update`; `offline` is -the engine's own check over `examples/rsync/fixtures/`: the list diff over `run-root` -and `run-empty`, `retry`'s exit codes, and `prepare` and `verify` over `tree/`, a signed -subtree whose tlpdb is signed by a throwaway key pinned in the example. Regenerate the -tree with a new key only to change its shape; the private half was never kept. +each engine's own check. The rsync one runs over `examples/rsync/fixtures/`: the list +diff over `run-root` and `run-empty`, `retry`'s exit codes, and `prepare` and `verify` +over `tree/`, a signed subtree whose tlpdb is signed by a throwaway key pinned in the +example. Regenerate the tree with a new key only to change its shape; the private half +was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an +accepting and a refusing upload summary, and the `age` verb round trip with a throwaway +identity. diff --git a/README.md b/README.md index ec7e2ee..6880fc2 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,9 @@ The toolbox every katoptra mirror includes by URL. The rule it enforces: the code that starts a run, contains it, resolves its secrets, checks it and reports it lives here, -once. The code that moves bytes for a transport lives here too, once per engine, and -`engines/rsync.yml` is the first. A mirror holds only its identity, the order of its +once. The code that moves bytes for a transport lives here too, once per engine: +`engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a +staging tree into Proton Drive. A mirror holds only its identity, the order of its pipeline, and the few verbs no other mirror shares. ## The layers @@ -266,6 +267,60 @@ so an hourly run costs one listing of upstream and none of the bucket. Two thing What a mirror cannot afford to lose is the bucket. Everything else, the state file and the staging tree included, is derived from it and from upstream. +### The proton engine + +`engines/proton.yml` moves a staging tree into one Proton Drive folder, through the +official `proton-drive` CLI, for a mirror whose upstream fits in a run: the mirror fills +`staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket +but the CLI session, because the CLI skips a file whose content Proton already holds and +`-f create-new-revision` makes a revision of one that changed; Proton's version history +is the history of the mirror. + +| Verb | Does | +|---|---| +| `pipeline`, `plan-pipeline` | `clock`, `session`, `destination`, `stage`, `upload`, `confirm`, `prune`, `report`, `ping`; and the read-only half, through `stage` | +| `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` | +| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit | +| `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` | +| `stage` | Hook. The mirror fills `staging/` with what Proton should hold | +| `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` | +| `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` | +| `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` | +| `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash | +| `session-seal -- ` | Host side: a laptop login's two files, encrypted into the bucket | +| `empty-trash` | Host side, asks first: everything in Proton's trash, permanently | +| `report-engine` | Hook. The engine's rows of the run summary | + +The engine reads no root var. Its inputs are the environment, by the names every Proton +mirror's `op.env` carries: `MIRROR_PROTON_DESTINATION` and its `_UID`, `MIRROR_R2_BUCKET`, +`MIRROR_AGE_IDENTITY`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and +`AWS_ENDPOINT_URL_S3`. The session is the one thing to set up by hand: the CLI can only be +seeded by a browser sign-in, so log in once on a laptop into a directory inside the repo +and run `task session-seal -- `. Two mirrors never share a session; its +refresh token rotates on every call, and the loser of a race needs a fresh login. + +A mirror of every repository under two GitHub owners, shaped like github: + +```yaml +version: '3' +vars: + OWNERS: jshvn katoptra +includes: + toolbox: + taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml + flatten: true + excludes: [report-engine, report-mirror] + vars: {NAME: github, DESC: a nightly mirror of every repository under jshvn and katoptra into Proton Drive, IMAGE: ghcr.io/katoptra/toolbox:proton-v2} + proton: + taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml + flatten: true + excludes: [stage, prune] +tasks: + stage: {cmds: ['# list the repositories, clone each as a mirror, bundle it under {{.STAGING}}//']} + prune: {cmds: ['# list-folder each owner in Proton; trash the bundles no repository has']} + report-mirror: {cmds: ['# the mirror rows']} +``` + ## Overriding a verb List it under `excludes:` on the include that defines it and define it in the mirror. @@ -314,7 +369,7 @@ two matrix entries: | Variant | Base | Tools | For | |---|---|---|---| | `rsync` | ubuntu 24.04 | rsync, gnupg, xz, curl, perl, go-task, AWS CLI v2 (s3, sts) | rsync upstreams to R2: ctan, tlnet, cran, cpan | -| `proton` | python 3.13 slim | proton-drive, age, go-task, boto3, requests, pytest, ruff | Python pipelines: dropbox, photos | +| `proton` | python 3.13 slim | proton-drive, age, git, go-task, boto3, requests, pytest, ruff, and `s3`, a boto3 get/put | Proton Drive sinks: github through the proton engine, dropbox through its own Python | An HTTPS engine would be a third row: the same base as `rsync`, curl and the AWS CLI, and a `list` that reads an index instead of `rsync --list-only`. diff --git a/docker/proton.Dockerfile b/docker/proton.Dockerfile index b2e74ec..877f95e 100644 --- a/docker/proton.Dockerfile +++ b/docker/proton.Dockerfile @@ -1,7 +1,8 @@ # syntax=docker/dockerfile:1.7 -# The proton toolbox: Python plus the Proton Drive CLI and age, for the mirrors whose -# engine is a Python package. The repo is bind-mounted at /work; PYTHONPATH finds its -# src/. Base pinned by digest; every tool from toolchain.lock.toml at the build context. +# The proton toolbox: Python plus the Proton Drive CLI, age and git, for the mirrors whose +# sink is Proton Drive, whether their engine is engines/proton.yml or a Python package of +# their own. The repo is bind-mounted at /work; PYTHONPATH finds its src/. Base pinned by +# digest; every tool from toolchain.lock.toml at the build context, git and curl from apt. FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS fetch RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \ @@ -26,10 +27,11 @@ RUN set -eu; \ FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS toolbox -RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \ +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl git \ && rm -rf /var/lib/apt/lists/* COPY --from=fetch /usr/local/bin/proton-drive /usr/local/bin/age /usr/local/bin/age-keygen /usr/local/bin/task /usr/local/bin/ COPY toolchain.lock.toml /etc/toolchain.lock.toml +COPY docker/s3.py /usr/local/bin/s3 RUN python - <<'PY' import subprocess, tomllib @@ -49,12 +51,17 @@ first = lambda argv: subprocess.check_output(argv, text=True).splitlines()[0] assert "@" + lock["proton_drive_cli"]["version"] in first(["proton-drive", "version"]) assert lock["age"]["version"] in first(["age", "--version"]) assert lock["task"]["version"] in subprocess.check_output(["task", "--version"], text=True) +assert first(["git", "--version"]).startswith("git version 2.") +assert subprocess.run(["s3"], capture_output=True).returncode == 2 # usage; boto3 imports PY # Inside a run there is no network for Taskfiles: the mirror's .task/remote cache rides -# in with the bind mount. R2 has one region; the value is a literal, not a secret. +# in with the bind mount. R2 has one region and rejects the SDK's default checksum +# headers; both values are literals, not secrets. ENV PYTHONUNBUFFERED=1 \ PYTHONPATH=/work/src \ TASK_REMOTE_OFFLINE=1 \ - AWS_REGION=auto + AWS_REGION=auto \ + AWS_REQUEST_CHECKSUM_CALCULATION=when_required \ + AWS_RESPONSE_CHECKSUM_VALIDATION=when_required WORKDIR /work diff --git a/docker/s3.py b/docker/s3.py new file mode 100755 index 0000000..af2176c --- /dev/null +++ b/docker/s3.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""s3 get KEY FILE | s3 put FILE KEY: one object of the bucket MIRROR_R2_BUCKET names. + +The proton image's S3 client, for the mirrors whose engine keeps a Proton session in the +bucket. Credentials and the endpoint come from the environment as boto3 reads them: +AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL_S3. `get` exits 3 when the +key is absent, so a caller can tell a missing object from a failed read. A failure prints +its class alone: the message may carry the endpoint, and the logs are public. +""" + +import os +import sys + +import boto3 +from botocore.config import Config +from botocore.exceptions import BotoCoreError, ClientError + +MISSING = {"404", "NoSuchKey"} + + +def main(argv: list[str]) -> int: + if len(argv) != 4 or argv[1] not in {"get", "put"}: + print(__doc__.strip(), file=sys.stderr) + return 2 + names = ("MIRROR_R2_BUCKET", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_ENDPOINT_URL_S3") + unset = [n for n in names if not os.environ.get(n)] + if unset: + print(f"s3: unset in the environment: {' '.join(unset)}", file=sys.stderr) + return 2 + bucket = os.environ["MIRROR_R2_BUCKET"] + client = boto3.client("s3", config=Config(retries={"max_attempts": 5, "mode": "standard"})) + try: + if argv[1] == "get": + client.download_file(bucket, argv[2], argv[3]) + else: + client.upload_file(argv[2], bucket, argv[3]) + except ClientError as exc: + if argv[1] == "get" and exc.response.get("Error", {}).get("Code") in MISSING: + return 3 + print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr) + return 1 + except BotoCoreError as exc: + print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/engines/proton.yml b/engines/proton.yml new file mode 100644 index 0000000..16de42a --- /dev/null +++ b/engines/proton.yml @@ -0,0 +1,240 @@ +# The proton engine: how bytes move from a staging tree into one Proton Drive folder, for +# every mirror that includes this file flattened beside the toolbox. A run restores the +# Proton CLI session from the bucket, checks that the destination is the folder the vault +# names, lets the mirror fill the staging tree, uploads it in one CLI call (the CLI skips a +# file whose content Proton already holds and makes a new revision of one that changed), +# confirms the upload from the CLI's own summary, lets the mirror prune what its upstream +# dropped, and reports. Proton's version history is the cloud-side history, so the bucket +# holds nothing of the mirror's but the session. +# +# Read from the environment, by the names every Proton mirror's vault uses: +# MIRROR_PROTON_DESTINATION (the CLI path of the destination folder, /my-files/GitHub +# say), MIRROR_PROTON_DESTINATION_UID (its UID, compared on every run before a write), +# MIRROR_R2_BUCKET, MIRROR_AGE_IDENTITY, and the three AWS_* names boto3 reads, with +# AWS_ENDPOINT_URL_S3 for the endpoint. Each is a line of op.env, so it crosses into the +# container by name. +# +# Hooks: stage fills STAGING and every mirror defines it; prune trashes what upstream no +# longer has, and a mirror that wants that defines it. A mirror lists each it defines under +# excludes: on this include. pipeline and plan-pipeline are the vocabulary in order. +# +# Rules this file keeps, as the toolbox does: no vars: default for anything a mirror +# owns, and the CLI session is written back to the bucket after every CLI call, whatever +# the exit, because its refresh token rotates and a run that kept a rotated token to +# itself leaves the next run unable to log in. +version: '3' + +vars: + STAGING: '{{.ROOT_DIR}}/staging' + SESSION: '{{.ROOT_DIR}}/.run/session' + SESSION_KEY: .state/session.tar.age + UPLOAD: '{{.ROOT_DIR}}/.run/upload.json' + # The CLI with its session as plain files under RUN/session: the image has no keyring. + PD: PROTON_DRIVE_CACHE_DIR={{.SESSION}} PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive + +tasks: + pipeline: + desc: clock -> session -> destination -> stage -> upload -> confirm -> prune -> report -> ping + cmds: + - {task: clock} + - {task: session} + - {task: destination} + - {task: stage} + - {task: upload} + - {task: confirm} + - {task: prune} + - {task: report} + - {task: ping} + + plan-pipeline: + desc: 'The read-only half: clock -> session -> destination -> stage, then what an upload would carry' + cmds: + - {task: clock} + - {task: session} + - {task: destination} + - {task: stage} + - | + find {{.STAGING}} -type f -printf '%s\n' | awk '{ n++; s += $1 } END { printf "plan: %d files, %.1f MB staged\n", n, s / 1e6 }' + + # ---- the session ---- + session: + desc: Restore the Proton CLI session from the bucket into RUN/session + cmds: + - rm -rf {{.SESSION}} && mkdir -p {{.SESSION}} && chmod 700 {{.SESSION}} + - {task: pull, vars: {KEY: '{{.SESSION_KEY}}', OUT: '{{.RUN}}/session.tar.age', MISSING: 'echo "no Proton session at {{.SESSION_KEY}}: log in on a laptop and run task session-seal" >&2; exit 1'}} + - {task: age, vars: {MODE: decrypt, IN: '{{.RUN}}/session.tar.age', OUT: '{{.RUN}}/session.tar'}} + # The two members the CLI needs, and nothing else the archive might hold. + - tar -xf {{.RUN}}/session.tar -C {{.SESSION}} auth-session.json clientUid.json && chmod 600 {{.SESSION}}/*.json + - rm -f {{.RUN}}/session.tar {{.RUN}}/session.tar.age + - sha256sum {{.SESSION}}/auth-session.json > {{.RUN}}/session.sha + + session-push: + desc: Seal RUN/session back into the bucket when the CLI rotated its token since the last push + internal: true + status: ['sha256sum --status -c {{.RUN}}/session.sha'] + cmds: + - 'echo "session: token rotated, sealing it back to the bucket"' + - {task: seal, vars: {DIR: '{{.SESSION}}'}} + - sha256sum {{.SESSION}}/auth-session.json > {{.RUN}}/session.sha + + seal: + desc: 'The two session files of DIR, tarred and age-encrypted, to the bucket: task seal DIR=.run/pd' + requires: {vars: [DIR]} + cmds: + - defer: rm -f {{.RUN}}/session.tar {{.RUN}}/session.tar.age + - mkdir -p {{.RUN}} && tar -cf {{.RUN}}/session.tar -C {{.DIR}} auth-session.json clientUid.json + - {task: age, vars: {MODE: encrypt, IN: '{{.RUN}}/session.tar', OUT: '{{.RUN}}/session.tar.age'}} + - {task: push, vars: {FILE: '{{.RUN}}/session.tar.age', KEY: '{{.SESSION_KEY}}'}} + + session-seal: + desc: 'Encrypt a laptop Proton CLI session into the bucket: task session-seal -- .run/pd' + cmds: + - {task: op, vars: {ARGS: 'task seal DIR={{.CLI_ARGS}}'}} + + age: + desc: age MODE (encrypt or decrypt) IN to OUT with MIRROR_AGE_IDENTITY, which touches disk for the call alone, mode 0600 + internal: true + requires: {vars: [MODE, IN, OUT]} + cmds: + - | + test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } + k={{.RUN}}/age.key; mkdir -p {{.RUN}}; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" + rc=0 + case {{.MODE}} in + encrypt) age -r "$(age-keygen -y "$k")" -o {{.OUT}} {{.IN}} || rc=$? ;; + decrypt) age -d -i "$k" -o {{.OUT}} {{.IN}} || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; + esac + rm -f "$k"; exit $rc + + # ---- the bucket ---- + pull: + desc: KEY from the bucket to OUT; MISSING runs when the bucket has no such key + internal: true + requires: {vars: [KEY, OUT, MISSING]} + cmds: + - | + rc=0; s3 get {{.KEY}} {{.OUT}} || rc=$? + case $rc in 0) ;; 3) {{.MISSING}} ;; *) exit $rc ;; esac + + push: + desc: FILE to the bucket's KEY as one PutObject + internal: true + requires: {vars: [FILE, KEY]} + cmds: + - s3 put {{.FILE}} {{.KEY}} + + # ---- the CLI ---- + # Every CLI call goes through here: stderr to RUN/pd.err rather than the public log, + # the session pushed back whatever the exit, then the exit judged against OK, the codes + # accepted besides 0. + pd: + desc: One CLI call, ARGS, with the session written back afterwards + internal: true + requires: {vars: [ARGS]} + cmds: + - rc=0; {{.PD}} {{.ARGS}} 2> {{.RUN}}/pd.err || rc=$?; echo $rc > {{.RUN}}/pd.rc + - {task: session-push} + - | + rc=$(cat {{.RUN}}/pd.rc) + case " {{.OK | default 0}} " in *" $rc "*) exit 0 ;; esac + echo "proton-drive exited $rc; its stderr, $(wc -l < {{.RUN}}/pd.err) lines, is in .run/pd.err" >&2; exit $rc + + destination: + desc: Refuse the run unless MIRROR_PROTON_DESTINATION resolves to exactly one folder whose UID is MIRROR_PROTON_DESTINATION_UID + cmds: + - defer: rm -f {{.RUN}}/parent.json + - {task: pd, vars: {ARGS: 'filesystem list -j "$(dirname "$MIRROR_PROTON_DESTINATION")" > {{.RUN}}/parent.json'}} + # ponytail: the folder is found by basename, so a destination whose own name holds a + # slash is unsupported; the CLI escapes one with a backslash and dirname does not. + - | + python3 - "$(basename "$MIRROR_PROTON_DESTINATION")" "$MIRROR_PROTON_DESTINATION_UID" {{.RUN}}/parent.json <<'PY' + import json, sys + name, uid, path = sys.argv[1:] + unwrap = lambda v: v.get("value") if isinstance(v, dict) else v + nodes = json.load(open(path)) + found = [n for n in nodes if isinstance(n, dict) and unwrap(n.get("name")) == name] + if len(found) != 1 or unwrap(found[0].get("type")) != "folder": + sys.exit(f"destination: {len(found)} folder(s) of that name among {len(nodes)} entries of the parent; expected exactly one") + if not uid or str(unwrap(found[0].get("uid")) or "") != uid: + sys.exit("destination: the folder's UID is not the one the vault names") + print("destination: the folder and its UID match the vault") + PY + + upload: + desc: STAGING into the destination in one CLI call; identical files are skipped, changed ones become new revisions; the CLI's summary goes to RUN/upload.json + cmds: + - 'test -n "$(ls -A {{.STAGING}} 2>/dev/null)" || { echo "upload: staging is empty; stage put nothing there" >&2; exit 1; }' + # Exit 1 is the CLI refusing some items and handling the rest; confirm reads the summary. + - {task: pd, vars: {ARGS: 'filesystem upload -f create-new-revision -d merge -t --json {{.STAGING}}/* "$MIRROR_PROTON_DESTINATION" > {{.UPLOAD}}', OK: '0 1'}} + + confirm: + desc: The CLI's summary must account for every staged file and folder, with no failures; the verdict goes to RUN/confirm.txt + cmds: + - | + files=$(find {{.STAGING}} -type f | wc -l); folders=$(find {{.STAGING}} -mindepth 1 -type d | wc -l) + python3 - "$files" "$folders" {{.UPLOAD}} {{.RUN}}/confirm.txt <<'PY' + import json, sys + files, folders = int(sys.argv[1]), int(sys.argv[2]) + # The CLI writes progress and its summary as one JSON object per line; the last + # line carrying transferredItems is the summary. + summary = None + for line in open(sys.argv[3]): + try: + candidate = json.loads(line) + except ValueError: + continue + if isinstance(candidate, dict) and "transferredItems" in candidate: + summary = candidate + if summary is None: + sys.exit("confirm: no upload summary in the CLI's output") + t, s, f = (int(summary.get(k, 0) or 0) for k in ("transferredItems", "skippedItems", "failedItems")) + verdict = f"{t} uploaded as new files or revisions, {s} skipped as identical, {f} failed, of {files} files in {folders} folders staged" + ok = f == 0 and t + s == files + folders + open(sys.argv[4], "w").write(verdict + ("" if ok else " (not confirmed)") + "\n") + print("confirm: " + verdict) + sys.exit(0 if ok else 1) + PY + + list-folder: + desc: The JSON listing of the Proton folder FOLDER to OUT; a folder the CLI cannot list gives an empty one, so a caller prunes nothing + internal: true + requires: {vars: [FOLDER, OUT]} + cmds: + - {task: pd, vars: {ARGS: 'filesystem list -j "{{.FOLDER}}" > {{.OUT}} || echo "[]" > {{.OUT}}'}} + + trash: + desc: Trash the Proton nodes at PATHS, CLI paths space separated; Proton's trash keeps them + status: ['test -z "{{.PATHS}}"'] + cmds: + - {task: pd, vars: {ARGS: 'filesystem trash {{.PATHS}}'}} + + empty-trash: + desc: Permanently delete everything in Proton Drive's trash, inside the toolbox; asks first, never scheduled + prompt: This permanently deletes everything in Proton Drive trash. Continue? + cmds: + - {task: op, vars: {ARGS: task empty-trash-pipeline}} + + empty-trash-pipeline: + internal: true + cmds: + - {task: clock} + - {task: session} + - {task: pd, vars: {ARGS: filesystem empty-trash}} + + # ---- hooks ---- + stage: {cmds: []} # the mirror's: fill STAGING with what Proton should hold + prune: {cmds: []} # the mirror's: trash what its upstream no longer has + + report-engine: + desc: The engine's rows of the run summary, appended after the toolbox's + silent: true + cmds: + - | + staged=$(find {{.STAGING}} -type f 2>/dev/null | wc -l) + mb=$(find {{.STAGING}} -type f -printf '%s\n' 2>/dev/null | awk '{ s += $1 } END { printf "%.1f", s / 1e6 }') + cat >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" </dev/null || echo "not confirmed this run") | + | Session | $(test -f {{.RUN}}/session.sha && echo "restored, and sealed back on rotation" || echo "not restored") | + EOF diff --git a/examples/proton/Taskfile.yml b/examples/proton/Taskfile.yml index ed0c08e..02341b8 100644 --- a/examples/proton/Taskfile.yml +++ b/examples/proton/Taskfile.yml @@ -1,34 +1,62 @@ -# A consumer of the toolbox with the proton image. No engine, so the mirror defines the -# two pipeline verbs itself, from the toolbox's clock, report and ping and its own steps. +# A consumer of the toolbox and the proton engine, with the proton image. Inside this repo +# the includes are paths; a mirror includes +# https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml and +# https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml. The pipeline is +# the engine's; render.txt, every command of it, is the library's own check of the engine, +# and `task run -- task offline` runs the verbs that need neither Proton nor a bucket. version: '3' includes: toolbox: taskfile: ../../toolbox.yml flatten: true + excludes: [report-engine] # the engine's vars: NAME: example-proton - DESC: the proton toolbox, exercised + DESC: the proton toolbox and engine, exercised IMAGE: ghcr.io/katoptra/toolbox:proton-dev LIB_DIR: ../.. + MENU: '"" "${c}engine -- offline, inside the image${r}" " task run -- task offline confirm over fixtures/, and an age round trip"' + proton: + taskfile: ../../engines/proton.yml + flatten: true + excludes: [stage] tasks: - pipeline: - desc: clock -> tools -> report -> ping - cmds: - - {task: clock} - - {task: tools} - - {task: report} - - {task: ping} - plan-pipeline: - desc: 'The read-only half: clock -> tools' + stage: + desc: One file in one folder under STAGING, so the pipeline has something to upload cmds: - - {task: clock} - - {task: tools} + - rm -rf {{.STAGING}} && mkdir -p {{.STAGING}}/example && printf 'hello\n' > {{.STAGING}}/example/hello.txt tools: desc: Every tool the proton image promises, by version cmds: - python --version - proton-drive version | head -1 - age --version + - git --version - python -c 'import boto3, requests; print(boto3.__version__, requests.__version__)' - ruff --version - pytest --version + - s3 2>&1 | head -1 + offline: + desc: The engine's verbs that need neither Proton nor a bucket, inside the image + cmds: + # confirm: one file in one folder staged; a summary that accounts for both with no + # failure confirms, one naming a failure does not. + - {task: stage} + - cp {{.ROOT_DIR}}/fixtures/upload-ok.json {{.UPLOAD}} + - {task: confirm} + - grep -q 'skipped as identical, 0 failed' {{.RUN}}/confirm.txt + - cp {{.ROOT_DIR}}/fixtures/upload-failed.json {{.UPLOAD}} + - if task confirm >/dev/null 2>&1; then echo "confirm accepted a summary with a failure" >&2; exit 1; fi + - grep -q '(not confirmed)' {{.RUN}}/confirm.txt + # age: a throwaway identity round-trips a file through the engine's verb, and the + # identity file is gone afterwards. + - MIRROR_AGE_IDENTITY="$(age-keygen 2>/dev/null)" task age-roundtrip + - test ! -e {{.RUN}}/age.key + - 'echo "offline: every check passed"' + age-roundtrip: + desc: encrypt then decrypt RUN/plain.txt through the engine's age verb, and compare + cmds: + - printf 'round trip\n' > {{.RUN}}/plain.txt + - {task: age, vars: {MODE: encrypt, IN: '{{.RUN}}/plain.txt', OUT: '{{.RUN}}/plain.age'}} + - {task: age, vars: {MODE: decrypt, IN: '{{.RUN}}/plain.age', OUT: '{{.RUN}}/plain.out'}} + - cmp {{.RUN}}/plain.txt {{.RUN}}/plain.out diff --git a/examples/proton/fixtures/upload-failed.json b/examples/proton/fixtures/upload-failed.json new file mode 100644 index 0000000..26902ec --- /dev/null +++ b/examples/proton/fixtures/upload-failed.json @@ -0,0 +1,2 @@ +{"type":"progress","name":"hello.txt","transferred":6} +{"transferredItems":1,"skippedItems":0,"failedItems":1,"failures":[{"name":"hello.txt","error":"upload failed"}]} diff --git a/examples/proton/fixtures/upload-ok.json b/examples/proton/fixtures/upload-ok.json new file mode 100644 index 0000000..9ea615c --- /dev/null +++ b/examples/proton/fixtures/upload-ok.json @@ -0,0 +1,2 @@ +{"type":"progress","name":"hello.txt","transferred":6} +{"transferredItems":1,"skippedItems":1,"failedItems":0,"failures":[]} diff --git a/examples/proton/op.env b/examples/proton/op.env index 77f81ab..119e979 100644 --- a/examples/proton/op.env +++ b/examples/proton/op.env @@ -1,5 +1,9 @@ # op:// references only; `op run --env-file=op.env` resolves them at run time. +MIRROR_PROTON_DESTINATION=op://VAULT/proton/destination +MIRROR_PROTON_DESTINATION_UID=op://VAULT/proton/destination_uid AWS_ACCESS_KEY_ID=op://VAULT/r2/access_key_id AWS_SECRET_ACCESS_KEY=op://VAULT/r2/secret_access_key -AWS_ENDPOINT_URL=op://VAULT/r2/endpoint +AWS_ENDPOINT_URL_S3=op://VAULT/r2/endpoint +MIRROR_R2_BUCKET=op://VAULT/r2/bucket +MIRROR_AGE_IDENTITY=op://VAULT/age/identity HEALTHCHECK_URL=op://VAULT/healthcheck/url diff --git a/examples/proton/render.txt b/examples/proton/render.txt index 6499d89..58dfe47 100644 --- a/examples/proton/render.txt +++ b/examples/proton/render.txt @@ -1,8 +1,99 @@ task: [clock] mkdir -p /work/examples/proton/.run && rm -f /work/examples/proton/.run/chain && date -u '+%s %H %u' > /work/examples/proton/.run/start.txt -task: [tools] python --version -task: [tools] proton-drive version | head -1 -task: [tools] age --version -task: [tools] python -c 'import boto3, requests; print(boto3.__version__, requests.__version__)' -task: [tools] ruff --version -task: [tools] pytest --version +task: [session] rm -rf /work/examples/proton/.run/session && mkdir -p /work/examples/proton/.run/session && chmod 700 /work/examples/proton/.run/session +task: [pull] rc=0; s3 get .state/session.tar.age /work/examples/proton/.run/session.tar.age || rc=$? +case $rc in 0) ;; 3) echo "no Proton session at .state/session.tar.age: log in on a laptop and run task session-seal" >&2; exit 1 ;; *) exit $rc ;; esac + +task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } +k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" +rc=0 +case decrypt in + encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age || rc=$? ;; + decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; +esac +rm -f "$k"; exit $rc + +task: [session] tar -xf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json && chmod 600 /work/examples/proton/.run/session/*.json +task: [session] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age +task: [session] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha +task: [pd] rc=0; PROTON_DRIVE_CACHE_DIR=/work/examples/proton/.run/session PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem list -j "$(dirname "$MIRROR_PROTON_DESTINATION")" > /work/examples/proton/.run/parent.json 2> /work/examples/proton/.run/pd.err || rc=$?; echo $rc > /work/examples/proton/.run/pd.rc +task: [session-push] echo "session: token rotated, sealing it back to the bucket" +task: [seal] mkdir -p /work/examples/proton/.run && tar -cf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json +task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } +k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" +rc=0 +case encrypt in + encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; +esac +rm -f "$k"; exit $rc + +task: [push] s3 put /work/examples/proton/.run/session.tar.age .state/session.tar.age +task: [seal] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age +task: [session-push] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha +task: [pd] rc=$(cat /work/examples/proton/.run/pd.rc) +case " 0 " in *" $rc "*) exit 0 ;; esac +echo "proton-drive exited $rc; its stderr, $(wc -l < /work/examples/proton/.run/pd.err) lines, is in .run/pd.err" >&2; exit $rc + +task: [destination] python3 - "$(basename "$MIRROR_PROTON_DESTINATION")" "$MIRROR_PROTON_DESTINATION_UID" /work/examples/proton/.run/parent.json <<'PY' +import json, sys +name, uid, path = sys.argv[1:] +unwrap = lambda v: v.get("value") if isinstance(v, dict) else v +nodes = json.load(open(path)) +found = [n for n in nodes if isinstance(n, dict) and unwrap(n.get("name")) == name] +if len(found) != 1 or unwrap(found[0].get("type")) != "folder": + sys.exit(f"destination: {len(found)} folder(s) of that name among {len(nodes)} entries of the parent; expected exactly one") +if not uid or str(unwrap(found[0].get("uid")) or "") != uid: + sys.exit("destination: the folder's UID is not the one the vault names") +print("destination: the folder and its UID match the vault") +PY + +task: [destination] rm -f /work/examples/proton/.run/parent.json +task: [stage] rm -rf /work/examples/proton/staging && mkdir -p /work/examples/proton/staging/example && printf 'hello\n' > /work/examples/proton/staging/example/hello.txt +task: [upload] test -n "$(ls -A /work/examples/proton/staging 2>/dev/null)" || { echo "upload: staging is empty; stage put nothing there" >&2; exit 1; } +task: [pd] rc=0; PROTON_DRIVE_CACHE_DIR=/work/examples/proton/.run/session PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem upload -f create-new-revision -d merge -t --json /work/examples/proton/staging/* "$MIRROR_PROTON_DESTINATION" > /work/examples/proton/.run/upload.json 2> /work/examples/proton/.run/pd.err || rc=$?; echo $rc > /work/examples/proton/.run/pd.rc +task: [session-push] echo "session: token rotated, sealing it back to the bucket" +task: [seal] mkdir -p /work/examples/proton/.run && tar -cf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json +task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; } +k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k" +rc=0 +case encrypt in + encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;; + *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;; +esac +rm -f "$k"; exit $rc + +task: [push] s3 put /work/examples/proton/.run/session.tar.age .state/session.tar.age +task: [seal] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age +task: [session-push] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha +task: [pd] rc=$(cat /work/examples/proton/.run/pd.rc) +case " 0 1 " in *" $rc "*) exit 0 ;; esac +echo "proton-drive exited $rc; its stderr, $(wc -l < /work/examples/proton/.run/pd.err) lines, is in .run/pd.err" >&2; exit $rc + +task: [confirm] files=$(find /work/examples/proton/staging -type f | wc -l); folders=$(find /work/examples/proton/staging -mindepth 1 -type d | wc -l) +python3 - "$files" "$folders" /work/examples/proton/.run/upload.json /work/examples/proton/.run/confirm.txt <<'PY' +import json, sys +files, folders = int(sys.argv[1]), int(sys.argv[2]) +# The CLI writes progress and its summary as one JSON object per line; the last +# line carrying transferredItems is the summary. +summary = None +for line in open(sys.argv[3]): + try: + candidate = json.loads(line) + except ValueError: + continue + if isinstance(candidate, dict) and "transferredItems" in candidate: + summary = candidate +if summary is None: + sys.exit("confirm: no upload summary in the CLI's output") +t, s, f = (int(summary.get(k, 0) or 0) for k in ("transferredItems", "skippedItems", "failedItems")) +verdict = f"{t} uploaded as new files or revisions, {s} skipped as identical, {f} failed, of {files} files in {folders} folders staged" +ok = f == 0 and t + s == files + folders +open(sys.argv[4], "w").write(verdict + ("" if ok else " (not confirmed)") + "\n") +print("confirm: " + verdict) +sys.exit(0 if ok else 1) +PY + task: [ping] test -z "$HEALTHCHECK_URL" || curl -fsS -m 10 --retry 3 -o /dev/null "$HEALTHCHECK_URL" diff --git a/toolbox.yml b/toolbox.yml index 6a63b3b..e1a03c1 100644 --- a/toolbox.yml +++ b/toolbox.yml @@ -8,11 +8,14 @@ # Rules this file keeps: no vars: default for anything a mirror owns (it would shadow # the mirror's value; defaults go inline), and host-side and in-container verbs share # one namespace, so the names below are reserved, and so is the engine vocabulary a -# second include adds beside them: pipeline, plan-pipeline, list, normalise, state, -# pull, push, rebuild, diff, split, batches, batch, fetch, publish, merge, checkpoint, -# remove, delete, reconcile, retry, and the hooks prepare, verify, index, smoke, -# smoke-mirror and report-engine. report-engine lives here as a no-op and in every -# engine, so an engine consumer's toolbox include lists it under excludes:. +# second include adds beside them. The rsync engine's: pipeline, plan-pipeline, list, +# normalise, state, pull, push, rebuild, diff, split, batches, batch, fetch, publish, +# merge, checkpoint, remove, delete, reconcile, retry, and the hooks prepare, verify, +# index, smoke and smoke-mirror. The proton engine's: session, session-push, seal, +# session-seal, age, pull, push, pd, destination, upload, confirm, list-folder, trash, +# empty-trash, empty-trash-pipeline, and the hooks stage and prune. report-engine lives +# here as a no-op and in every engine, so an engine consumer's toolbox include lists it +# under excludes:. version: '3' vars: