diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 884d396..275d5d4 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -30,6 +30,5 @@ jobs:
working-directory: examples/${{ matrix.variant }}
run: task check
- name: The engine's verbs that need no bucket, over the fixtures
- if: matrix.variant == 'rsync'
working-directory: examples/${{ matrix.variant }}
run: task run -- task offline
diff --git a/CLAUDE.md b/CLAUDE.md
index b843e98..754d0f3 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -10,10 +10,11 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont
mirror's root value.
- Verb names are reserved across host and container. `plan` is the host-side read-only
run; an engine's batch planner is `split`. The hooks, `prepare`, `verify`, `index`,
- `smoke`, `smoke-mirror`, `report-engine` in the engine and `report-mirror` in the
- toolbox, are the only verbs a mirror redefines, each excluded on the include that
- defines it; `pipeline` and `plan-pipeline` come from the engine and a mirror excludes
- them only for another order. `report-engine` exists in both files, so an engine
+ `smoke`, `smoke-mirror`, `report-engine` in the rsync engine, `stage` and `prune` in
+ the proton engine, and `report-mirror` in the toolbox, are the only verbs a mirror
+ redefines, each excluded on the include that defines it; `pipeline` and
+ `plan-pipeline` come from the engine and a mirror excludes them only for another
+ order. `report-engine` exists in both files, so an engine
consumer excludes it on the toolbox include. Extension is a hook, never a copy: a
verb that needs more than the engine does gets a `-mirror` hook the engine calls last.
- A mirror's root var shadows a command-line `KEY=value` inside an included verb, so an
@@ -48,6 +49,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont
so the container engine's own progress lines never reach `render.txt`.
- Inside a `sh:` var, `printf -- '-e %s'` prints dashes: task's built-in shell takes
the `--` as the format. Use `printf '%s %s ' -e "$v"`.
+- Task's built-in shell has no `umask`. A file that must be born 0600 is
+ `install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does.
+- The proton engine keeps no state of the mirror's in the bucket, only the CLI session,
+ because `filesystem upload` skips a file whose content Proton already holds and
+ `-f create-new-revision` handles one that changed. Every CLI call goes through `pd`,
+ which pushes the session back whatever the exit: the refresh token rotates, and a run
+ that kept a rotated token to itself leaves the next run unable to log in. Two mirrors
+ never share one session for the same reason.
- Actions pinned to a full SHA with the version in a trailing comment. A mirror pins the
two reusable workflows that way; each checks this repository out at its own commit
(`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is
@@ -58,11 +67,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont
```sh
cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline
-cd examples/proton && task image-build && task run -- task tools && task check
+cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline
```
A verb change updates the `render.txt` files via `task render-update`; `offline` is
-the engine's own check over `examples/rsync/fixtures/`: the list diff over `run-root`
-and `run-empty`, `retry`'s exit codes, and `prepare` and `verify` over `tree/`, a signed
-subtree whose tlpdb is signed by a throwaway key pinned in the example. Regenerate the
-tree with a new key only to change its shape; the private half was never kept.
+each engine's own check. The rsync one runs over `examples/rsync/fixtures/`: the list
+diff over `run-root` and `run-empty`, `retry`'s exit codes, and `prepare` and `verify`
+over `tree/`, a signed subtree whose tlpdb is signed by a throwaway key pinned in the
+example. Regenerate the tree with a new key only to change its shape; the private half
+was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an
+accepting and a refusing upload summary, and the `age` verb round trip with a throwaway
+identity.
diff --git a/README.md b/README.md
index ec7e2ee..6880fc2 100644
--- a/README.md
+++ b/README.md
@@ -5,8 +5,9 @@
The toolbox every katoptra mirror includes by URL. The rule it enforces: the code that
starts a run, contains it, resolves its secrets, checks it and reports it lives here,
-once. The code that moves bytes for a transport lives here too, once per engine, and
-`engines/rsync.yml` is the first. A mirror holds only its identity, the order of its
+once. The code that moves bytes for a transport lives here too, once per engine:
+`engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a
+staging tree into Proton Drive. A mirror holds only its identity, the order of its
pipeline, and the few verbs no other mirror shares.
## The layers
@@ -266,6 +267,60 @@ so an hourly run costs one listing of upstream and none of the bucket. Two thing
What a mirror cannot afford to lose is the bucket. Everything else, the state file and the
staging tree included, is derived from it and from upstream.
+### The proton engine
+
+`engines/proton.yml` moves a staging tree into one Proton Drive folder, through the
+official `proton-drive` CLI, for a mirror whose upstream fits in a run: the mirror fills
+`staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket
+but the CLI session, because the CLI skips a file whose content Proton already holds and
+`-f create-new-revision` makes a revision of one that changed; Proton's version history
+is the history of the mirror.
+
+| Verb | Does |
+|---|---|
+| `pipeline`, `plan-pipeline` | `clock`, `session`, `destination`, `stage`, `upload`, `confirm`, `prune`, `report`, `ping`; and the read-only half, through `stage` |
+| `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` |
+| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit |
+| `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` |
+| `stage` | Hook. The mirror fills `staging/` with what Proton should hold |
+| `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` |
+| `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` |
+| `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` |
+| `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash |
+| `session-seal --
` | Host side: a laptop login's two files, encrypted into the bucket |
+| `empty-trash` | Host side, asks first: everything in Proton's trash, permanently |
+| `report-engine` | Hook. The engine's rows of the run summary |
+
+The engine reads no root var. Its inputs are the environment, by the names every Proton
+mirror's `op.env` carries: `MIRROR_PROTON_DESTINATION` and its `_UID`, `MIRROR_R2_BUCKET`,
+`MIRROR_AGE_IDENTITY`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and
+`AWS_ENDPOINT_URL_S3`. The session is the one thing to set up by hand: the CLI can only be
+seeded by a browser sign-in, so log in once on a laptop into a directory inside the repo
+and run `task session-seal -- `. Two mirrors never share a session; its
+refresh token rotates on every call, and the loser of a race needs a fresh login.
+
+A mirror of every repository under two GitHub owners, shaped like github:
+
+```yaml
+version: '3'
+vars:
+ OWNERS: jshvn katoptra
+includes:
+ toolbox:
+ taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml
+ flatten: true
+ excludes: [report-engine, report-mirror]
+ vars: {NAME: github, DESC: a nightly mirror of every repository under jshvn and katoptra into Proton Drive, IMAGE: ghcr.io/katoptra/toolbox:proton-v2}
+ proton:
+ taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml
+ flatten: true
+ excludes: [stage, prune]
+tasks:
+ stage: {cmds: ['# list the repositories, clone each as a mirror, bundle it under {{.STAGING}}//']}
+ prune: {cmds: ['# list-folder each owner in Proton; trash the bundles no repository has']}
+ report-mirror: {cmds: ['# the mirror rows']}
+```
+
## Overriding a verb
List it under `excludes:` on the include that defines it and define it in the mirror.
@@ -314,7 +369,7 @@ two matrix entries:
| Variant | Base | Tools | For |
|---|---|---|---|
| `rsync` | ubuntu 24.04 | rsync, gnupg, xz, curl, perl, go-task, AWS CLI v2 (s3, sts) | rsync upstreams to R2: ctan, tlnet, cran, cpan |
-| `proton` | python 3.13 slim | proton-drive, age, go-task, boto3, requests, pytest, ruff | Python pipelines: dropbox, photos |
+| `proton` | python 3.13 slim | proton-drive, age, git, go-task, boto3, requests, pytest, ruff, and `s3`, a boto3 get/put | Proton Drive sinks: github through the proton engine, dropbox through its own Python |
An HTTPS engine would be a third row: the same base as `rsync`, curl and the AWS CLI,
and a `list` that reads an index instead of `rsync --list-only`.
diff --git a/docker/proton.Dockerfile b/docker/proton.Dockerfile
index b2e74ec..877f95e 100644
--- a/docker/proton.Dockerfile
+++ b/docker/proton.Dockerfile
@@ -1,7 +1,8 @@
# syntax=docker/dockerfile:1.7
-# The proton toolbox: Python plus the Proton Drive CLI and age, for the mirrors whose
-# engine is a Python package. The repo is bind-mounted at /work; PYTHONPATH finds its
-# src/. Base pinned by digest; every tool from toolchain.lock.toml at the build context.
+# The proton toolbox: Python plus the Proton Drive CLI, age and git, for the mirrors whose
+# sink is Proton Drive, whether their engine is engines/proton.yml or a Python package of
+# their own. The repo is bind-mounted at /work; PYTHONPATH finds its src/. Base pinned by
+# digest; every tool from toolchain.lock.toml at the build context, git and curl from apt.
FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS fetch
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \
@@ -26,10 +27,11 @@ RUN set -eu; \
FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS toolbox
-RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \
+RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl git \
&& rm -rf /var/lib/apt/lists/*
COPY --from=fetch /usr/local/bin/proton-drive /usr/local/bin/age /usr/local/bin/age-keygen /usr/local/bin/task /usr/local/bin/
COPY toolchain.lock.toml /etc/toolchain.lock.toml
+COPY docker/s3.py /usr/local/bin/s3
RUN python - <<'PY'
import subprocess, tomllib
@@ -49,12 +51,17 @@ first = lambda argv: subprocess.check_output(argv, text=True).splitlines()[0]
assert "@" + lock["proton_drive_cli"]["version"] in first(["proton-drive", "version"])
assert lock["age"]["version"] in first(["age", "--version"])
assert lock["task"]["version"] in subprocess.check_output(["task", "--version"], text=True)
+assert first(["git", "--version"]).startswith("git version 2.")
+assert subprocess.run(["s3"], capture_output=True).returncode == 2 # usage; boto3 imports
PY
# Inside a run there is no network for Taskfiles: the mirror's .task/remote cache rides
-# in with the bind mount. R2 has one region; the value is a literal, not a secret.
+# in with the bind mount. R2 has one region and rejects the SDK's default checksum
+# headers; both values are literals, not secrets.
ENV PYTHONUNBUFFERED=1 \
PYTHONPATH=/work/src \
TASK_REMOTE_OFFLINE=1 \
- AWS_REGION=auto
+ AWS_REGION=auto \
+ AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
+ AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
WORKDIR /work
diff --git a/docker/s3.py b/docker/s3.py
new file mode 100755
index 0000000..af2176c
--- /dev/null
+++ b/docker/s3.py
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+"""s3 get KEY FILE | s3 put FILE KEY: one object of the bucket MIRROR_R2_BUCKET names.
+
+The proton image's S3 client, for the mirrors whose engine keeps a Proton session in the
+bucket. Credentials and the endpoint come from the environment as boto3 reads them:
+AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL_S3. `get` exits 3 when the
+key is absent, so a caller can tell a missing object from a failed read. A failure prints
+its class alone: the message may carry the endpoint, and the logs are public.
+"""
+
+import os
+import sys
+
+import boto3
+from botocore.config import Config
+from botocore.exceptions import BotoCoreError, ClientError
+
+MISSING = {"404", "NoSuchKey"}
+
+
+def main(argv: list[str]) -> int:
+ if len(argv) != 4 or argv[1] not in {"get", "put"}:
+ print(__doc__.strip(), file=sys.stderr)
+ return 2
+ names = ("MIRROR_R2_BUCKET", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_ENDPOINT_URL_S3")
+ unset = [n for n in names if not os.environ.get(n)]
+ if unset:
+ print(f"s3: unset in the environment: {' '.join(unset)}", file=sys.stderr)
+ return 2
+ bucket = os.environ["MIRROR_R2_BUCKET"]
+ client = boto3.client("s3", config=Config(retries={"max_attempts": 5, "mode": "standard"}))
+ try:
+ if argv[1] == "get":
+ client.download_file(bucket, argv[2], argv[3])
+ else:
+ client.upload_file(argv[2], bucket, argv[3])
+ except ClientError as exc:
+ if argv[1] == "get" and exc.response.get("Error", {}).get("Code") in MISSING:
+ return 3
+ print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr)
+ return 1
+ except BotoCoreError as exc:
+ print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr)
+ return 1
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv))
diff --git a/engines/proton.yml b/engines/proton.yml
new file mode 100644
index 0000000..16de42a
--- /dev/null
+++ b/engines/proton.yml
@@ -0,0 +1,240 @@
+# The proton engine: how bytes move from a staging tree into one Proton Drive folder, for
+# every mirror that includes this file flattened beside the toolbox. A run restores the
+# Proton CLI session from the bucket, checks that the destination is the folder the vault
+# names, lets the mirror fill the staging tree, uploads it in one CLI call (the CLI skips a
+# file whose content Proton already holds and makes a new revision of one that changed),
+# confirms the upload from the CLI's own summary, lets the mirror prune what its upstream
+# dropped, and reports. Proton's version history is the cloud-side history, so the bucket
+# holds nothing of the mirror's but the session.
+#
+# Read from the environment, by the names every Proton mirror's vault uses:
+# MIRROR_PROTON_DESTINATION (the CLI path of the destination folder, /my-files/GitHub
+# say), MIRROR_PROTON_DESTINATION_UID (its UID, compared on every run before a write),
+# MIRROR_R2_BUCKET, MIRROR_AGE_IDENTITY, and the three AWS_* names boto3 reads, with
+# AWS_ENDPOINT_URL_S3 for the endpoint. Each is a line of op.env, so it crosses into the
+# container by name.
+#
+# Hooks: stage fills STAGING and every mirror defines it; prune trashes what upstream no
+# longer has, and a mirror that wants that defines it. A mirror lists each it defines under
+# excludes: on this include. pipeline and plan-pipeline are the vocabulary in order.
+#
+# Rules this file keeps, as the toolbox does: no vars: default for anything a mirror
+# owns, and the CLI session is written back to the bucket after every CLI call, whatever
+# the exit, because its refresh token rotates and a run that kept a rotated token to
+# itself leaves the next run unable to log in.
+version: '3'
+
+vars:
+ STAGING: '{{.ROOT_DIR}}/staging'
+ SESSION: '{{.ROOT_DIR}}/.run/session'
+ SESSION_KEY: .state/session.tar.age
+ UPLOAD: '{{.ROOT_DIR}}/.run/upload.json'
+ # The CLI with its session as plain files under RUN/session: the image has no keyring.
+ PD: PROTON_DRIVE_CACHE_DIR={{.SESSION}} PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive
+
+tasks:
+ pipeline:
+ desc: clock -> session -> destination -> stage -> upload -> confirm -> prune -> report -> ping
+ cmds:
+ - {task: clock}
+ - {task: session}
+ - {task: destination}
+ - {task: stage}
+ - {task: upload}
+ - {task: confirm}
+ - {task: prune}
+ - {task: report}
+ - {task: ping}
+
+ plan-pipeline:
+ desc: 'The read-only half: clock -> session -> destination -> stage, then what an upload would carry'
+ cmds:
+ - {task: clock}
+ - {task: session}
+ - {task: destination}
+ - {task: stage}
+ - |
+ find {{.STAGING}} -type f -printf '%s\n' | awk '{ n++; s += $1 } END { printf "plan: %d files, %.1f MB staged\n", n, s / 1e6 }'
+
+ # ---- the session ----
+ session:
+ desc: Restore the Proton CLI session from the bucket into RUN/session
+ cmds:
+ - rm -rf {{.SESSION}} && mkdir -p {{.SESSION}} && chmod 700 {{.SESSION}}
+ - {task: pull, vars: {KEY: '{{.SESSION_KEY}}', OUT: '{{.RUN}}/session.tar.age', MISSING: 'echo "no Proton session at {{.SESSION_KEY}}: log in on a laptop and run task session-seal" >&2; exit 1'}}
+ - {task: age, vars: {MODE: decrypt, IN: '{{.RUN}}/session.tar.age', OUT: '{{.RUN}}/session.tar'}}
+ # The two members the CLI needs, and nothing else the archive might hold.
+ - tar -xf {{.RUN}}/session.tar -C {{.SESSION}} auth-session.json clientUid.json && chmod 600 {{.SESSION}}/*.json
+ - rm -f {{.RUN}}/session.tar {{.RUN}}/session.tar.age
+ - sha256sum {{.SESSION}}/auth-session.json > {{.RUN}}/session.sha
+
+ session-push:
+ desc: Seal RUN/session back into the bucket when the CLI rotated its token since the last push
+ internal: true
+ status: ['sha256sum --status -c {{.RUN}}/session.sha']
+ cmds:
+ - 'echo "session: token rotated, sealing it back to the bucket"'
+ - {task: seal, vars: {DIR: '{{.SESSION}}'}}
+ - sha256sum {{.SESSION}}/auth-session.json > {{.RUN}}/session.sha
+
+ seal:
+ desc: 'The two session files of DIR, tarred and age-encrypted, to the bucket: task seal DIR=.run/pd'
+ requires: {vars: [DIR]}
+ cmds:
+ - defer: rm -f {{.RUN}}/session.tar {{.RUN}}/session.tar.age
+ - mkdir -p {{.RUN}} && tar -cf {{.RUN}}/session.tar -C {{.DIR}} auth-session.json clientUid.json
+ - {task: age, vars: {MODE: encrypt, IN: '{{.RUN}}/session.tar', OUT: '{{.RUN}}/session.tar.age'}}
+ - {task: push, vars: {FILE: '{{.RUN}}/session.tar.age', KEY: '{{.SESSION_KEY}}'}}
+
+ session-seal:
+ desc: 'Encrypt a laptop Proton CLI session into the bucket: task session-seal -- .run/pd'
+ cmds:
+ - {task: op, vars: {ARGS: 'task seal DIR={{.CLI_ARGS}}'}}
+
+ age:
+ desc: age MODE (encrypt or decrypt) IN to OUT with MIRROR_AGE_IDENTITY, which touches disk for the call alone, mode 0600
+ internal: true
+ requires: {vars: [MODE, IN, OUT]}
+ cmds:
+ - |
+ test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; }
+ k={{.RUN}}/age.key; mkdir -p {{.RUN}}; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k"
+ rc=0
+ case {{.MODE}} in
+ encrypt) age -r "$(age-keygen -y "$k")" -o {{.OUT}} {{.IN}} || rc=$? ;;
+ decrypt) age -d -i "$k" -o {{.OUT}} {{.IN}} || rc=$? ;;
+ *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;;
+ esac
+ rm -f "$k"; exit $rc
+
+ # ---- the bucket ----
+ pull:
+ desc: KEY from the bucket to OUT; MISSING runs when the bucket has no such key
+ internal: true
+ requires: {vars: [KEY, OUT, MISSING]}
+ cmds:
+ - |
+ rc=0; s3 get {{.KEY}} {{.OUT}} || rc=$?
+ case $rc in 0) ;; 3) {{.MISSING}} ;; *) exit $rc ;; esac
+
+ push:
+ desc: FILE to the bucket's KEY as one PutObject
+ internal: true
+ requires: {vars: [FILE, KEY]}
+ cmds:
+ - s3 put {{.FILE}} {{.KEY}}
+
+ # ---- the CLI ----
+ # Every CLI call goes through here: stderr to RUN/pd.err rather than the public log,
+ # the session pushed back whatever the exit, then the exit judged against OK, the codes
+ # accepted besides 0.
+ pd:
+ desc: One CLI call, ARGS, with the session written back afterwards
+ internal: true
+ requires: {vars: [ARGS]}
+ cmds:
+ - rc=0; {{.PD}} {{.ARGS}} 2> {{.RUN}}/pd.err || rc=$?; echo $rc > {{.RUN}}/pd.rc
+ - {task: session-push}
+ - |
+ rc=$(cat {{.RUN}}/pd.rc)
+ case " {{.OK | default 0}} " in *" $rc "*) exit 0 ;; esac
+ echo "proton-drive exited $rc; its stderr, $(wc -l < {{.RUN}}/pd.err) lines, is in .run/pd.err" >&2; exit $rc
+
+ destination:
+ desc: Refuse the run unless MIRROR_PROTON_DESTINATION resolves to exactly one folder whose UID is MIRROR_PROTON_DESTINATION_UID
+ cmds:
+ - defer: rm -f {{.RUN}}/parent.json
+ - {task: pd, vars: {ARGS: 'filesystem list -j "$(dirname "$MIRROR_PROTON_DESTINATION")" > {{.RUN}}/parent.json'}}
+ # ponytail: the folder is found by basename, so a destination whose own name holds a
+ # slash is unsupported; the CLI escapes one with a backslash and dirname does not.
+ - |
+ python3 - "$(basename "$MIRROR_PROTON_DESTINATION")" "$MIRROR_PROTON_DESTINATION_UID" {{.RUN}}/parent.json <<'PY'
+ import json, sys
+ name, uid, path = sys.argv[1:]
+ unwrap = lambda v: v.get("value") if isinstance(v, dict) else v
+ nodes = json.load(open(path))
+ found = [n for n in nodes if isinstance(n, dict) and unwrap(n.get("name")) == name]
+ if len(found) != 1 or unwrap(found[0].get("type")) != "folder":
+ sys.exit(f"destination: {len(found)} folder(s) of that name among {len(nodes)} entries of the parent; expected exactly one")
+ if not uid or str(unwrap(found[0].get("uid")) or "") != uid:
+ sys.exit("destination: the folder's UID is not the one the vault names")
+ print("destination: the folder and its UID match the vault")
+ PY
+
+ upload:
+ desc: STAGING into the destination in one CLI call; identical files are skipped, changed ones become new revisions; the CLI's summary goes to RUN/upload.json
+ cmds:
+ - 'test -n "$(ls -A {{.STAGING}} 2>/dev/null)" || { echo "upload: staging is empty; stage put nothing there" >&2; exit 1; }'
+ # Exit 1 is the CLI refusing some items and handling the rest; confirm reads the summary.
+ - {task: pd, vars: {ARGS: 'filesystem upload -f create-new-revision -d merge -t --json {{.STAGING}}/* "$MIRROR_PROTON_DESTINATION" > {{.UPLOAD}}', OK: '0 1'}}
+
+ confirm:
+ desc: The CLI's summary must account for every staged file and folder, with no failures; the verdict goes to RUN/confirm.txt
+ cmds:
+ - |
+ files=$(find {{.STAGING}} -type f | wc -l); folders=$(find {{.STAGING}} -mindepth 1 -type d | wc -l)
+ python3 - "$files" "$folders" {{.UPLOAD}} {{.RUN}}/confirm.txt <<'PY'
+ import json, sys
+ files, folders = int(sys.argv[1]), int(sys.argv[2])
+ # The CLI writes progress and its summary as one JSON object per line; the last
+ # line carrying transferredItems is the summary.
+ summary = None
+ for line in open(sys.argv[3]):
+ try:
+ candidate = json.loads(line)
+ except ValueError:
+ continue
+ if isinstance(candidate, dict) and "transferredItems" in candidate:
+ summary = candidate
+ if summary is None:
+ sys.exit("confirm: no upload summary in the CLI's output")
+ t, s, f = (int(summary.get(k, 0) or 0) for k in ("transferredItems", "skippedItems", "failedItems"))
+ verdict = f"{t} uploaded as new files or revisions, {s} skipped as identical, {f} failed, of {files} files in {folders} folders staged"
+ ok = f == 0 and t + s == files + folders
+ open(sys.argv[4], "w").write(verdict + ("" if ok else " (not confirmed)") + "\n")
+ print("confirm: " + verdict)
+ sys.exit(0 if ok else 1)
+ PY
+
+ list-folder:
+ desc: The JSON listing of the Proton folder FOLDER to OUT; a folder the CLI cannot list gives an empty one, so a caller prunes nothing
+ internal: true
+ requires: {vars: [FOLDER, OUT]}
+ cmds:
+ - {task: pd, vars: {ARGS: 'filesystem list -j "{{.FOLDER}}" > {{.OUT}} || echo "[]" > {{.OUT}}'}}
+
+ trash:
+ desc: Trash the Proton nodes at PATHS, CLI paths space separated; Proton's trash keeps them
+ status: ['test -z "{{.PATHS}}"']
+ cmds:
+ - {task: pd, vars: {ARGS: 'filesystem trash {{.PATHS}}'}}
+
+ empty-trash:
+ desc: Permanently delete everything in Proton Drive's trash, inside the toolbox; asks first, never scheduled
+ prompt: This permanently deletes everything in Proton Drive trash. Continue?
+ cmds:
+ - {task: op, vars: {ARGS: task empty-trash-pipeline}}
+
+ empty-trash-pipeline:
+ internal: true
+ cmds:
+ - {task: clock}
+ - {task: session}
+ - {task: pd, vars: {ARGS: filesystem empty-trash}}
+
+ # ---- hooks ----
+ stage: {cmds: []} # the mirror's: fill STAGING with what Proton should hold
+ prune: {cmds: []} # the mirror's: trash what its upstream no longer has
+
+ report-engine:
+ desc: The engine's rows of the run summary, appended after the toolbox's
+ silent: true
+ cmds:
+ - |
+ staged=$(find {{.STAGING}} -type f 2>/dev/null | wc -l)
+ mb=$(find {{.STAGING}} -type f -printf '%s\n' 2>/dev/null | awk '{ s += $1 } END { printf "%.1f", s / 1e6 }')
+ cat >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" </dev/null || echo "not confirmed this run") |
+ | Session | $(test -f {{.RUN}}/session.sha && echo "restored, and sealed back on rotation" || echo "not restored") |
+ EOF
diff --git a/examples/proton/Taskfile.yml b/examples/proton/Taskfile.yml
index ed0c08e..02341b8 100644
--- a/examples/proton/Taskfile.yml
+++ b/examples/proton/Taskfile.yml
@@ -1,34 +1,62 @@
-# A consumer of the toolbox with the proton image. No engine, so the mirror defines the
-# two pipeline verbs itself, from the toolbox's clock, report and ping and its own steps.
+# A consumer of the toolbox and the proton engine, with the proton image. Inside this repo
+# the includes are paths; a mirror includes
+# https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml and
+# https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml. The pipeline is
+# the engine's; render.txt, every command of it, is the library's own check of the engine,
+# and `task run -- task offline` runs the verbs that need neither Proton nor a bucket.
version: '3'
includes:
toolbox:
taskfile: ../../toolbox.yml
flatten: true
+ excludes: [report-engine] # the engine's
vars:
NAME: example-proton
- DESC: the proton toolbox, exercised
+ DESC: the proton toolbox and engine, exercised
IMAGE: ghcr.io/katoptra/toolbox:proton-dev
LIB_DIR: ../..
+ MENU: '"" "${c}engine -- offline, inside the image${r}" " task run -- task offline confirm over fixtures/, and an age round trip"'
+ proton:
+ taskfile: ../../engines/proton.yml
+ flatten: true
+ excludes: [stage]
tasks:
- pipeline:
- desc: clock -> tools -> report -> ping
- cmds:
- - {task: clock}
- - {task: tools}
- - {task: report}
- - {task: ping}
- plan-pipeline:
- desc: 'The read-only half: clock -> tools'
+ stage:
+ desc: One file in one folder under STAGING, so the pipeline has something to upload
cmds:
- - {task: clock}
- - {task: tools}
+ - rm -rf {{.STAGING}} && mkdir -p {{.STAGING}}/example && printf 'hello\n' > {{.STAGING}}/example/hello.txt
tools:
desc: Every tool the proton image promises, by version
cmds:
- python --version
- proton-drive version | head -1
- age --version
+ - git --version
- python -c 'import boto3, requests; print(boto3.__version__, requests.__version__)'
- ruff --version
- pytest --version
+ - s3 2>&1 | head -1
+ offline:
+ desc: The engine's verbs that need neither Proton nor a bucket, inside the image
+ cmds:
+ # confirm: one file in one folder staged; a summary that accounts for both with no
+ # failure confirms, one naming a failure does not.
+ - {task: stage}
+ - cp {{.ROOT_DIR}}/fixtures/upload-ok.json {{.UPLOAD}}
+ - {task: confirm}
+ - grep -q 'skipped as identical, 0 failed' {{.RUN}}/confirm.txt
+ - cp {{.ROOT_DIR}}/fixtures/upload-failed.json {{.UPLOAD}}
+ - if task confirm >/dev/null 2>&1; then echo "confirm accepted a summary with a failure" >&2; exit 1; fi
+ - grep -q '(not confirmed)' {{.RUN}}/confirm.txt
+ # age: a throwaway identity round-trips a file through the engine's verb, and the
+ # identity file is gone afterwards.
+ - MIRROR_AGE_IDENTITY="$(age-keygen 2>/dev/null)" task age-roundtrip
+ - test ! -e {{.RUN}}/age.key
+ - 'echo "offline: every check passed"'
+ age-roundtrip:
+ desc: encrypt then decrypt RUN/plain.txt through the engine's age verb, and compare
+ cmds:
+ - printf 'round trip\n' > {{.RUN}}/plain.txt
+ - {task: age, vars: {MODE: encrypt, IN: '{{.RUN}}/plain.txt', OUT: '{{.RUN}}/plain.age'}}
+ - {task: age, vars: {MODE: decrypt, IN: '{{.RUN}}/plain.age', OUT: '{{.RUN}}/plain.out'}}
+ - cmp {{.RUN}}/plain.txt {{.RUN}}/plain.out
diff --git a/examples/proton/fixtures/upload-failed.json b/examples/proton/fixtures/upload-failed.json
new file mode 100644
index 0000000..26902ec
--- /dev/null
+++ b/examples/proton/fixtures/upload-failed.json
@@ -0,0 +1,2 @@
+{"type":"progress","name":"hello.txt","transferred":6}
+{"transferredItems":1,"skippedItems":0,"failedItems":1,"failures":[{"name":"hello.txt","error":"upload failed"}]}
diff --git a/examples/proton/fixtures/upload-ok.json b/examples/proton/fixtures/upload-ok.json
new file mode 100644
index 0000000..9ea615c
--- /dev/null
+++ b/examples/proton/fixtures/upload-ok.json
@@ -0,0 +1,2 @@
+{"type":"progress","name":"hello.txt","transferred":6}
+{"transferredItems":1,"skippedItems":1,"failedItems":0,"failures":[]}
diff --git a/examples/proton/op.env b/examples/proton/op.env
index 77f81ab..119e979 100644
--- a/examples/proton/op.env
+++ b/examples/proton/op.env
@@ -1,5 +1,9 @@
# op:// references only; `op run --env-file=op.env` resolves them at run time.
+MIRROR_PROTON_DESTINATION=op://VAULT/proton/destination
+MIRROR_PROTON_DESTINATION_UID=op://VAULT/proton/destination_uid
AWS_ACCESS_KEY_ID=op://VAULT/r2/access_key_id
AWS_SECRET_ACCESS_KEY=op://VAULT/r2/secret_access_key
-AWS_ENDPOINT_URL=op://VAULT/r2/endpoint
+AWS_ENDPOINT_URL_S3=op://VAULT/r2/endpoint
+MIRROR_R2_BUCKET=op://VAULT/r2/bucket
+MIRROR_AGE_IDENTITY=op://VAULT/age/identity
HEALTHCHECK_URL=op://VAULT/healthcheck/url
diff --git a/examples/proton/render.txt b/examples/proton/render.txt
index 6499d89..58dfe47 100644
--- a/examples/proton/render.txt
+++ b/examples/proton/render.txt
@@ -1,8 +1,99 @@
task: [clock] mkdir -p /work/examples/proton/.run && rm -f /work/examples/proton/.run/chain && date -u '+%s %H %u' > /work/examples/proton/.run/start.txt
-task: [tools] python --version
-task: [tools] proton-drive version | head -1
-task: [tools] age --version
-task: [tools] python -c 'import boto3, requests; print(boto3.__version__, requests.__version__)'
-task: [tools] ruff --version
-task: [tools] pytest --version
+task: [session] rm -rf /work/examples/proton/.run/session && mkdir -p /work/examples/proton/.run/session && chmod 700 /work/examples/proton/.run/session
+task: [pull] rc=0; s3 get .state/session.tar.age /work/examples/proton/.run/session.tar.age || rc=$?
+case $rc in 0) ;; 3) echo "no Proton session at .state/session.tar.age: log in on a laptop and run task session-seal" >&2; exit 1 ;; *) exit $rc ;; esac
+
+task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; }
+k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k"
+rc=0
+case decrypt in
+ encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age || rc=$? ;;
+ decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age || rc=$? ;;
+ *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;;
+esac
+rm -f "$k"; exit $rc
+
+task: [session] tar -xf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json && chmod 600 /work/examples/proton/.run/session/*.json
+task: [session] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age
+task: [session] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha
+task: [pd] rc=0; PROTON_DRIVE_CACHE_DIR=/work/examples/proton/.run/session PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem list -j "$(dirname "$MIRROR_PROTON_DESTINATION")" > /work/examples/proton/.run/parent.json 2> /work/examples/proton/.run/pd.err || rc=$?; echo $rc > /work/examples/proton/.run/pd.rc
+task: [session-push] echo "session: token rotated, sealing it back to the bucket"
+task: [seal] mkdir -p /work/examples/proton/.run && tar -cf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json
+task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; }
+k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k"
+rc=0
+case encrypt in
+ encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;;
+ decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;;
+ *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;;
+esac
+rm -f "$k"; exit $rc
+
+task: [push] s3 put /work/examples/proton/.run/session.tar.age .state/session.tar.age
+task: [seal] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age
+task: [session-push] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha
+task: [pd] rc=$(cat /work/examples/proton/.run/pd.rc)
+case " 0 " in *" $rc "*) exit 0 ;; esac
+echo "proton-drive exited $rc; its stderr, $(wc -l < /work/examples/proton/.run/pd.err) lines, is in .run/pd.err" >&2; exit $rc
+
+task: [destination] python3 - "$(basename "$MIRROR_PROTON_DESTINATION")" "$MIRROR_PROTON_DESTINATION_UID" /work/examples/proton/.run/parent.json <<'PY'
+import json, sys
+name, uid, path = sys.argv[1:]
+unwrap = lambda v: v.get("value") if isinstance(v, dict) else v
+nodes = json.load(open(path))
+found = [n for n in nodes if isinstance(n, dict) and unwrap(n.get("name")) == name]
+if len(found) != 1 or unwrap(found[0].get("type")) != "folder":
+ sys.exit(f"destination: {len(found)} folder(s) of that name among {len(nodes)} entries of the parent; expected exactly one")
+if not uid or str(unwrap(found[0].get("uid")) or "") != uid:
+ sys.exit("destination: the folder's UID is not the one the vault names")
+print("destination: the folder and its UID match the vault")
+PY
+
+task: [destination] rm -f /work/examples/proton/.run/parent.json
+task: [stage] rm -rf /work/examples/proton/staging && mkdir -p /work/examples/proton/staging/example && printf 'hello\n' > /work/examples/proton/staging/example/hello.txt
+task: [upload] test -n "$(ls -A /work/examples/proton/staging 2>/dev/null)" || { echo "upload: staging is empty; stage put nothing there" >&2; exit 1; }
+task: [pd] rc=0; PROTON_DRIVE_CACHE_DIR=/work/examples/proton/.run/session PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem upload -f create-new-revision -d merge -t --json /work/examples/proton/staging/* "$MIRROR_PROTON_DESTINATION" > /work/examples/proton/.run/upload.json 2> /work/examples/proton/.run/pd.err || rc=$?; echo $rc > /work/examples/proton/.run/pd.rc
+task: [session-push] echo "session: token rotated, sealing it back to the bucket"
+task: [seal] mkdir -p /work/examples/proton/.run && tar -cf /work/examples/proton/.run/session.tar -C /work/examples/proton/.run/session auth-session.json clientUid.json
+task: [age] test -n "$MIRROR_AGE_IDENTITY" || { echo "MIRROR_AGE_IDENTITY is unset" >&2; exit 1; }
+k=/work/examples/proton/.run/age.key; mkdir -p /work/examples/proton/.run; install -m 600 /dev/null "$k" && printf '%s\n' "$MIRROR_AGE_IDENTITY" > "$k"
+rc=0
+case encrypt in
+ encrypt) age -r "$(age-keygen -y "$k")" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;;
+ decrypt) age -d -i "$k" -o /work/examples/proton/.run/session.tar.age /work/examples/proton/.run/session.tar || rc=$? ;;
+ *) echo "age: MODE must be encrypt or decrypt" >&2; rc=2 ;;
+esac
+rm -f "$k"; exit $rc
+
+task: [push] s3 put /work/examples/proton/.run/session.tar.age .state/session.tar.age
+task: [seal] rm -f /work/examples/proton/.run/session.tar /work/examples/proton/.run/session.tar.age
+task: [session-push] sha256sum /work/examples/proton/.run/session/auth-session.json > /work/examples/proton/.run/session.sha
+task: [pd] rc=$(cat /work/examples/proton/.run/pd.rc)
+case " 0 1 " in *" $rc "*) exit 0 ;; esac
+echo "proton-drive exited $rc; its stderr, $(wc -l < /work/examples/proton/.run/pd.err) lines, is in .run/pd.err" >&2; exit $rc
+
+task: [confirm] files=$(find /work/examples/proton/staging -type f | wc -l); folders=$(find /work/examples/proton/staging -mindepth 1 -type d | wc -l)
+python3 - "$files" "$folders" /work/examples/proton/.run/upload.json /work/examples/proton/.run/confirm.txt <<'PY'
+import json, sys
+files, folders = int(sys.argv[1]), int(sys.argv[2])
+# The CLI writes progress and its summary as one JSON object per line; the last
+# line carrying transferredItems is the summary.
+summary = None
+for line in open(sys.argv[3]):
+ try:
+ candidate = json.loads(line)
+ except ValueError:
+ continue
+ if isinstance(candidate, dict) and "transferredItems" in candidate:
+ summary = candidate
+if summary is None:
+ sys.exit("confirm: no upload summary in the CLI's output")
+t, s, f = (int(summary.get(k, 0) or 0) for k in ("transferredItems", "skippedItems", "failedItems"))
+verdict = f"{t} uploaded as new files or revisions, {s} skipped as identical, {f} failed, of {files} files in {folders} folders staged"
+ok = f == 0 and t + s == files + folders
+open(sys.argv[4], "w").write(verdict + ("" if ok else " (not confirmed)") + "\n")
+print("confirm: " + verdict)
+sys.exit(0 if ok else 1)
+PY
+
task: [ping] test -z "$HEALTHCHECK_URL" || curl -fsS -m 10 --retry 3 -o /dev/null "$HEALTHCHECK_URL"
diff --git a/toolbox.yml b/toolbox.yml
index 6a63b3b..e1a03c1 100644
--- a/toolbox.yml
+++ b/toolbox.yml
@@ -8,11 +8,14 @@
# Rules this file keeps: no vars: default for anything a mirror owns (it would shadow
# the mirror's value; defaults go inline), and host-side and in-container verbs share
# one namespace, so the names below are reserved, and so is the engine vocabulary a
-# second include adds beside them: pipeline, plan-pipeline, list, normalise, state,
-# pull, push, rebuild, diff, split, batches, batch, fetch, publish, merge, checkpoint,
-# remove, delete, reconcile, retry, and the hooks prepare, verify, index, smoke,
-# smoke-mirror and report-engine. report-engine lives here as a no-op and in every
-# engine, so an engine consumer's toolbox include lists it under excludes:.
+# second include adds beside them. The rsync engine's: pipeline, plan-pipeline, list,
+# normalise, state, pull, push, rebuild, diff, split, batches, batch, fetch, publish,
+# merge, checkpoint, remove, delete, reconcile, retry, and the hooks prepare, verify,
+# index, smoke and smoke-mirror. The proton engine's: session, session-push, seal,
+# session-seal, age, pull, push, pd, destination, upload, confirm, list-folder, trash,
+# empty-trash, empty-trash-pipeline, and the hooks stage and prune. report-engine lives
+# here as a no-op and in every engine, so an engine consumer's toolbox include lists it
+# under excludes:.
version: '3'
vars: