feat(engine): the proton engine, and git in the proton image

.github/workflows/ci.yml

@@ -30,6 +30,5 @@ jobs:
         working-directory: examples/${{ matrix.variant }}
         run: task check
       - name: The engine's verbs that need no bucket, over the fixtures
-        if: matrix.variant == 'rsync'
         working-directory: examples/${{ matrix.variant }}
         run: task run -- task offline

CLAUDE.md

@@ -10,10 +10,11 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont
   mirror's root value.
 - Verb names are reserved across host and container. `plan` is the host-side read-only
   run; an engine's batch planner is `split`. The hooks, `prepare`, `verify`, `index`,
-  `smoke`, `smoke-mirror`, `report-engine` in the engine and `report-mirror` in the
-  toolbox, are the only verbs a mirror redefines, each excluded on the include that
-  defines it; `pipeline` and `plan-pipeline` come from the engine and a mirror excludes
-  them only for another order. `report-engine` exists in both files, so an engine
+  `smoke`, `smoke-mirror`, `report-engine` in the rsync engine, `stage` and `prune` in
+  the proton engine, and `report-mirror` in the toolbox, are the only verbs a mirror
+  redefines, each excluded on the include that defines it; `pipeline` and
+  `plan-pipeline` come from the engine and a mirror excludes them only for another
+  order. `report-engine` exists in both files, so an engine
   consumer excludes it on the toolbox include. Extension is a hook, never a copy: a
   verb that needs more than the engine does gets a `-mirror` hook the engine calls last.
 - A mirror's root var shadows a command-line `KEY=value` inside an included verb, so an
@@ -48,6 +49,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont
   so the container engine's own progress lines never reach `render.txt`.
 - Inside a `sh:` var, `printf -- '-e %s'` prints dashes: task's built-in shell takes
   the `--` as the format. Use `printf '%s %s ' -e "$v"`.
+- Task's built-in shell has no `umask`. A file that must be born 0600 is
+  `install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does.
+- The proton engine keeps no state of the mirror's in the bucket, only the CLI session,
+  because `filesystem upload` skips a file whose content Proton already holds and
+  `-f create-new-revision` handles one that changed. Every CLI call goes through `pd`,
+  which pushes the session back whatever the exit: the refresh token rotates, and a run
+  that kept a rotated token to itself leaves the next run unable to log in. Two mirrors
+  never share one session for the same reason.
 - Actions pinned to a full SHA with the version in a trailing comment. A mirror pins the
   two reusable workflows that way; each checks this repository out at its own commit
   (`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is
@@ -58,11 +67,14 @@ The toolbox every katoptra mirror includes by URL. Read `README.md` for the cont
 
 ```sh
 cd examples/rsync  && task image-build && task run -- task tools && task check && task run -- task offline
-cd examples/proton && task image-build && task run -- task tools && task check
+cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline
 ```
 
 A verb change updates the `render.txt` files via `task render-update`; `offline` is
-the engine's own check over `examples/rsync/fixtures/`: the list diff over `run-root`
-and `run-empty`, `retry`'s exit codes, and `prepare` and `verify` over `tree/`, a signed
-subtree whose tlpdb is signed by a throwaway key pinned in the example. Regenerate the
-tree with a new key only to change its shape; the private half was never kept.
+each engine's own check. The rsync one runs over `examples/rsync/fixtures/`: the list
+diff over `run-root` and `run-empty`, `retry`'s exit codes, and `prepare` and `verify`
+over `tree/`, a signed subtree whose tlpdb is signed by a throwaway key pinned in the
+example. Regenerate the tree with a new key only to change its shape; the private half
+was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an
+accepting and a refusing upload summary, and the `age` verb round trip with a throwaway
+identity.

README.md

@@ -5,8 +5,9 @@
 
 The toolbox every katoptra mirror includes by URL. The rule it enforces: the code that
 starts a run, contains it, resolves its secrets, checks it and reports it lives here,
-once. The code that moves bytes for a transport lives here too, once per engine, and
-`engines/rsync.yml` is the first. A mirror holds only its identity, the order of its
+once. The code that moves bytes for a transport lives here too, once per engine:
+`engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a
+staging tree into Proton Drive. A mirror holds only its identity, the order of its
 pipeline, and the few verbs no other mirror shares.
 
 ## The layers
@@ -266,6 +267,60 @@ so an hourly run costs one listing of upstream and none of the bucket. Two thing
 What a mirror cannot afford to lose is the bucket. Everything else, the state file and the
 staging tree included, is derived from it and from upstream.
 
+### The proton engine
+
+`engines/proton.yml` moves a staging tree into one Proton Drive folder, through the
+official `proton-drive` CLI, for a mirror whose upstream fits in a run: the mirror fills
+`staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket
+but the CLI session, because the CLI skips a file whose content Proton already holds and
+`-f create-new-revision` makes a revision of one that changed; Proton's version history
+is the history of the mirror.
+
+| Verb | Does |
+|---|---|
+| `pipeline`, `plan-pipeline` | `clock`, `session`, `destination`, `stage`, `upload`, `confirm`, `prune`, `report`, `ping`; and the read-only half, through `stage` |
+| `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` |
+| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit |
+| `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` |
+| `stage` | Hook. The mirror fills `staging/` with what Proton should hold |
+| `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` |
+| `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` |
+| `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` |
+| `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash |
+| `session-seal -- <dir>` | Host side: a laptop login's two files, encrypted into the bucket |
+| `empty-trash` | Host side, asks first: everything in Proton's trash, permanently |
+| `report-engine` | Hook. The engine's rows of the run summary |
+
+The engine reads no root var. Its inputs are the environment, by the names every Proton
+mirror's `op.env` carries: `MIRROR_PROTON_DESTINATION` and its `_UID`, `MIRROR_R2_BUCKET`,
+`MIRROR_AGE_IDENTITY`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and
+`AWS_ENDPOINT_URL_S3`. The session is the one thing to set up by hand: the CLI can only be
+seeded by a browser sign-in, so log in once on a laptop into a directory inside the repo
+and run `task session-seal -- <that directory>`. Two mirrors never share a session; its
+refresh token rotates on every call, and the loser of a race needs a fresh login.
+
+A mirror of every repository under two GitHub owners, shaped like github:
+
+```yaml
+version: '3'
+vars:
+  OWNERS: jshvn katoptra
+includes:
+  toolbox:
+    taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/toolbox.yml
+    flatten: true
+    excludes: [report-engine, report-mirror]
+    vars: {NAME: github, DESC: a nightly mirror of every repository under jshvn and katoptra into Proton Drive, IMAGE: ghcr.io/katoptra/toolbox:proton-v2}
+  proton:
+    taskfile: https://raw.githubusercontent.com/katoptra/lib/v2/engines/proton.yml
+    flatten: true
+    excludes: [stage, prune]
+tasks:
+  stage: {cmds: ['# list the repositories, clone each as a mirror, bundle it under {{.STAGING}}/<owner>/']}
+  prune: {cmds: ['# list-folder each owner in Proton; trash the bundles no repository has']}
+  report-mirror: {cmds: ['# the mirror rows']}
+```
+
 ## Overriding a verb
 
 List it under `excludes:` on the include that defines it and define it in the mirror.
@@ -314,7 +369,7 @@ two matrix entries:
 | Variant | Base | Tools | For |
 |---|---|---|---|
 | `rsync` | ubuntu 24.04 | rsync, gnupg, xz, curl, perl, go-task, AWS CLI v2 (s3, sts) | rsync upstreams to R2: ctan, tlnet, cran, cpan |
-| `proton` | python 3.13 slim | proton-drive, age, go-task, boto3, requests, pytest, ruff | Python pipelines: dropbox, photos |
+| `proton` | python 3.13 slim | proton-drive, age, git, go-task, boto3, requests, pytest, ruff, and `s3`, a boto3 get/put | Proton Drive sinks: github through the proton engine, dropbox through its own Python |
 
 An HTTPS engine would be a third row: the same base as `rsync`, curl and the AWS CLI,
 and a `list` that reads an index instead of `rsync --list-only`.

docker/proton.Dockerfile

@@ -1,7 +1,8 @@
 # syntax=docker/dockerfile:1.7
-# The proton toolbox: Python plus the Proton Drive CLI and age, for the mirrors whose
-# engine is a Python package. The repo is bind-mounted at /work; PYTHONPATH finds its
-# src/. Base pinned by digest; every tool from toolchain.lock.toml at the build context.
+# The proton toolbox: Python plus the Proton Drive CLI, age and git, for the mirrors whose
+# sink is Proton Drive, whether their engine is engines/proton.yml or a Python package of
+# their own. The repo is bind-mounted at /work; PYTHONPATH finds its src/. Base pinned by
+# digest; every tool from toolchain.lock.toml at the build context, git and curl from apt.
 FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS fetch
 
 RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \
@@ -26,10 +27,11 @@ RUN set -eu; \
 
 FROM python:3.13.15-slim-bookworm@sha256:ed86c82274b3c69b52fb5820f358f0bd7df0b603332063cb5c6e32bd220c3e6e AS toolbox
 
-RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl \
+RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl git \
  && rm -rf /var/lib/apt/lists/*
 COPY --from=fetch /usr/local/bin/proton-drive /usr/local/bin/age /usr/local/bin/age-keygen /usr/local/bin/task /usr/local/bin/
 COPY toolchain.lock.toml /etc/toolchain.lock.toml
+COPY docker/s3.py /usr/local/bin/s3
 
 RUN python - <<'PY'
 import subprocess, tomllib
@@ -49,12 +51,17 @@ first = lambda argv: subprocess.check_output(argv, text=True).splitlines()[0]
 assert "@" + lock["proton_drive_cli"]["version"] in first(["proton-drive", "version"])
 assert lock["age"]["version"] in first(["age", "--version"])
 assert lock["task"]["version"] in subprocess.check_output(["task", "--version"], text=True)
+assert first(["git", "--version"]).startswith("git version 2.")
+assert subprocess.run(["s3"], capture_output=True).returncode == 2   # usage; boto3 imports
 PY
 
 # Inside a run there is no network for Taskfiles: the mirror's .task/remote cache rides
-# in with the bind mount. R2 has one region; the value is a literal, not a secret.
+# in with the bind mount. R2 has one region and rejects the SDK's default checksum
+# headers; both values are literals, not secrets.
 ENV PYTHONUNBUFFERED=1 \
     PYTHONPATH=/work/src \
     TASK_REMOTE_OFFLINE=1 \
-    AWS_REGION=auto
+    AWS_REGION=auto \
+    AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
+    AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
 WORKDIR /work

docker/s3.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+"""s3 get KEY FILE | s3 put FILE KEY: one object of the bucket MIRROR_R2_BUCKET names.
+
+The proton image's S3 client, for the mirrors whose engine keeps a Proton session in the
+bucket. Credentials and the endpoint come from the environment as boto3 reads them:
+AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL_S3. `get` exits 3 when the
+key is absent, so a caller can tell a missing object from a failed read. A failure prints
+its class alone: the message may carry the endpoint, and the logs are public.
+"""
+
+import os
+import sys
+
+import boto3
+from botocore.config import Config
+from botocore.exceptions import BotoCoreError, ClientError
+
+MISSING = {"404", "NoSuchKey"}
+
+
+def main(argv: list[str]) -> int:
+    if len(argv) != 4 or argv[1] not in {"get", "put"}:
+        print(__doc__.strip(), file=sys.stderr)
+        return 2
+    names = ("MIRROR_R2_BUCKET", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_ENDPOINT_URL_S3")
+    unset = [n for n in names if not os.environ.get(n)]
+    if unset:
+        print(f"s3: unset in the environment: {' '.join(unset)}", file=sys.stderr)
+        return 2
+    bucket = os.environ["MIRROR_R2_BUCKET"]
+    client = boto3.client("s3", config=Config(retries={"max_attempts": 5, "mode": "standard"}))
+    try:
+        if argv[1] == "get":
+            client.download_file(bucket, argv[2], argv[3])
+        else:
+            client.upload_file(argv[2], bucket, argv[3])
+    except ClientError as exc:
+        if argv[1] == "get" and exc.response.get("Error", {}).get("Code") in MISSING:
+            return 3
+        print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr)
+        return 1
+    except BotoCoreError as exc:
+        print(f"s3 {argv[1]} failed: {type(exc).__name__}", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))