From 71be0d66b5b2c7e6e9a88459ad92342237ec4d44 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 08:58:15 -0700 Subject: [PATCH 01/14] Record the production deploy, and what the off-host audit found The State table and the robots.txt entry both said production had not been redeployed. It has: release 20260808-154717, from main, by pipeline, with the 1,245-URL contract verified against the live site. Read from the served bytes rather than the run's exit code: /robots.txt answers 200 advertising the .net sitemap, /robots.txt/ 301s to it, sitemap.xml is 312 .net and zero .com, and the gallery fix is live. Three things learned since about the off-host copy, all recorded where the log review will read them rather than left in a channel file. The stale misnamed archive is deleted and the count reads 58 again. The general case has a mechanism now: the host keeps an append-only RECONCILE.md inside the archive directory, so the pull carries it with the data it explains instead of depending on a note nobody rereads. A copy predates the timer. Directory mtimes on the backup host are copy times where file mtimes are the source's, and they show a pull at 03:31 UTC that the service journal never recorded, because the script was run directly rather than through systemd. A journal with one entry is not evidence of one copy. Nothing else had diverged, audited both ways: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no --delete. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 4 +++- TODO.md | 10 ++++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index 9e87c85..d80eab5 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -239,7 +239,9 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access **The off-host copy of the access log exists, and the schedule that maintains it is younger than the copy.** The pull to the backup host is installed as a `systemd` timer running daily at 09:00 UTC, chosen to sit behind both producers on the VPS rather than beside them, and its first copy was made by hand rather than by the timer. Read the unit and its last run on the backup host rather than trusting this paragraph, for the same reason retention is read from the VPS: a claim about a schedule is only worth what the machine says. -**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it**, and treat a rename on the VPS as something the channel has to carry, because no transfer will. +**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it.** The reconciliation itself now travels with the data: the VPS keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it automatically and a rename does not depend on someone rereading a channel file. It records what a file contained rather than what it was called, and it is counted among the pulled log files, so expect the file count to exceed the number of logs by one. + +**A journal with one entry is not evidence of one copy.** The pull can be run directly as well as by its timer, and a direct run writes no service record. Directory mtimes on the backup host are the copy times, where the file mtimes are the VPS's, so those are what to read when establishing when something arrived. ## Who Owns What diff --git a/TODO.md b/TODO.md index 6795791..7922b27 100644 --- a/TODO.md +++ b/TODO.md @@ -18,7 +18,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical | | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | -| VPS production | **M7a done 2026-08-08.** Serving release `20260808-041050` at `blog.insanegenius.net`, answering `200` unauthenticated, verified 9/9 from the host side with the built `baseURL` read from the deployed bytes. DNS for the public name is still on the old platform | +| VPS production | **M7a done 2026-08-08.** Serving release `20260808-154717` at `blog.insanegenius.net`, answering `200` unauthenticated, deployed from `main` by pipeline with the 1,245-URL contract verified against the live site. `/robots.txt` answers 200 carrying a `.net` sitemap line, and the gallery fix is live. DNS for the public name is still on the old platform | | Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has not run at all | ## Blocked on the maintainer @@ -33,8 +33,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. -- **`robots.txt` is decided and built, 2026-08-08, and what remains is that production has not been redeployed since.** The site emits one now, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. The record below is kept because the reasoning is what the next decision about crawl directives will need, and because production still answers 404 until a deploy carries this. - - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set, so this describes the release production is still serving rather than the current build. +- **`robots.txt` is decided, built, and deployed, 2026-08-08.** The site emits one, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. Verified from the served bytes on release `20260808-154717`: `/robots.txt` answers 200 advertising `https://blog.insanegenius.net/sitemap.xml`, `/robots.txt/` 301s to it, and `sitemap.xml` carries 312 `.net` URLs and zero `.com`. The record below is kept because the reasoning is what the next decision about crawl directives will need. + - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set and a deploy has carried it, so this describes the state up to release `20260808-041050` rather than what is served today. - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Because this site emitted none, M7b would not have been a return to a previous state, it would have been a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer would have gone with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. @@ -52,7 +52,9 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Review the logs for non-200s**, daily for the first week after cutover, then monthly. Real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor, and a staging probe for `/wp-login.php` answered by the auth gate rather than by the site is the shape of what Caddy never sees. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". - **Pull the log off the VPS, on a schedule**, which is [#53][issue-53]. The access log is deliberately outside the nightly encrypted archives, because those are fourteen full copies with no dedupe and an append-only file would be multiplied by fourteen for no recovery benefit, so the VPS's 400-day window was the **only** copy until this ran. **It is installed**, as `vps-backup-pull.timer` at 09:00 UTC daily with `Persistent=true`, and a first copy exists: 42 archives and 4 log files, pulled 2026-08-08 12:59 UTC. **That run was started by hand, so the timer has never fired**, which is the distinction worth keeping until 2026-08-09 09:00 UTC proves the schedule rather than the script. One copy is a fact; "backed up daily" is still a unit file. - - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returns 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently, and the only propagation mechanism is a note in the channel. Removing that one file is the maintainer's call, since it is a deletion inside a backup tree. + - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returned 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently. **That one file is deleted and the archive reads 58**, and the general case now has a mechanism: the host side keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it alongside the data it explains rather than relying on a note in a channel file nobody rereads. It records what a file contained rather than what it was called, and it will be counted among the pulled log files. + - **Read a date from a line's `StartUTC` rather than from the filename holding it.** That is the durable form of the lesson, and it is in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" as well. + - **An off-host copy also predates the timer.** The directory mtimes on the backup host are copy times where the file mtimes are the VPS's, and they show a pull at 2026-08-08 03:31 UTC that the service journal has no record of, because the script was run directly rather than through `systemd`. So a journal with one entry is not evidence of one copy. The whole set was audited both ways afterwards and nothing else had diverged: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no `--delete`. ## Owed to the hub From 45e1cc9b43825f34be4c35d54eae1bd7de2c3745 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 09:25:15 -0700 Subject: [PATCH 02/14] Record the backup and 404 flow test, and what reading it wrong looked like Both halves were exercised without running the service, so the second journal entry stays the proof that 2026-08-09 09:00 UTC fired the timer rather than a hand. The transport was checked with --dry-run and the timer was read: NEXT 09:08:45 UTC, LAST still `-`. Four things the run found, each recorded where it binds rather than only in a session log: - A referer does not implicate this site unless it points somewhere else. All 36 referer-bearing site-host 404s were scanners setting Referer to the request URL, so the unrefined rule reported three dozen broken links on a site that had none. - ServiceName separates an edge 404 from a site 404 inside one log, which the tier table described conceptually and gave no way to apply. 99 of 101 reached the site; 2 did not. - jq 'select(...)' with no projection pretty-prints, so piping it to wc -l counts lines rather than records. It reported 37 and 1,332 where the true counts were 1 and 36. - The archive directory carries MANIFEST.txt as well as RECONCILE.md, so the file count exceeds the number of logs by two, not one. The outward pass itself was clean: 73 distinct site-host 404 paths, every one a scanner shape, nothing owed to golden-urls.txt. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 18 +++++++++++++++++- TODO.md | 4 ++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index d80eab5..3bb24b2 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -223,8 +223,24 @@ A request crosses the proxy before it reaches the site, so no single log answers **A 404 count taken from Caddy alone is therefore a floor, not a total.** A request the edge refused is a reader who found nothing just as surely, and it appears in no Caddy log. Read the edge for what never arrived and Caddy for what arrived and failed, and treat the two as one answer. +**`ServiceName` is what separates those two cases inside the edge log itself**, which is otherwise a distinction this table draws conceptually and leaves you no way to apply. A Traefik line carrying a service name was routed and the 404 came from the site; a line with the field absent matched no router at all, so the edge answered and the site never saw the request. The second kind is the one Caddy is structurally blind to, and it is rare enough that it reads as noise in a total and is worth listing individually. On 2026-08-08, 99 of 101 site-host 404s carried `1-Blog-Production-service@http` and 2 carried nothing, the pair being `/` and `/favicon.ico` from one client inside the same second. + Two properties of the Caddy side are worth knowing before parsing it. Its access log is `format console`, so each line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON itself, and a parser that assumes one object per line reads nothing. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, which is the same setting "Serving" describes as a security boundary. Without it every request in the log appears to come from one internal address, and the inward pass cannot distinguish a reader from a health check. +### Reading a 404 list without being fooled by it + +The outward pass is four filters over the edge log, and each one exists because skipping it produced a wrong answer once. + +**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. Filter on user agent: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0` and the 711 that remained are the entire real dataset. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. + +**A referer does not implicate this site unless it points somewhere else.** The rule worth applying is that a 404 carrying a referer is a broken link and a 404 without one is a typed or probed address, and it fails on scanners, which set `Referer` to the request URL itself. Every one of the 36 referer-bearing site-host 404s on 2026-08-08 was self-referential, so the unrefined rule reported three dozen broken links on a site that had none. Compare the referer against `scheme://RequestHost + RequestPath` and discard the matches before counting. + +**Filter the scanner shapes by shape, never by investigating them.** A site that used to run WordPress attracts probes for `.env` and its dozen variants, `wp-config.php`, `.git/config`, `phpinfo.php`, cloud credential files, and framework config paths. They dominate the raw list and none is ever a finding. What is left after the three filters above is small enough to read line by line, which is the point of running them. + +**Then cross-reference what remains against the contract**, because that is the only step with an action. A surviving 404 whose path appears in [`checks/golden-urls.txt`](./checks/golden-urls.txt) or in [`deploy/maps/`](./deploy/maps/) is a redirect that is not working; a surviving 404 shaped like real content and present in neither is the case this whole pass exists to find, and it is added to the golden list with a redirect per that file's maintenance rules. A run where nothing survives is the expected result and should be recorded as one. + +**Two `jq` mistakes each read as a plausible answer rather than as an error.** A hyphenated key parses as subtraction, so `.request_User-Agent` silently is not the field you meant and `.["request_User-Agent"]` is; the same holds for `Referer`. And `jq 'select(...)'` with no projection pretty-prints each match across many lines, so piping it to `wc -l` counts lines rather than records and overstates by roughly the width of the object — it reported 37 and 1,332 where the true counts were 1 and 36. Project with `@tsv` or pass `-c` before counting anything. + ### Retention Is the Prerequisite, and It Belongs to the Host **On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=` traffic is visible as itself. Request headers are dropped except `Referer` and `User-Agent`, which is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. @@ -239,7 +255,7 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access **The off-host copy of the access log exists, and the schedule that maintains it is younger than the copy.** The pull to the backup host is installed as a `systemd` timer running daily at 09:00 UTC, chosen to sit behind both producers on the VPS rather than beside them, and its first copy was made by hand rather than by the timer. Read the unit and its last run on the backup host rather than trusting this paragraph, for the same reason retention is read from the VPS: a claim about a schedule is only worth what the machine says. -**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it.** The reconciliation itself now travels with the data: the VPS keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it automatically and a rename does not depend on someone rereading a channel file. It records what a file contained rather than what it was called, and it is counted among the pulled log files, so expect the file count to exceed the number of logs by one. +**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it.** The reconciliation itself now travels with the data: the VPS keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it automatically and a rename does not depend on someone rereading a channel file. It records what a file contained rather than what it was called, and it is counted among the pulled log files. **The VPS keeps a `MANIFEST.txt` in the same directory**, so expect the count to exceed the number of logs by two rather than by one, and expect any further explanatory file the host side adds to raise it again. Read the count as logs-plus-prose rather than as a number with a fixed offset. **A journal with one entry is not evidence of one copy.** The pull can be run directly as well as by its timer, and a direct run writes no service record. Directory mtimes on the backup host are the copy times, where the file mtimes are the VPS's, so those are what to read when establishing when something arrived. diff --git a/TODO.md b/TODO.md index 7922b27..ea72b90 100644 --- a/TODO.md +++ b/TODO.md @@ -55,6 +55,10 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returned 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently. **That one file is deleted and the archive reads 58**, and the general case now has a mechanism: the host side keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it alongside the data it explains rather than relying on a note in a channel file nobody rereads. It records what a file contained rather than what it was called, and it will be counted among the pulled log files. - **Read a date from a line's `StartUTC` rather than from the filename holding it.** That is the durable form of the lesson, and it is in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" as well. - **An off-host copy also predates the timer.** The directory mtimes on the backup host are copy times where the file mtimes are the VPS's, and they show a pull at 2026-08-08 03:31 UTC that the service journal has no record of, because the script was run directly rather than through `systemd`. So a journal with one entry is not evidence of one copy. The whole set was audited both ways afterwards and nothing else had diverged: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no `--delete`. + - **Both halves were exercised 2026-08-08 16:18–16:22 UTC, deliberately without running the service.** `systemctl start` would have written the second journal entry that 2026-08-09 is supposed to prove, so the transport was exercised with `--dry-run` instead and the timer was read rather than triggered: `NEXT` is 2026-08-09 09:08:45 UTC inside the 15-minute randomized window, `LAST` is `-`, and the journal still holds exactly one entry. **If a second entry exists before that time, someone ran it by hand and the schedule is still unproven.** The dry run reached the VPS over SSH and all three legs planned cleanly. Pending for the first timed run: five encrypted archives dated 2026-08-08, plus `RECONCILE.md` **and** `MANIFEST.txt`, so the log-file count rises by two non-log files rather than the one recorded above. No rotated access log is pending, which is correct — the 00:00 UTC rotation that produces `access.log-2026-08-08` has not happened yet. + - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is in `/home/pieter/vps-backup-pull-patch/vps-backup-pull` and is not installed**: writing `/usr/local/bin/vps-backup-pull` needs root and was refused, so the installed copy and the patch copy now differ by exactly that hunk, where they had been identical. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. + - **The outward pass ran end to end on 2026-08-08 traffic and found nothing to add.** 9,996 edge requests, 9,285 of them this repo's own deploy gate; of the 711 that remain, 101 were site-host 404s across 73 distinct paths, and every one was a scanner shape. Only `/` and `/robots.txt` intersect the URL contract at all, and both are explained rather than open: `/robots.txt` 404ed until the 15:47:17 deploy and has answered 200 since 15:48:45, and `/` 404ed twice at the edge, below. **No legacy content URL 404ed, so `checks/golden-urls.txt` needs no addition from this run** — which is the expected result and is recorded because an unrecorded clean pass is indistinguishable from a pass nobody ran. + - **Two site-host 404s came from the edge rather than the site, and belong to the VPS side.** `/` and `/favicon.ico` at 2026-08-08T15:39:01, one client, same second, both carrying no `ServiceName` at all where the other 99 carried `1-Blog-Production-service@http`. No router matched, so Traefik answered and the blog never saw the request; requests to `/` seventeen seconds later were routed normally. It sits inside the window the host side was reconfiguring Pangolin in, which is a plausible cause and not a measured one. Raise it in the channel rather than diagnosing it from this side, and note that Caddy is structurally blind to it: a 404 count taken from the site's own log would report zero of these. ## Owed to the hub From d36a40f3869049a70bbecf22b5f2b85effca8fd1 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 09:31:29 -0700 Subject: [PATCH 03/14] Correct the pull script's install path, and name the merge that loses the log leg The header's own deploy command wrote /usr/local/sbin/vps-backup-pull. The service runs /usr/local/bin/vps-backup-pull and sbin/ is empty, so following it installs a copy nothing executes while scp and chmod both report success and the timer keeps running the previous one. The VPS canonical had already corrected this and explains why bin/ is deliberate: the script authenticates with the pieter key and must not run under sudo. Measuring that turned up the larger problem. The two copies have diverged in both directions, not one. This side has the entire access-log leg, the tell() fix and the VERIFIED counter; the canonical has the corrected install block and the no-sudo rationale. An scp from the canonical therefore deletes the log pull outright, which is the half the log review runs on -- where the record said it would cost only the --quiet fix. The stale block is replaced in the patch copy with the canonical's wording, plus a refusal to run that scp until #53 reconciles the two. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/TODO.md b/TODO.md index ea72b90..fbc0f85 100644 --- a/TODO.md +++ b/TODO.md @@ -57,6 +57,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **An off-host copy also predates the timer.** The directory mtimes on the backup host are copy times where the file mtimes are the VPS's, and they show a pull at 2026-08-08 03:31 UTC that the service journal has no record of, because the script was run directly rather than through `systemd`. So a journal with one entry is not evidence of one copy. The whole set was audited both ways afterwards and nothing else had diverged: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no `--delete`. - **Both halves were exercised 2026-08-08 16:18–16:22 UTC, deliberately without running the service.** `systemctl start` would have written the second journal entry that 2026-08-09 is supposed to prove, so the transport was exercised with `--dry-run` instead and the timer was read rather than triggered: `NEXT` is 2026-08-09 09:08:45 UTC inside the 15-minute randomized window, `LAST` is `-`, and the journal still holds exactly one entry. **If a second entry exists before that time, someone ran it by hand and the schedule is still unproven.** The dry run reached the VPS over SSH and all three legs planned cleanly. Pending for the first timed run: five encrypted archives dated 2026-08-08, plus `RECONCILE.md` **and** `MANIFEST.txt`, so the log-file count rises by two non-log files rather than the one recorded above. No rotated access log is pending, which is correct — the 00:00 UTC rotation that produces `access.log-2026-08-08` has not happened yet. - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is in `/home/pieter/vps-backup-pull-patch/vps-backup-pull` and is not installed**: writing `/usr/local/bin/vps-backup-pull` needs root and was refused, so the installed copy and the patch copy now differ by exactly that hunk, where they had been identical. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. + - **The two copies have diverged in both directions, and the header told you to run the merge that loses this side.** The record so far says this side owes the canonical the `--quiet` fix; measured 2026-08-08, it owes considerably more and is owed something back. This copy carries the entire access-log leg — section 4, `--no-logs`, `LOG_DEST`, both `REMOTE_*_LOG_DIR` variables and `LOG_RSYNC_OPTS` — plus the `tell()` fix and the `VERIFIED` counter, none of which exist on the canonical. The canonical carries a corrected install block and a paragraph explaining that the script must run as `pieter` rather than under `sudo`, neither of which existed here. **So `scp`-ing the canonical over this file deletes the log pull outright**, which is the half [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" depends on, and the previously recorded cost of that copy — losing the `--quiet` fix — understated it by the whole feature. + - **The header's own install command pointed at `/usr/local/sbin/vps-backup-pull`, which nothing runs.** `vps-backup-pull.service` runs `/usr/local/bin/vps-backup-pull` and `/usr/local/sbin/` is empty, so following the instruction would have written a second copy nobody executes while `scp` and `chmod` both reported success and the timer went on running the old one. The canonical had already corrected this to a `sudo install` into `bin/`, and states the reason `bin/` is deliberate. **The stale block is replaced in the patch copy** with the canonical's wording plus an explicit refusal to run that `scp` until the divergence above is reconciled. It is another reason a plain overwrite in either direction is the wrong merge. - **The outward pass ran end to end on 2026-08-08 traffic and found nothing to add.** 9,996 edge requests, 9,285 of them this repo's own deploy gate; of the 711 that remain, 101 were site-host 404s across 73 distinct paths, and every one was a scanner shape. Only `/` and `/robots.txt` intersect the URL contract at all, and both are explained rather than open: `/robots.txt` 404ed until the 15:47:17 deploy and has answered 200 since 15:48:45, and `/` 404ed twice at the edge, below. **No legacy content URL 404ed, so `checks/golden-urls.txt` needs no addition from this run** — which is the expected result and is recorded because an unrecorded clean pass is indistinguishable from a pass nobody ran. - **Two site-host 404s came from the edge rather than the site, and belong to the VPS side.** `/` and `/favicon.ico` at 2026-08-08T15:39:01, one client, same second, both carrying no `ServiceName` at all where the other 99 carried `1-Blog-Production-service@http`. No router matched, so Traefik answered and the blog never saw the request; requests to `/` seventeen seconds later were routed normally. It sits inside the window the host side was reconfiguring Pangolin in, which is a plausible cause and not a measured one. Raise it in the channel rather than diagnosing it from this side, and note that Caddy is structurally blind to it: a 404 count taken from the site's own log would report zero of these. From 4ea6f94773896c3bcfe8ee7eecab70e10fdd4045 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 09:40:48 -0700 Subject: [PATCH 04/14] Give the VPS an operational entry point, and its paths an env home Every VPS fact was recoverable only by asking: the host and the log paths lived in the pull script's own defaults and a gitignored session note, "The Channel" used a placeholder with no value behind it, and Log Review told you to read an archive without saying where it is. So the docs described the work and could not start it. Six values now name what a machine owns rather than what the project does, following the CAPTURE_ROOT precedent -- environment-independent, so they sit in the default file: VPS_SSH_HOST VPS_TRAEFIK_LOG VPS_TRAEFIK_LOG_ARCHIVE VPS_COMMS_DIR LOG_ARCHIVE_ROOT BACKUP_ARCHIVE_ROOT All six verified against both ends before being written down. A new "Working With the VPS" section is the entry point: the values, the two credentials and why confusing them fails badly, why the copy is a pull, why today's traffic is never in it, and pointers to the four sections that already cover the rest. The channel transfers are documented as the one deliberate exception. An allow rule matches command text rather than its expansion, so substituting the variables there converts an allowlisted transfer into one that prompts while looking like a tidy-up. Also records the measured backup exposure: duplicacy mounts /data only, so the canonical is protected three ways while the copy that actually runs -- the superset carrying the log leg -- is in no snapshot, no archive and no repository. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 32 ++++++++++++++++++++++++++++++++ TODO.md | 3 +++ deploy/env.example | 26 ++++++++++++++++++++++++++ 3 files changed, 61 insertions(+) diff --git a/OPERATIONS.md b/OPERATIONS.md index 3bb24b2..5e8d582 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -196,6 +196,36 @@ The script asserts both halves of that rather than assuming them. It fails when **Nothing prunes on the deploy path, and that is what keeps the deploy key's capability small.** A prune racing a deploy could take the rollback target, where a lingering release only costs disk. This is also why the key needs no delete capability, which is the property "Server Hardening" depends on. The count and the timer belong to the host, so this section records what the host declares rather than holding a second copy of it. See "Who Owns What". +## Working With the VPS + +**Every path and hostname on this page is a value in `secrets/`, never a literal to be remembered or asked for.** The convention is the one "Environments" describes and `CAPTURE_ROOT` already follows: a value naming a machine rather than the project lives in the environment file, is sourced with `set -a`, and is read from there rather than searched for. The VPS values are environment-independent, because there is one such host rather than one per environment, so they sit in the default file alongside `CAPTURE_ROOT`. + +```sh +set -a; . secrets/local.production.env; set +a +ssh "$VPS_SSH_HOST" true && echo reachable +``` + +| Value | Names | Side | +| --- | --- | --- | +| `VPS_SSH_HOST` | the administrative login | the VPS | +| `VPS_TRAEFIK_LOG` | today's live access log, still being appended to | the VPS | +| `VPS_TRAEFIK_LOG_ARCHIVE` | the rotated access logs, and the source of the off-host copy | the VPS | +| `VPS_COMMS_DIR` | the two agent channel files | the VPS | +| `LOG_ARCHIVE_ROOT` | the off-host copy of the rotated logs | the backup host | +| `BACKUP_ARCHIVE_ROOT` | the off-host encrypted archives and the plaintext hostconfig tree beside them | the backup host | + +**There are two credentials to this host and picking the wrong one is the first mistake to avoid.** `DEPLOY_SSH_USER`, held per environment and used only by the deploy, reaches a confined account behind an `rrsync` forced command that can write one release tree and read nothing else. `VPS_SSH_HOST` is the ordinary administrative login used for everything on this page. They are deliberately separate credentials with different blast radii, so reaching for the deploy account to read a log fails in a way that reads like an outage, and reaching for the admin account to deploy grants far more than the deploy needs. + +**The off-host copy is a pull, made by the backup host, and nothing on the VPS knows it happens.** That direction is the security property rather than an implementation detail: the VPS holds no credential reaching any other system, so a compromise there cannot walk into the backup. The pull writes `LOG_ARCHIVE_ROOT` and `BACKUP_ARCHIVE_ROOT`, runs on its own timer on the backup host, and is described under "Retention Is the Prerequisite, and It Belongs to the Host". Read its unit and its last run on that host rather than trusting a schedule written down anywhere, including here. + +**Today's traffic is never in the off-host copy, and that is deliberate.** Rotation is what makes a file eligible to be pulled, so a live log would be copied as a torn prefix and fetched again on the next run. An analysis covering today therefore reads `VPS_TRAEFIK_LOG` over SSH and everything older from `LOG_ARCHIVE_ROOT`, and treats the two as one series joined on `StartUTC` rather than on which file a line came from. + +**The plaintext `hostconfig` tree under `BACKUP_ARCHIVE_ROOT` is the readable copy of the VPS's own configuration**, carrying the same non-secret files the encrypted archives hold. It exists so a rebuild does not depend on the encryption key, which is not on the backup host and must never be put there — beside the ciphertext it would make the encryption decorative. What that tree covers is whatever the VPS advertises, read from the host rather than duplicated here, so it tracks the host instead of drifting from a list. + +**The channel transfers are the one exception, and they must stay literal.** The permission allowlist in `.claude/settings.local.json` matches the text of a command rather than what it expands to, so substituting `"$VPS_SSH_HOST:$VPS_COMMS_DIR/..."` into those two `rsync` lines turns an allowed command into one that prompts, while looking like a tidy-up that changed nothing. Use the values above everywhere else, and leave the two commands under "The Channel Between the Two Sides" spelled out exactly as they are written there. + +**What this section does not cover, and where it lives instead.** Reading the logs for content is "Log Review"; exchanging rounds with the agent that owns the host is "The Channel Between the Two Sides"; the boundary of which side fixes what is "Who Owns What"; and what a rebuild restores, including the host-key step that blocks both deploy and rollback, is "Backup and Restore". + ## Log Review **Real traffic is the only source that finds what every check here is blind to.** The URL contract proves the URLs someone thought to list and the redirects derived from the export. It cannot know about a URL nobody recorded, because the lists are their own standard: the gates check the built site and the running server against those lists, never against the old platform that served the addresses. An address the crawl missed is therefore missing from every gate that reads them, and a visitor following a sixteen-year-old link is the one reader who tests for it. @@ -289,6 +319,8 @@ rsync -a root@:/srv/agent-comms/vps-agent.md comms/vps-agent.md rsync -a --no-o --no-g --chmod=F644 comms/blog-agent.md root@:/srv/agent-comms/blog-agent.md ``` +**Spell both commands out rather than reading the host and directory from `secrets/`**, which is the opposite of the rule "Working With the VPS" sets for every other path, and is deliberate. These two are allowlisted in `.claude/settings.local.json`, and an allow rule matches the text of the command rather than the value it expands to, so replacing the literals with `$VPS_SSH_HOST` and `$VPS_COMMS_DIR` turns an allowed transfer into one that prompts. The same rule is why neither may be chained behind `cd` or `&&`: an allow rule matches a standalone command only. + **The push suppresses owner and group deliberately.** `-a` implies `-o` and `-g`, and the transfer connects as root, so a plain `rsync -a` carries this workstation's numeric uid onto a host that has no such user and leaves the file owned by a number. Four rules, each covering a way the channel has already failed or could: diff --git a/TODO.md b/TODO.md index fbc0f85..52c4bec 100644 --- a/TODO.md +++ b/TODO.md @@ -78,6 +78,9 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions - **Where the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy are an operational asset built from another agent's instructions, and they exist only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. + - **Measured 2026-08-08, and the exposure is inverted from the way it reads above.** The canonical on the VPS is not the copy at risk: it is captured three times over, as plaintext at `/data/backup/vps/hostconfig/usr/local/share/pangolin-maint/vps-pull.sh`, inside every encrypted archive, and off-site through duplicacy. **The copy with no protection at all is the one that actually runs.** The duplicacy container mounts `/data` read-only and nothing else, so `/usr/local/bin/vps-backup-pull`, `/home/pieter/vps-backup-pull-patch/`, and `/etc/systemd/system/vps-backup-pull.{service,timer}` are in no snapshot, no archive, and no repository. That is precisely the superset described above — the log leg, `tell()`, `VERIFIED` — so **losing this host loses the only copy of the half the log review depends on, while the version without it survives everywhere.** A restore would silently bring back a pull that copies no logs. + - **The directory holding it is named as though it were disposable.** `~/vps-backup-pull-patch` reads as a patch staged against a source, and there is no source: it is the most complete copy of the script in existence. A directory named for a temporary artifact is the one a cleanup deletes, and nothing here would notice until a restore produced the wrong script. + - **The candidate homes are no longer balanced.** The duplicacy container that performs this host's off-site backup is already defined in `HomeAutomation-Config/Docker/stacks/services/compose.yml`, and its B2 credential already lives in that repository's `secrets/`. So server-2's service configuration has a git home, `vps-backup-pull` is a server-2 service, and it is the one such service absent from it. The argument for this repository — that "Log Review" stops working without the pull — argues for a pointer from [`OPERATIONS.md`](./OPERATIONS.md), not for a second copy of another host's tooling. Still the maintainer's call, but it is now a recommendation rather than a pair of equal options. - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. - **What `robots.txt` says, which is undecided and is the last non-mechanical item before M7b.** Recorded under "Next" above, where it blocks the cutover. diff --git a/deploy/env.example b/deploy/env.example index 7978bdd..f9aa36f 100644 --- a/deploy/env.example +++ b/deploy/env.example @@ -68,3 +68,29 @@ EXPECT_SITE_ENV=production # Environment-independent, unlike every value above, so keep it in the default file, secrets/local.production.env, and drop it from any per-environment copy of this template. # Nothing sources this value, so it is recorded to keep a rebuild from depending on memory. CAPTURE_ROOT=/path/to/blog-capture + +# The off-host copy of the VPS access log, which OPERATIONS.md "Log Review" reads and vps-backup-pull writes. +# The encrypted host archives sit in a sibling directory and belong to the host rather than to this repo, so they are not named here. +# Environment-independent, like CAPTURE_ROOT above, so keep it in secrets/local.production.env and drop it from any per-environment copy. +# Nothing sources this value either. It is recorded because the alternative is a path that lives only in a gitignored session note and in the pull script's own defaults, which is how a review procedure ends up unable to say where the data it reviews is kept. +LOG_ARCHIVE_ROOT=/path/to/vps-logs + +# The off-host copy of the encrypted host archives and the plaintext hostconfig tree beside them. +# The plaintext copy is what makes the VPS's own configuration readable without the encryption key, which is not held here and must not be. +BACKUP_ARCHIVE_ROOT=/path/to/vps-archives + +# The VPS administrative connection, which is NOT the deploy connection above. +# DEPLOY_SSH_USER reaches a confined account behind an rrsync forced command that can write one release tree and read nothing. +# This one is the ordinary admin login, used for reading logs, reading the archive directory, and moving the channel files. +# They are separate values because they are separate credentials with different blast radii, and reaching for the wrong one either fails confusingly or grants far more than the task needed. +# Environment-independent, like CAPTURE_ROOT above: there is one such host, not one per environment. +VPS_SSH_HOST=root@vps.example.com + +# Where the VPS keeps the access log, whose rotated files LOG_ARCHIVE_ROOT is the off-host copy of. +# Today's live file is deliberately never pulled, because it is still being appended to and a copy is a torn prefix. +# So an analysis covering today reads VPS_TRAEFIK_LOG over SSH and everything older from LOG_ARCHIVE_ROOT. +VPS_TRAEFIK_LOG=/var/log/traefik/access.log +VPS_TRAEFIK_LOG_ARCHIVE=/var/log/traefik/archive + +# The two agent channel files, described under "The Channel Between the Two Sides". +VPS_COMMS_DIR=/srv/agent-comms From 6377d9cb7b3b0bc24c2404e624110dd4ef97bb11 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 10:17:29 -0700 Subject: [PATCH 05/14] Put the backup pull in git, and make its host values configuration The script that makes the off-host copy was the one thing with no copy anywhere: /usr/local/bin on the backup host and an untracked directory beside it, on the same machine the copy lives on. It is now ops/, with both units, an EnvironmentFile template, a drop-in template, and a README covering what it does and how to check it. Nothing in ops/ names a machine. VPS_HOST, DEST and LOG_DEST have no defaults and preflight refuses to run without them, because a wrong-but-valid destination is a backup nobody can find. User= and RequiresMountsFor= move to a drop-in for the same reason, and the script now refuses to run as root rather than warning about it, since that is what a missing drop-in would otherwise produce. The guard sits ahead of the banner deliberately. A banner printed with empty values reads, in a journal skim, exactly like a run that started and then failed somewhere real. OPERATIONS.md "Working With the VPS" names the script, and states which of its variables pair with this repo's own: DEST is BACKUP_ARCHIVE_ROOT and LOG_DEST is LOG_ARCHIVE_ROOT. Neither side can source the other's file, so the correspondence is stated rather than enforced. Gates run per GOVERNANCE.md, through Docker rather than npx: markdownlint, cspell, editorconfig-checker, and the hub's prose_lint.py. The last one found British spelling and an observed home path in prose I had written, and both files are back to their baseline counts. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 24 +- README.md | 2 + TODO.md | 17 +- ops/README.md | 79 +++++ ops/vps-backup-pull | 285 ++++++++++++++++++ ops/vps-backup-pull.env.example | 39 +++ ops/vps-backup-pull.service | 43 +++ ...s-backup-pull.service.d-local.conf.example | 19 ++ ops/vps-backup-pull.timer | 19 ++ 9 files changed, 514 insertions(+), 13 deletions(-) create mode 100644 ops/README.md create mode 100755 ops/vps-backup-pull create mode 100644 ops/vps-backup-pull.env.example create mode 100644 ops/vps-backup-pull.service create mode 100644 ops/vps-backup-pull.service.d-local.conf.example create mode 100644 ops/vps-backup-pull.timer diff --git a/OPERATIONS.md b/OPERATIONS.md index 5e8d582..34d5656 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -216,11 +216,25 @@ ssh "$VPS_SSH_HOST" true && echo reachable **There are two credentials to this host and picking the wrong one is the first mistake to avoid.** `DEPLOY_SSH_USER`, held per environment and used only by the deploy, reaches a confined account behind an `rrsync` forced command that can write one release tree and read nothing else. `VPS_SSH_HOST` is the ordinary administrative login used for everything on this page. They are deliberately separate credentials with different blast radii, so reaching for the deploy account to read a log fails in a way that reads like an outage, and reaching for the admin account to deploy grants far more than the deploy needs. -**The off-host copy is a pull, made by the backup host, and nothing on the VPS knows it happens.** That direction is the security property rather than an implementation detail: the VPS holds no credential reaching any other system, so a compromise there cannot walk into the backup. The pull writes `LOG_ARCHIVE_ROOT` and `BACKUP_ARCHIVE_ROOT`, runs on its own timer on the backup host, and is described under "Retention Is the Prerequisite, and It Belongs to the Host". Read its unit and its last run on that host rather than trusting a schedule written down anywhere, including here. +**The off-host copy is made by a script in this repository, [`ops/vps-backup-pull`](./ops/vps-backup-pull), on a `systemd` timer on the backup host.** It copies three things off the VPS into `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT`: the encrypted archives, a plaintext copy of the same non-secret host files, and the rotated access logs. What it does, the three behaviors that look like bugs and are not, how to install it, and how to check it ran are in [`ops/README.md`](./ops/README.md). Read the unit and its last run on the backup host rather than trusting a schedule written down anywhere, including here. + +**It is a pull rather than a push, and nothing on the VPS knows it happens.** That direction is the security property rather than an implementation detail: the backup host holds a key the VPS trusts, and the VPS holds no credential reaching any other system, so a compromise of the web server cannot walk into the backups that exist to survive it. + +**The two sides use different names for the same two directories, and the pairs must agree.** The pull is configured on the backup host and writes, where this repository's `secrets/` file is read by the log review. Neither can source the other's file, because one is a `systemd` `EnvironmentFile` on another machine and the other is a shell file here, so the correspondence is stated rather than enforced, and it is the thing to check first when a review finds an empty archive. + +| The pull writes | The review reads | Holding | +| --- | --- | --- | +| `DEST` | `BACKUP_ARCHIVE_ROOT` | the encrypted archives and the plaintext `hostconfig` tree | +| `LOG_DEST` | `LOG_ARCHIVE_ROOT` | both sets of rotated logs | + +```sh +set -a; . secrets/local.production.env; set +a +ls -d "$LOG_ARCHIVE_ROOT" "$BACKUP_ARCHIVE_ROOT" +``` **Today's traffic is never in the off-host copy, and that is deliberate.** Rotation is what makes a file eligible to be pulled, so a live log would be copied as a torn prefix and fetched again on the next run. An analysis covering today therefore reads `VPS_TRAEFIK_LOG` over SSH and everything older from `LOG_ARCHIVE_ROOT`, and treats the two as one series joined on `StartUTC` rather than on which file a line came from. -**The plaintext `hostconfig` tree under `BACKUP_ARCHIVE_ROOT` is the readable copy of the VPS's own configuration**, carrying the same non-secret files the encrypted archives hold. It exists so a rebuild does not depend on the encryption key, which is not on the backup host and must never be put there — beside the ciphertext it would make the encryption decorative. What that tree covers is whatever the VPS advertises, read from the host rather than duplicated here, so it tracks the host instead of drifting from a list. +**The plaintext `hostconfig` tree under `BACKUP_ARCHIVE_ROOT` is the readable copy of the VPS's own configuration**, carrying the same non-secret files the encrypted archives hold. It exists so a rebuild does not depend on the encryption key, which is not on the backup host and must never be put there, because beside the ciphertext it would make the encryption decorative. What that tree covers is whatever the VPS advertises, read from the host rather than duplicated here, so it tracks the host instead of drifting from a list. **The channel transfers are the one exception, and they must stay literal.** The permission allowlist in `.claude/settings.local.json` matches the text of a command rather than what it expands to, so substituting `"$VPS_SSH_HOST:$VPS_COMMS_DIR/..."` into those two `rsync` lines turns an allowed command into one that prompts, while looking like a tidy-up that changed nothing. Use the values above everywhere else, and leave the two commands under "The Channel Between the Two Sides" spelled out exactly as they are written there. @@ -253,7 +267,7 @@ A request crosses the proxy before it reaches the site, so no single log answers **A 404 count taken from Caddy alone is therefore a floor, not a total.** A request the edge refused is a reader who found nothing just as surely, and it appears in no Caddy log. Read the edge for what never arrived and Caddy for what arrived and failed, and treat the two as one answer. -**`ServiceName` is what separates those two cases inside the edge log itself**, which is otherwise a distinction this table draws conceptually and leaves you no way to apply. A Traefik line carrying a service name was routed and the 404 came from the site; a line with the field absent matched no router at all, so the edge answered and the site never saw the request. The second kind is the one Caddy is structurally blind to, and it is rare enough that it reads as noise in a total and is worth listing individually. On 2026-08-08, 99 of 101 site-host 404s carried `1-Blog-Production-service@http` and 2 carried nothing, the pair being `/` and `/favicon.ico` from one client inside the same second. +**`ServiceName` is what separates those two cases inside the edge log itself**, which is otherwise a distinction this table draws conceptually and leaves you no way to apply. A Traefik line carrying a service name was routed, so the 404 came from the site. A line with the field absent matched no router at all, so the edge answered and the site never saw the request. The second kind is the one Caddy is structurally blind to, and it is rare enough that it reads as noise in a total and is worth listing individually. On 2026-08-08, 99 of 101 site-host 404s carried `1-Blog-Production-service@http` and 2 carried nothing, the pair being `/` and `/favicon.ico` from one client inside the same second. Two properties of the Caddy side are worth knowing before parsing it. Its access log is `format console`, so each line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON itself, and a parser that assumes one object per line reads nothing. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, which is the same setting "Serving" describes as a security boundary. Without it every request in the log appears to come from one internal address, and the inward pass cannot distinguish a reader from a health check. @@ -267,9 +281,9 @@ The outward pass is four filters over the edge log, and each one exists because **Filter the scanner shapes by shape, never by investigating them.** A site that used to run WordPress attracts probes for `.env` and its dozen variants, `wp-config.php`, `.git/config`, `phpinfo.php`, cloud credential files, and framework config paths. They dominate the raw list and none is ever a finding. What is left after the three filters above is small enough to read line by line, which is the point of running them. -**Then cross-reference what remains against the contract**, because that is the only step with an action. A surviving 404 whose path appears in [`checks/golden-urls.txt`](./checks/golden-urls.txt) or in [`deploy/maps/`](./deploy/maps/) is a redirect that is not working; a surviving 404 shaped like real content and present in neither is the case this whole pass exists to find, and it is added to the golden list with a redirect per that file's maintenance rules. A run where nothing survives is the expected result and should be recorded as one. +**Then cross-reference what remains against the contract**, because that is the only step with an action. A surviving 404 whose path appears in [`checks/golden-urls.txt`](./checks/golden-urls.txt) or in [`deploy/maps/`](./deploy/maps/) is a redirect that is not working. A surviving 404 shaped like real content and present in neither is the case this whole pass exists to find, and it is added to the golden list with a redirect per that file's maintenance rules. A run where nothing survives is the expected result and should be recorded as one. -**Two `jq` mistakes each read as a plausible answer rather than as an error.** A hyphenated key parses as subtraction, so `.request_User-Agent` silently is not the field you meant and `.["request_User-Agent"]` is; the same holds for `Referer`. And `jq 'select(...)'` with no projection pretty-prints each match across many lines, so piping it to `wc -l` counts lines rather than records and overstates by roughly the width of the object — it reported 37 and 1,332 where the true counts were 1 and 36. Project with `@tsv` or pass `-c` before counting anything. +**Two `jq` mistakes each read as a plausible answer rather than as an error.** A hyphenated key parses as subtraction, so `.request_User-Agent` silently is not the field you meant and `.["request_User-Agent"]` is, and the same holds for `Referer`. And `jq 'select(...)'` with no projection pretty-prints each match across many lines, so piping it to `wc -l` counts lines rather than records and overstates by roughly the width of the object. It reported 37 and 1,332 where the true counts were 1 and 36. Project with `@tsv` or pass `-c` before counting anything. ### Retention Is the Prerequisite, and It Belongs to the Host diff --git a/README.md b/README.md index 10cb4dd..1e46680 100644 --- a/README.md +++ b/README.md @@ -152,6 +152,7 @@ flowchart LR | [`hugo.yaml`][hugo-config] | site configuration, taxonomy URLs, and the feed name | | [`checks/`][checks] | the URL contract and the gates that enforce it | | [`deploy/`][deploy] | the release script, the web-server config, and the redirect maps | +| [`ops/`][ops] | the pull that copies the server's backups and access logs off it, and its schedule | Deploy paths, environment variables, and the server layout are documented in [OPERATIONS.md][operations]. @@ -212,6 +213,7 @@ Licensed under the [MIT License][license]\ [checks]: ./checks/ [commits-link]: https://github.com/ptr727/Blog/commits [deploy]: ./deploy/ +[ops]: ./ops/ [deploy-readme]: ./deploy/README.md [discussions-link]: https://github.com/ptr727/Blog/discussions [issues-link]: https://github.com/ptr727/Blog/issues diff --git a/TODO.md b/TODO.md index 52c4bec..540f60e 100644 --- a/TODO.md +++ b/TODO.md @@ -55,12 +55,12 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returned 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently. **That one file is deleted and the archive reads 58**, and the general case now has a mechanism: the host side keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it alongside the data it explains rather than relying on a note in a channel file nobody rereads. It records what a file contained rather than what it was called, and it will be counted among the pulled log files. - **Read a date from a line's `StartUTC` rather than from the filename holding it.** That is the durable form of the lesson, and it is in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" as well. - **An off-host copy also predates the timer.** The directory mtimes on the backup host are copy times where the file mtimes are the VPS's, and they show a pull at 2026-08-08 03:31 UTC that the service journal has no record of, because the script was run directly rather than through `systemd`. So a journal with one entry is not evidence of one copy. The whole set was audited both ways afterwards and nothing else had diverged: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no `--delete`. - - **Both halves were exercised 2026-08-08 16:18–16:22 UTC, deliberately without running the service.** `systemctl start` would have written the second journal entry that 2026-08-09 is supposed to prove, so the transport was exercised with `--dry-run` instead and the timer was read rather than triggered: `NEXT` is 2026-08-09 09:08:45 UTC inside the 15-minute randomized window, `LAST` is `-`, and the journal still holds exactly one entry. **If a second entry exists before that time, someone ran it by hand and the schedule is still unproven.** The dry run reached the VPS over SSH and all three legs planned cleanly. Pending for the first timed run: five encrypted archives dated 2026-08-08, plus `RECONCILE.md` **and** `MANIFEST.txt`, so the log-file count rises by two non-log files rather than the one recorded above. No rotated access log is pending, which is correct — the 00:00 UTC rotation that produces `access.log-2026-08-08` has not happened yet. - - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is in `/home/pieter/vps-backup-pull-patch/vps-backup-pull` and is not installed**: writing `/usr/local/bin/vps-backup-pull` needs root and was refused, so the installed copy and the patch copy now differ by exactly that hunk, where they had been identical. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. - - **The two copies have diverged in both directions, and the header told you to run the merge that loses this side.** The record so far says this side owes the canonical the `--quiet` fix; measured 2026-08-08, it owes considerably more and is owed something back. This copy carries the entire access-log leg — section 4, `--no-logs`, `LOG_DEST`, both `REMOTE_*_LOG_DIR` variables and `LOG_RSYNC_OPTS` — plus the `tell()` fix and the `VERIFIED` counter, none of which exist on the canonical. The canonical carries a corrected install block and a paragraph explaining that the script must run as `pieter` rather than under `sudo`, neither of which existed here. **So `scp`-ing the canonical over this file deletes the log pull outright**, which is the half [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" depends on, and the previously recorded cost of that copy — losing the `--quiet` fix — understated it by the whole feature. + - **Both halves were exercised 2026-08-08 between 16:18 and 16:22 UTC, deliberately without running the service.** `systemctl start` would have written the second journal entry that 2026-08-09 is supposed to prove, so the transport was exercised with `--dry-run` instead and the timer was read rather than triggered: `NEXT` is 2026-08-09 09:08:45 UTC inside the 15-minute randomized window, `LAST` is `-`, and the journal still holds exactly one entry. **If a second entry exists before that time, someone ran it by hand and the schedule is still unproven.** The dry run reached the VPS over SSH and all three legs planned cleanly. Pending for the first timed run: five encrypted archives dated 2026-08-08, plus `RECONCILE.md` **and** `MANIFEST.txt`, so the log-file count rises by two non-log files rather than the one recorded above. No rotated access log is pending, which is correct, because the 00:00 UTC rotation that produces `access.log-2026-08-08` has not happened yet. + - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is committed at [`ops/vps-backup-pull`](./ops/vps-backup-pull) and is not installed**: writing `/usr/local/bin/vps-backup-pull` needs root and was refused, so the running copy still lacks it. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. + - **The VPS's older copy of the script is not a source, and reconciling the two is [#53][issue-53].** The committed copy carries the whole access-log leg, the `tell()` fix and the `VERIFIED` counter. The VPS's carries an install block and a no-sudo rationale that this side lacked, now folded in. Neither direction is a safe overwrite, so #53 is a merge rather than a copy, and copying the VPS's over the committed one would delete the log pull that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" runs on. - **The header's own install command pointed at `/usr/local/sbin/vps-backup-pull`, which nothing runs.** `vps-backup-pull.service` runs `/usr/local/bin/vps-backup-pull` and `/usr/local/sbin/` is empty, so following the instruction would have written a second copy nobody executes while `scp` and `chmod` both reported success and the timer went on running the old one. The canonical had already corrected this to a `sudo install` into `bin/`, and states the reason `bin/` is deliberate. **The stale block is replaced in the patch copy** with the canonical's wording plus an explicit refusal to run that `scp` until the divergence above is reconciled. It is another reason a plain overwrite in either direction is the wrong merge. - - **The outward pass ran end to end on 2026-08-08 traffic and found nothing to add.** 9,996 edge requests, 9,285 of them this repo's own deploy gate; of the 711 that remain, 101 were site-host 404s across 73 distinct paths, and every one was a scanner shape. Only `/` and `/robots.txt` intersect the URL contract at all, and both are explained rather than open: `/robots.txt` 404ed until the 15:47:17 deploy and has answered 200 since 15:48:45, and `/` 404ed twice at the edge, below. **No legacy content URL 404ed, so `checks/golden-urls.txt` needs no addition from this run** — which is the expected result and is recorded because an unrecorded clean pass is indistinguishable from a pass nobody ran. - - **Two site-host 404s came from the edge rather than the site, and belong to the VPS side.** `/` and `/favicon.ico` at 2026-08-08T15:39:01, one client, same second, both carrying no `ServiceName` at all where the other 99 carried `1-Blog-Production-service@http`. No router matched, so Traefik answered and the blog never saw the request; requests to `/` seventeen seconds later were routed normally. It sits inside the window the host side was reconfiguring Pangolin in, which is a plausible cause and not a measured one. Raise it in the channel rather than diagnosing it from this side, and note that Caddy is structurally blind to it: a 404 count taken from the site's own log would report zero of these. + - **The outward pass ran end to end on 2026-08-08 traffic and found nothing to add.** 9,996 edge requests, 9,285 of them this repo's own deploy gate. Of the 711 that remain, 101 were site-host 404s across 73 distinct paths, and every one was a scanner shape. Only `/` and `/robots.txt` intersect the URL contract at all, and both are explained rather than open: `/robots.txt` 404ed until the 15:47:17 deploy and has answered 200 since 15:48:45, and `/` 404ed twice at the edge, below. **No legacy content URL 404ed, so `checks/golden-urls.txt` needs no addition from this run**, which is the expected result and is recorded because an unrecorded clean pass is indistinguishable from a pass nobody ran. + - **Two site-host 404s came from the edge rather than the site, and belong to the VPS side.** `/` and `/favicon.ico` at 2026-08-08T15:39:01, one client, same second, both carrying no `ServiceName` at all where the other 99 carried `1-Blog-Production-service@http`. No router matched, so Traefik answered and the blog never saw the request, and requests to `/` seventeen seconds later were routed normally. It sits inside the window the host side was reconfiguring Pangolin in, which is a plausible cause and not a measured one. Raise it in the channel rather than diagnosing it from this side, and note that Caddy is structurally blind to it: a 404 count taken from the site's own log would report zero of these. ## Owed to the hub @@ -77,10 +77,11 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions -- **Where the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy are an operational asset built from another agent's instructions, and they exist only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. - - **Measured 2026-08-08, and the exposure is inverted from the way it reads above.** The canonical on the VPS is not the copy at risk: it is captured three times over, as plaintext at `/data/backup/vps/hostconfig/usr/local/share/pangolin-maint/vps-pull.sh`, inside every encrypted archive, and off-site through duplicacy. **The copy with no protection at all is the one that actually runs.** The duplicacy container mounts `/data` read-only and nothing else, so `/usr/local/bin/vps-backup-pull`, `/home/pieter/vps-backup-pull-patch/`, and `/etc/systemd/system/vps-backup-pull.{service,timer}` are in no snapshot, no archive, and no repository. That is precisely the superset described above — the log leg, `tell()`, `VERIFIED` — so **losing this host loses the only copy of the half the log review depends on, while the version without it survives everywhere.** A restore would silently bring back a pull that copies no logs. +- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. What remains for the pull is mechanical: install the committed copy over `/usr/local/bin/vps-backup-pull`, which needs root, and reconcile [#53][issue-53] with the VPS's older copy in both directions rather than by overwriting either. +- **Where the rest of the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy were an operational asset built from another agent's instructions, and they existed only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. + - **The pull itself is resolved and the reasoning is kept because it applies to everything still listed here.** What made it urgent was measured: the copy protected everywhere was the VPS's older one, while the copy that actually ran, carrying the log leg the review depends on, was in no snapshot and no repository. Committing it is what closed that, not the backup host's own off-site copy, which never reached the script. - **The directory holding it is named as though it were disposable.** `~/vps-backup-pull-patch` reads as a patch staged against a source, and there is no source: it is the most complete copy of the script in existence. A directory named for a temporary artifact is the one a cleanup deletes, and nothing here would notice until a restore produced the wrong script. - - **The candidate homes are no longer balanced.** The duplicacy container that performs this host's off-site backup is already defined in `HomeAutomation-Config/Docker/stacks/services/compose.yml`, and its B2 credential already lives in that repository's `secrets/`. So server-2's service configuration has a git home, `vps-backup-pull` is a server-2 service, and it is the one such service absent from it. The argument for this repository — that "Log Review" stops working without the pull — argues for a pointer from [`OPERATIONS.md`](./OPERATIONS.md), not for a second copy of another host's tooling. Still the maintainer's call, but it is now a recommendation rather than a pair of equal options. + - **The same reasoning points at the home-automation configuration repository for anything that is purely this host's**, since that is where the rest of the backup host's service configuration already lives. The pull is here instead because [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is what stops working without it. Revisit if a second unrelated host service ends up here. - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. - **What `robots.txt` says, which is undecided and is the last non-mechanical item before M7b.** Recorded under "Next" above, where it blocks the cutover. diff --git a/ops/README.md b/ops/README.md new file mode 100644 index 0000000..8b508ed --- /dev/null +++ b/ops/README.md @@ -0,0 +1,79 @@ +# ops + +Tooling that runs on the **backup host**, not on the web server and not in CI. One thing lives here today: the pull that copies the VPS's backup set and its access logs off the VPS. + +| File | Installs to | +| --- | --- | +| `vps-backup-pull` | `/usr/local/bin/vps-backup-pull` | +| `vps-backup-pull.service` | `/etc/systemd/system/` | +| `vps-backup-pull.timer` | `/etc/systemd/system/` | +| `vps-backup-pull.env.example` | `/etc/vps-backup-pull.env` | +| `vps-backup-pull.service.d-local.conf.example` | `/etc/systemd/system/vps-backup-pull.service.d/local.conf` | + +**The last two are required, not optional.** Nothing in this directory names a machine, because this repository is public, so the address, the destination paths, and the account the pull runs as are all supplied at install time. A missing value stops the run with the name of what is missing rather than falling back to something plausible, since a wrong-but-valid destination is a backup nobody can find. + +## What it does + +Three legs, each skippable, in one direction only: + +| Leg | From the VPS | To the backup host | +| --- | --- | --- | +| archives | the encrypted backup set | `DEST` | +| host config | the same non-secret files in plaintext | `DEST/hostconfig` | +| access logs | the rotated edge logs | `LOG_DEST` | + +**It is a pull rather than a push, and that is a security property rather than a convenience.** The backup host holds a key the VPS trusts, and the VPS holds no credential reaching anything else. A push would have to invert that, so a compromise of the web server would reach the backups that exist to survive it. + +**The plaintext host-config leg exists so a rebuild does not need the encryption key.** That key is not on the backup host and must never be put there, because beside the ciphertext it would make the encryption decorative. The file list is fetched from the VPS rather than duplicated here, so it tracks the host instead of drifting from a copy. + +**The log leg is the one with a deadline.** Those logs are deliberately excluded from the encrypted archives, because that set is many full copies with no dedupe and an append-only file would be multiplied across all of them for no recovery benefit. So the VPS's own retention window is the only copy until this runs. + +## Three behaviors that look like bugs and are not + +- **`--delete` never reaches the log leg**, whatever is passed. It exists to mirror the VPS's archive window, and applying it to an append-only record would delete the only remaining copy at exactly the moment it became the only one. The option array is copied before `--delete` is appended, rather than filtered afterwards, because a filter is a thing to get wrong later. +- **Today's live log is never fetched.** It is still being appended to, so a copy is a torn prefix that the next run fetches again. Rotation is what makes a file eligible. Read the live file over SSH when the analysis covers today. +- **Nothing prunes the destination.** Because the log leg passes no `--delete`, anything the VPS renames or re-compresses after it has been pulled keeps its old name on the backup host permanently, and a count that walks the tree by filename double-counts the overlap. **Read a date from a line's `StartUTC`, never from the filename holding it.** The VPS keeps an append-only `RECONCILE.md` inside the archive directory so a rename travels with the data it explains. + +## Install + +```sh +sudo install -m 755 ops/vps-backup-pull /usr/local/bin/vps-backup-pull +sudo install -m 644 ops/vps-backup-pull.service ops/vps-backup-pull.timer /etc/systemd/system/ + +sudo install -m 600 ops/vps-backup-pull.env.example /etc/vps-backup-pull.env +sudoedit /etc/vps-backup-pull.env + +sudo mkdir -p /etc/systemd/system/vps-backup-pull.service.d +sudo install -m 644 ops/vps-backup-pull.service.d-local.conf.example \ + /etc/systemd/system/vps-backup-pull.service.d/local.conf +sudoedit /etc/systemd/system/vps-backup-pull.service.d/local.conf + +sudo systemctl daemon-reload +sudo systemctl enable --now vps-backup-pull.timer +systemctl list-timers vps-backup-pull.timer +``` + +**Check the install before waiting a day to find out.** `vps-backup-pull --dry-run` exercises the SSH and rsync legs, lists what the next run would transfer, and writes nothing. + +**The timer's hour sits behind both producers on the VPS rather than beside them**, because the VPS rotates its log and writes its archive at times of its own. Pulling before the day's archive exists fetches the previous one and reports success, which is the failure mode that looks like a working backup. `Persistent=true` covers a host that was powered off when the timer should have fired. + +**Run it as the account that owns the destination, never under `sudo`.** It authenticates with that user's SSH key and writes into a tree that user owns. Under `sudo` it uses root's identity, which the VPS does not trust, and starts mixing root-owned files into a user-owned backup tree. That is why it installs to `bin/` rather than `sbin/`, why the drop-in sets `User=`, and why the script refuses to start as root rather than warning about it. + +## Checking it, without trusting anything written down + +```sh +journalctl -u vps-backup-pull.service -o short-iso | grep done +systemctl list-timers vps-backup-pull.timer --all +``` + +**A journal with one entry is not evidence of one copy.** The script can be run directly as well as by its timer, and a direct run writes no service record. Directory mtimes on the backup host are the copy times, where the file mtimes are the VPS's, so those are what to read when establishing when something arrived. + +## Variables + +Every path is a variable, so a host states its own layout rather than editing a file git owns. They are listed with their meanings in [`vps-backup-pull.env.example`](./vps-backup-pull.env.example), and the two `systemd` settings that cannot come from an environment file are in [`vps-backup-pull.service.d-local.conf.example`](./vps-backup-pull.service.d-local.conf.example). + +**Two of them are named a second time in Blog's own `secrets/` file, and the pairs must agree.** This side writes, and the log review reads. `DEST` here is `BACKUP_ARCHIVE_ROOT` there, and `LOG_DEST` here is `LOG_ARCHIVE_ROOT` there. The correspondence, and why the two vocabularies exist rather than one, is in [`OPERATIONS.md`](../OPERATIONS.md) "Working With the VPS". + +## This directory is the source + +Edit the copy here and install it. Nothing else is a source, and a copy found on a host is an installed artifact rather than a place to make a change. diff --git a/ops/vps-backup-pull b/ops/vps-backup-pull new file mode 100755 index 0000000..d604a49 --- /dev/null +++ b/ops/vps-backup-pull @@ -0,0 +1,285 @@ +#!/usr/bin/env bash +# vps-pull.sh -- pull the Pangolin VPS backup set to this host. +# +# RUNS ON THE BACKUP HOST, NOT on the VPS. It installs as +# /usr/local/bin/vps-backup-pull, which is the path vps-backup-pull.service runs +# and the only path that matters. Installing to /usr/local/sbin/ instead leaves the +# timer running the previous copy and reports no error while doing it. +# +# sudo install -m 755 ops/vps-backup-pull /usr/local/bin/vps-backup-pull +# +# RUN IT AS YOUR NORMAL USER, NOT WITH SUDO. It authenticates to the VPS with that +# user's SSH key and writes to a tree that user owns. Under sudo it uses root's +# identity, which the VPS does not trust, and starts mixing root-owned files into a +# user-owned backup tree. That is why it lives in bin/ rather than sbin/, and why +# the unit sets User=. Same applies when scheduling it: use the user's crontab or a +# user systemd timer, not root's. +# +# The source is ops/vps-backup-pull in the Blog repository. Edit it there and install +# it. A copy found on a host is an installed artifact, not a place to make a change. +# +# Paths below are read from /etc/vps-backup-pull.env, so a host states its own layout +# there rather than editing this file. See ops/vps-backup-pull.env.example. +# +# Why a PULL and not a push from the VPS: it keeps the VPS free of any credential +# reaching another system, which is the one security rule in CONFIG.md item 0. +# This host already has an SSH key trusted by the VPS; the VPS has none for here. +# +# What it fetches: +# pangolin/ the encrypted archives -- the actual backup. Contains config/, +# docker-compose.yml, hostconfig/ and MANIFEST.txt. +# hostconfig/ a PLAINTEXT copy of the same non-secret host files, so the +# rebuild surface stays readable WITHOUT the encryption key. The +# file list is fetched from the VPS (pangolin-backup --list-files) +# rather than duplicated here, so it cannot drift. +# logs the rotated access logs, which are deliberately NOT in the +# encrypted archives -- see section 4 for why, and why this leg +# never mirrors deletions even under --delete. +# +# The encryption key is NOT fetched and must never live here -- beside the +# ciphertext it would make the encryption decorative. It is in 1Password. +# +# Every path is an environment variable, read from /etc/vps-backup-pull.env, so a host +# states its own layout there rather than editing a file git owns. The three naming THIS +# pair of machines carry no default at all, because an address and a destination belong to +# one host and this repository is public: +# VPS_HOST (required) where to pull from +# DEST (required) archives + hostconfig land here +# LOG_DEST (required) both log sets land here +# REMOTE_ARCHIVE_DIR /var/backups/pangolin the archives ON THE VPS +# REMOTE_TRAEFIK_LOG_DIR /var/log/traefik/archive rotated edge access logs ON THE VPS +# REMOTE_BLOG_LOG_DIR /var/log/blog/legacy one-off Caddy container dumps ON THE VPS +# +# Usage: vps-backup-pull [--delete] [--no-hostconfig] [--no-logs] [--dry-run] [--quiet] +# --delete mirror deletions (Proxmox tracks the VPS's 14-archive window +# instead of growing without limit). Safe only because +# duplicacy keeps snapshot history in B2 -- without that, a +# deletion on the VPS would propagate irreversibly. It applies +# to the archives and host config ONLY, never to the logs. +# --no-hostconfig archives only +# --no-logs skip the access logs +# --dry-run show what would transfer, change nothing +# --quiet errors and the summary only +set -euo pipefail + +# The three values naming THIS pair of machines have no defaults, deliberately. They are +# an address and two paths belonging to one host, and secrets/ is gitignored precisely so +# that nothing of that kind reaches a public history. They come from +# /etc/vps-backup-pull.env, and preflight refuses to run without them rather than falling +# back to something plausible -- a wrong-but-valid destination is a backup nobody can find. +VPS_HOST=${VPS_HOST:-} +DEST=${DEST:-} +# Layout on the VPS, which is the same for any host running this stack, so these do default. +REMOTE_ARCHIVE_DIR=${REMOTE_ARCHIVE_DIR:-/var/backups/pangolin} +SSH_OPTS=${SSH_OPTS:--o ConnectTimeout=15 -o BatchMode=yes} + +# The log leg. Separate from DEST because these are plaintext and long-lived: the +# VPS prunes at 400 days and this host is the copy that outlives that window, so +# they must not share a directory whose retention tracks the VPS's 14 archives. +REMOTE_TRAEFIK_LOG_DIR=${REMOTE_TRAEFIK_LOG_DIR:-/var/log/traefik/archive} +REMOTE_BLOG_LOG_DIR=${REMOTE_BLOG_LOG_DIR:-/var/log/blog/legacy} +LOG_DEST=${LOG_DEST:-} + +DELETE=0; HOSTCONFIG=1; LOGS=1; DRYRUN=0; QUIET=0 +LOGFILES=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --delete) DELETE=1; shift ;; + --no-hostconfig) HOSTCONFIG=0; shift ;; + --no-logs) LOGS=0; shift ;; + --dry-run) DRYRUN=1; shift ;; + --quiet) QUIET=1; shift ;; + -h|--help) + awk 'NR>1 && /^#/{sub(/^# ?/,""); print; next} NR>1{exit}' "$0"; exit 0 ;; + *) echo "unknown argument: $1" >&2; exit 2 ;; + esac +done + +say() { [[ $QUIET -eq 1 ]] || printf '%s\n' "$*"; } +warn() { printf 'WARNING: %s\n' "$*" >&2; } +die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } + +# The closing summary, which prints even under --quiet. It went through say() before, so --quiet +# silenced it too and delivered neither half of the "errors and the summary only" this script's +# own usage promises. That matters most exactly where --quiet is used: under the timer, where a +# successful run then left a journal saying it started and finished and nothing about whether it +# copied anything, which is a backup with no evidence. +tell() { printf '%s\n' "$*"; } + +# Declared up front and cleaned up by ONE trap. Both are conditional -- --dry-run +# skips the checksum step, so a trap referencing an unset $SUMS would fail under +# `set -u` at exit and mask the real exit status. +SUMS=""; LIST="" +cleanup() { [[ -n $SUMS ]] && rm -f "$SUMS"; [[ -n $LIST ]] && rm -f "$LIST"; return 0; } +trap cleanup EXIT + +RSYNC_OPTS=(-a --human-readable) +[[ $QUIET -eq 1 ]] || RSYNC_OPTS+=(--info=stats1) +# --itemize-changes rides with --dry-run because otherwise a dry run prints byte +# counts and no filenames, which proves the SSH and rsync legs work and previews +# nothing. "What would tomorrow's timed run bring?" is the only question a dry run +# is asked, and stats1 alone cannot answer it. +[[ $DRYRUN -eq 1 ]] && RSYNC_OPTS+=(--dry-run --itemize-changes) +# Taken before --delete is appended rather than filtered out afterwards, because a +# filter is a thing to get wrong later and this is a copy of two words. +LOG_RSYNC_OPTS=("${RSYNC_OPTS[@]}") +[[ $DELETE -eq 1 ]] && RSYNC_OPTS+=(--delete) + +# ---------------------------------------------------------------- preflight +# Ahead of the banner deliberately. A banner printed with empty values reads, in a journal +# skim, exactly like a run that started and then failed somewhere real, so nothing that +# looks like progress is printed until the configuration is known good. +# +# Each variable is named individually rather than reported as "configuration missing", +# which sends you to the wrong file when only one line of it is absent. LOG_DEST is +# required only when the log leg is running, so --no-logs still works on a host that +# copies archives and nothing else. +# +# Root is refused rather than warned about. It authenticates with an identity the VPS does +# not trust, so the run fails regardless, but it fails after creating root-owned +# directories inside a tree the real account owns, and the next ordinary run then fails on +# those instead. The unit omits User= deliberately, so this is what catches a missing +# drop-in rather than the failure surfacing a day later as a permissions error. +[[ $EUID -ne 0 ]] || die "refusing to run as root -- run as the account that owns the destination and holds the VPS key (systemd: set User= in the drop-in)" +[[ -n $VPS_HOST ]] || die "VPS_HOST is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" +[[ -n $DEST ]] || die "DEST is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" +[[ $LOGS -eq 0 || -n $LOG_DEST ]] || die "LOG_DEST is not set and the log leg is enabled -- set it, or pass --no-logs" +command -v rsync >/dev/null || die "rsync not installed on this host" + +START=$(date -u '+%Y-%m-%d %H:%M:%S UTC') +say "=== VPS backup pull -- $START" +say " source: $VPS_HOST dest: $DEST" +[[ $LOGS -eq 1 ]] && say " logs: $LOG_DEST" +[[ $DRYRUN -eq 1 ]] && say " DRY RUN -- nothing will be written" +# shellcheck disable=SC2086 +ssh $SSH_OPTS "$VPS_HOST" true 2>/dev/null \ + || die "cannot reach $VPS_HOST over SSH (key auth only -- password auth is disabled there)" + +if [[ $DRYRUN -eq 0 ]]; then + mkdir -p "$DEST/pangolin" || die "cannot create $DEST/pangolin" + chmod 700 "$DEST" +fi + +# ---------------------------------------------------------------- 1. archives +say +say "--- archives" +# shellcheck disable=SC2086 +rsync "${RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ + "$VPS_HOST:$REMOTE_ARCHIVE_DIR/" "$DEST/pangolin/" \ + || die "archive rsync failed" + +# ---------------------------------------------------------------- 2. verify +# rsync verifies its own transfers, but this re-reads what actually landed on +# disk. A backup that was never independently checked is a hope, not a backup. +VERIFY_RC=0 +# Reported in the closing summary, so a scheduled run records how many archives were +# re-read from disk rather than only that it finished. Zero on a dry run, which prints +# its own summary and never reaches that line. +VERIFIED=0 +if [[ $DRYRUN -eq 1 ]]; then + say " (dry run -- checksum verification skipped)" +else + say + say "--- verifying checksums" + SUMS=$(mktemp) + # shellcheck disable=SC2086 + ssh $SSH_OPTS "$VPS_HOST" "cd $REMOTE_ARCHIVE_DIR && sha256sum *.enc" >"$SUMS" \ + || die "could not read source checksums" + EXPECTED=$(wc -l <"$SUMS") + if OUTPUT=$(cd "$DEST/pangolin" && sha256sum -c "$SUMS" 2>&1); then + VERIFIED=$EXPECTED + say " $EXPECTED/$EXPECTED archives verified OK" + else + VERIFY_RC=1 + printf '%s\n' "$OUTPUT" | grep -v ': OK$' >&2 || true + warn "checksum verification FAILED -- the local copy does not match the VPS" + fi +fi + +# ---------------------------------------------------------------- 3. hostconfig +if [[ $HOSTCONFIG -eq 1 ]]; then + say + say "--- host config (plaintext, readable without the key)" + LIST=$(mktemp) + # Ask the VPS what it considers valuable. Single source of truth: the list is + # defined once, in pangolin-backup's HOST_FILES, and never copied to this host. + # shellcheck disable=SC2086 + if ssh $SSH_OPTS "$VPS_HOST" 'pangolin-backup --list-files' 2>/dev/null \ + | sed 's|^/||' >"$LIST" && [[ -s $LIST ]]; then + WANT=$(wc -l <"$LIST") + say " $WANT path(s) advertised by the VPS" + [[ $DRYRUN -eq 0 ]] && mkdir -p "$DEST/hostconfig" + # shellcheck disable=SC2086 + rsync "${RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ + --files-from="$LIST" "$VPS_HOST:/" "$DEST/hostconfig/" \ + || warn "host config rsync reported errors" + if [[ $DRYRUN -eq 0 ]]; then + GOT=$(find "$DEST/hostconfig" -type f | wc -l) + say " $GOT file(s) on disk" + [[ $GOT -eq $WANT ]] || warn "expected $WANT file(s), found $GOT -- a listed path may be missing on the VPS" + fi + else + warn "could not get the file list from the VPS (old pangolin-backup without --list-files?) -- skipping host config" + fi +fi + +# ---------------------------------------------------------------- 4. access logs +# Deliberately NOT in the encrypted archives: pangolin-backup keeps fourteen full +# copies, each encrypted with a fresh salt so Backblaze cannot dedupe them, and an +# append-only file that grows forever would be multiplied by fourteen for no +# recovery benefit. So this is the ONLY copy that outlives the VPS's 400 days. +# +# NEVER --delete here, whatever was passed. The VPS prunes at 400 days by design, +# and this host is the long-term copy; mirroring that prune would delete the only +# remaining copy at exactly the moment it became the only one. Both sources are +# immutable once written, so the transfer is genuinely incremental either way. +# +# Today's live access.log is NOT fetched. It is still being appended to, so a copy +# is a torn prefix that the next run would fetch again; the rotation at 00:00 UTC +# is what makes a file eligible. Read the live file over SSH when analysing today. +if [[ $LOGS -eq 1 ]]; then + say + say "--- access logs (plaintext, and the only copy past 400 days)" + if [[ $DRYRUN -eq 0 ]]; then + mkdir -p "$LOG_DEST/traefik" "$LOG_DEST/blog-legacy" || die "cannot create $LOG_DEST" + # Query strings are logged in full, so the tree is as sensitive as the log is. + chmod 700 "$LOG_DEST" + fi + # shellcheck disable=SC2086 + rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ + "$VPS_HOST:$REMOTE_TRAEFIK_LOG_DIR/" "$LOG_DEST/traefik/" \ + || warn "traefik log rsync reported errors" + # shellcheck disable=SC2086 + rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ + "$VPS_HOST:$REMOTE_BLOG_LOG_DIR/" "$LOG_DEST/blog-legacy/" \ + || warn "blog legacy log rsync reported errors" + if [[ $DRYRUN -eq 0 ]]; then + LOGFILES=$(find "$LOG_DEST" -type f | wc -l) + say " $LOGFILES log file(s) on disk" + # A pull that lands nothing looks identical to a pull with nothing new, and the + # difference is a broken path against a working one. Only the empty case is odd. + [[ $LOGFILES -gt 0 ]] || warn "no log files landed -- check REMOTE_TRAEFIK_LOG_DIR and REMOTE_BLOG_LOG_DIR" + fi +fi + +# ---------------------------------------------------------------- summary +say +if [[ $DRYRUN -eq 1 ]]; then + say "=== dry run complete -- nothing changed" + exit 0 +fi +ARCHIVES=$(find "$DEST/pangolin" -name '*.enc' -type f | wc -l) +SIZE=$(du -sh "$DEST" 2>/dev/null | cut -f1) +if [[ $LOGS -eq 1 ]]; then + LOGSIZE=$(du -sh "$LOG_DEST" 2>/dev/null | cut -f1) + tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $DEST; $LOGFILES log file(s), $LOGSIZE in $LOG_DEST" +else + tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $DEST; logs skipped" +fi +if [[ $VERIFY_RC -ne 0 ]]; then + tell "=== COMPLETED WITH ERRORS -- see warnings above" + exit 1 +fi +say " Restore needs the key from 1Password; it is deliberately not stored here." +exit 0 diff --git a/ops/vps-backup-pull.env.example b/ops/vps-backup-pull.env.example new file mode 100644 index 0000000..ef4ee5f --- /dev/null +++ b/ops/vps-backup-pull.env.example @@ -0,0 +1,39 @@ +# Copy to /etc/vps-backup-pull.env on the backup host and set for that host. +# The unit reads it with EnvironmentFile=-, so the leading - makes it optional: +# a host running the layout below needs no file at all and the script's defaults stand. +# +# This file configures the WRITING side, the pull itself. The reading side is +# Blog's own secrets/..env, which names the same two +# destinations so the log review can find what the pull produced. The pairs must +# agree, and OPERATIONS.md "Working With the VPS" states which pairs those are. +# +# systemd parses this file itself rather than passing it to a shell. There is no +# expansion, no command substitution, and no quoting to strip, so write a bare +# value and never wrap it in quotes -- the quotes become part of the path. + +# Where to pull from. The administrative login, not the confined deploy account. +VPS_HOST=root@vps.example.com + +# On the VPS: the encrypted archives, written by its own backup timer. +REMOTE_ARCHIVE_DIR=/var/backups/pangolin + +# On the VPS: the rotated edge access logs. Today's live file is deliberately not +# among them, because rotation is what makes a file eligible to be copied. +REMOTE_TRAEFIK_LOG_DIR=/var/log/traefik/archive + +# On the VPS: one-off Caddy container dumps kept from before rotation existed. +REMOTE_BLOG_LOG_DIR=/var/log/blog/legacy + +# On the backup host: the archives and the plaintext hostconfig tree beside them. +# Blog's secrets file names this same directory as BACKUP_ARCHIVE_ROOT. +DEST=/path/to/backup/vps + +# On the backup host: both log sets. Separate from DEST, and mode 700, because these +# are plaintext and long-lived where the archives are encrypted and pruned, and +# because query strings are logged in full so the tree is as sensitive as the log. +# Blog's secrets file names this same directory as LOG_ARCHIVE_ROOT. +LOG_DEST=/path/to/backup/vps-logs + +# Key auth only; the VPS has password auth disabled. BatchMode makes an unusable key +# fail immediately instead of hanging a timed run on a password prompt nobody sees. +SSH_OPTS=-o ConnectTimeout=15 -o BatchMode=yes diff --git a/ops/vps-backup-pull.service b/ops/vps-backup-pull.service new file mode 100644 index 0000000..015059f --- /dev/null +++ b/ops/vps-backup-pull.service @@ -0,0 +1,43 @@ +[Unit] +# Pulls the VPS's encrypted archives, its plaintext host config, and the rotated +# access logs. The logs are the reason this has a deadline rather than being a +# convenience: they are deliberately excluded from the encrypted archives, so the +# VPS's own retention window is the only copy until this lands. +Description=Pull the Pangolin VPS backup set and access logs to this host +Documentation=file:/usr/local/bin/vps-backup-pull +Documentation=https://github.com/ptr727/Blog/blob/main/ops/README.md +Wants=network-online.target +After=network-online.target + +[Service] +Type=oneshot +# The paths, the address, and the account are all host-specific, and this file is public, +# so none of them is written here. Two mechanisms supply them and both are required: +# +# /etc/vps-backup-pull.env the paths and the address +# /etc/systemd/system/vps-backup-pull.service.d/*.conf User=, Group=, RequiresMountsFor= +# +# Templates for both are in ops/, and ops/README.md has the install steps. The leading - +# below keeps systemd from failing the unit outright when the env file is absent, because +# the script's own preflight names the missing variable, which is the more useful error. +EnvironmentFile=-/etc/vps-backup-pull.env +# NOT set here, deliberately. A username belongs to one machine, and a wrong guess baked +# into a public file is worse than an absent one: the script refuses to run as root rather +# than authenticating with an identity the VPS does not trust and writing root-owned files +# into a user-owned tree. Set User= and Group= in the drop-in. +# +# RequiresMountsFor= belongs in the drop-in for the same reason, and it is not optional on +# a host whose destination is a mount: a run starting before the mount writes a full copy +# into the mountpoint underneath it, where nothing ever reads it and the space does not +# show up in du against the mounted path. +# +# --quiet prints errors and the one-line summary only, which is what belongs in a journal. +# Add --delete to mirror the VPS's archive window instead of growing without limit. It is +# deliberately not set: it is safe only where snapshot history exists off this host, and it +# never touches the logs whatever is passed. +ExecStart=/usr/local/bin/vps-backup-pull --quiet +# The first run transfers the whole archive set. +TimeoutStartSec=2h +# A backup pull is never the urgent thing on this host. +Nice=10 +IOSchedulingClass=idle diff --git a/ops/vps-backup-pull.service.d-local.conf.example b/ops/vps-backup-pull.service.d-local.conf.example new file mode 100644 index 0000000..cb5842c --- /dev/null +++ b/ops/vps-backup-pull.service.d-local.conf.example @@ -0,0 +1,19 @@ +# Copy to /etc/systemd/system/vps-backup-pull.service.d/local.conf on the backup host. +# The three settings here name one machine, which is why the unit in git does not carry them. +# +# systemd merges drop-ins over the unit, so a later install of the unit leaves this intact. + +[Unit] +# Required when the destination is on a mounted filesystem, which it usually is. +# A run that starts before the mount writes a full copy into the mountpoint underneath it, +# where nothing ever reads it and the space does not show up in du against the mounted path. +# Name the mount point, not the destination directory below it. +RequiresMountsFor=/path/to/mount + +[Service] +# The account whose SSH key the VPS trusts and that owns the destination tree. +# Not optional: without it the unit runs as root, and the script refuses to start, because +# root authenticates with an identity the VPS does not trust and writes root-owned files +# into a user-owned tree. +User=someuser +Group=somegroup diff --git a/ops/vps-backup-pull.timer b/ops/vps-backup-pull.timer new file mode 100644 index 0000000..43e70ae --- /dev/null +++ b/ops/vps-backup-pull.timer @@ -0,0 +1,19 @@ +[Unit] +Description=Daily VPS backup and access-log pull +Documentation=file:/usr/local/bin/vps-backup-pull + +[Timer] +# 09:00 UTC, chosen to sit behind both producers on the VPS rather than beside them: +# logrotate rotates the access log at 00:00 UTC, and pangolin-backup.timer writes the +# day's encrypted archive at 08:03 UTC. Running before the archive exists would pull +# yesterday's and report success, which is the failure that looks like a working backup. +# Stated in UTC explicitly, because this host runs local time and the VPS does not. +OnCalendar=*-*-* 09:00:00 UTC +# The VPS is a single small host; nothing here needs to hit it on the second. +RandomizedDelaySec=15m +# Runs on next boot if the host was down at 09:00. Without this a machine that is off +# overnight silently never backs up, and the gap is only visible by reading timestamps. +Persistent=true + +[Install] +WantedBy=timers.target From 8e738fe1cb86aa808c15373b6b587a4d8ba889f9 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 10:25:53 -0700 Subject: [PATCH 06/14] Derive the pull's host values at install time instead of asking for them The previous commit left three values to be set by hand in /etc, having just recorded the same three in secrets/local.production.env under the names the log review uses. That is the drift OPERATIONS.md warns about, created by the change that warned about it, and it made installing the maintainer's problem for no reason. ops/install.sh derives all of it: VPS_SSH_HOST, BACKUP_ARCHIVE_ROOT and LOG_ARCHIVE_ROOT map to VPS_HOST, DEST and LOG_DEST; the account is whoever runs it; the group is read from the destination; the mount comes from findmnt. Generating one vocabulary from the other is also what stops the pair from disagreeing later. Two derivations where the obvious answer is wrong, both caught by running it rather than by reading it: - The group must come from the destination, not `id -gn`. Here those are users and pieter, and Group= sets the process's primary group, so the obvious answer creates files the existing tree's group cannot read -- and only where no setgid bit already masks it, so it half works. - RequiresMountsFor= needs the mount point, not the destination below it. --check needs no root, prints both generated files, and verifies SSH. It reports "needs root to compare" rather than "already correct" when it cannot read an existing file, since an installer claiming agreement it never checked is the failure this is written against. Verified: --check output matches the running configuration exactly, and the real run stops at the first sudo having written nothing, leaving the working script installed and tomorrow's timer armed. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 4 +- ops/README.md | 25 ++++--- ops/install.sh | 174 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 187 insertions(+), 16 deletions(-) create mode 100755 ops/install.sh diff --git a/TODO.md b/TODO.md index 540f60e..f08980a 100644 --- a/TODO.md +++ b/TODO.md @@ -56,7 +56,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Read a date from a line's `StartUTC` rather than from the filename holding it.** That is the durable form of the lesson, and it is in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" as well. - **An off-host copy also predates the timer.** The directory mtimes on the backup host are copy times where the file mtimes are the VPS's, and they show a pull at 2026-08-08 03:31 UTC that the service journal has no record of, because the script was run directly rather than through `systemd`. So a journal with one entry is not evidence of one copy. The whole set was audited both ways afterwards and nothing else had diverged: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no `--delete`. - **Both halves were exercised 2026-08-08 between 16:18 and 16:22 UTC, deliberately without running the service.** `systemctl start` would have written the second journal entry that 2026-08-09 is supposed to prove, so the transport was exercised with `--dry-run` instead and the timer was read rather than triggered: `NEXT` is 2026-08-09 09:08:45 UTC inside the 15-minute randomized window, `LAST` is `-`, and the journal still holds exactly one entry. **If a second entry exists before that time, someone ran it by hand and the schedule is still unproven.** The dry run reached the VPS over SSH and all three legs planned cleanly. Pending for the first timed run: five encrypted archives dated 2026-08-08, plus `RECONCILE.md` **and** `MANIFEST.txt`, so the log-file count rises by two non-log files rather than the one recorded above. No rotated access log is pending, which is correct, because the 00:00 UTC rotation that produces `access.log-2026-08-08` has not happened yet. - - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is committed at [`ops/vps-backup-pull`](./ops/vps-backup-pull) and is not installed**: writing `/usr/local/bin/vps-backup-pull` needs root and was refused, so the running copy still lacks it. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. + - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is committed at [`ops/vps-backup-pull`](./ops/vps-backup-pull) and is not installed**, so the running copy still lacks it until `ops/install.sh` is run. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. - **The VPS's older copy of the script is not a source, and reconciling the two is [#53][issue-53].** The committed copy carries the whole access-log leg, the `tell()` fix and the `VERIFIED` counter. The VPS's carries an install block and a no-sudo rationale that this side lacked, now folded in. Neither direction is a safe overwrite, so #53 is a merge rather than a copy, and copying the VPS's over the committed one would delete the log pull that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" runs on. - **The header's own install command pointed at `/usr/local/sbin/vps-backup-pull`, which nothing runs.** `vps-backup-pull.service` runs `/usr/local/bin/vps-backup-pull` and `/usr/local/sbin/` is empty, so following the instruction would have written a second copy nobody executes while `scp` and `chmod` both reported success and the timer went on running the old one. The canonical had already corrected this to a `sudo install` into `bin/`, and states the reason `bin/` is deliberate. **The stale block is replaced in the patch copy** with the canonical's wording plus an explicit refusal to run that `scp` until the divergence above is reconciled. It is another reason a plain overwrite in either direction is the wrong merge. - **The outward pass ran end to end on 2026-08-08 traffic and found nothing to add.** 9,996 edge requests, 9,285 of them this repo's own deploy gate. Of the 711 that remain, 101 were site-host 404s across 73 distinct paths, and every one was a scanner shape. Only `/` and `/robots.txt` intersect the URL contract at all, and both are explained rather than open: `/robots.txt` 404ed until the 15:47:17 deploy and has answered 200 since 15:48:45, and `/` 404ed twice at the edge, below. **No legacy content URL 404ed, so `checks/golden-urls.txt` needs no addition from this run**, which is the expected result and is recorded because an unrecorded clean pass is indistinguishable from a pass nobody ran. @@ -77,7 +77,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions -- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. What remains for the pull is mechanical: install the committed copy over `/usr/local/bin/vps-backup-pull`, which needs root, and reconcile [#53][issue-53] with the VPS's older copy in both directions rather than by overwriting either. +- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **One step remains and it is one command:** `ops/install.sh`, run as the account that owns the backup. It derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env` and installs everything, so nothing is typed twice and the two vocabularies cannot drift. `ops/install.sh --check` proves the derivation without root and writes nothing, and it has been run: every value matches the running configuration. The install itself needs a password this session could not supply. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. - **Where the rest of the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy were an operational asset built from another agent's instructions, and they existed only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. - **The pull itself is resolved and the reasoning is kept because it applies to everything still listed here.** What made it urgent was measured: the copy protected everywhere was the VPS's older one, while the copy that actually ran, carrying the log leg the review depends on, was in no snapshot and no repository. Committing it is what closed that, not the backup host's own off-site copy, which never reached the script. - **The directory holding it is named as though it were disposable.** `~/vps-backup-pull-patch` reads as a patch staged against a source, and there is no source: it is the most complete copy of the script in existence. A directory named for a temporary artifact is the one a cleanup deletes, and nothing here would notice until a restore produced the wrong script. diff --git a/ops/README.md b/ops/README.md index 8b508ed..051f771 100644 --- a/ops/README.md +++ b/ops/README.md @@ -4,13 +4,14 @@ Tooling that runs on the **backup host**, not on the web server and not in CI. O | File | Installs to | | --- | --- | +| `install.sh` | nothing, it does the installing | | `vps-backup-pull` | `/usr/local/bin/vps-backup-pull` | | `vps-backup-pull.service` | `/etc/systemd/system/` | | `vps-backup-pull.timer` | `/etc/systemd/system/` | | `vps-backup-pull.env.example` | `/etc/vps-backup-pull.env` | | `vps-backup-pull.service.d-local.conf.example` | `/etc/systemd/system/vps-backup-pull.service.d/local.conf` | -**The last two are required, not optional.** Nothing in this directory names a machine, because this repository is public, so the address, the destination paths, and the account the pull runs as are all supplied at install time. A missing value stops the run with the name of what is missing rather than falling back to something plausible, since a wrong-but-valid destination is a backup nobody can find. +**The last two are required, not optional, and `install.sh` generates both.** Nothing in this directory names a machine, so the address, the destination paths, and the account are supplied at install time from values this repository already holds. A missing value stops the pull with the name of what is missing rather than falling back to something plausible, since a wrong-but-valid destination is a backup nobody can find. The two `.example` files document the format and are not the install path. ## What it does @@ -37,23 +38,19 @@ Three legs, each skippable, in one direction only: ## Install ```sh -sudo install -m 755 ops/vps-backup-pull /usr/local/bin/vps-backup-pull -sudo install -m 644 ops/vps-backup-pull.service ops/vps-backup-pull.timer /etc/systemd/system/ +ops/install.sh --check # derive, validate, print, write nothing +ops/install.sh # the same, then install +``` -sudo install -m 600 ops/vps-backup-pull.env.example /etc/vps-backup-pull.env -sudoedit /etc/vps-backup-pull.env +**Nothing is typed twice.** The address, both destinations, and the account are already known to this checkout, so `install.sh` derives them rather than asking: `VPS_SSH_HOST`, `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT` come from `secrets/..env` under the names the log review uses, the account is whoever runs the script, the group is read from the destination, and the mount is resolved with `findmnt`. That is also what keeps the two vocabularies from drifting, since one is generated from the other rather than maintained beside it. -sudo mkdir -p /etc/systemd/system/vps-backup-pull.service.d -sudo install -m 644 ops/vps-backup-pull.service.d-local.conf.example \ - /etc/systemd/system/vps-backup-pull.service.d/local.conf -sudoedit /etc/systemd/system/vps-backup-pull.service.d/local.conf +**Run it as the account that will own the backup, not under `sudo`.** It refuses to start as root, because that account's name and its SSH key are what the unit is built around. It calls `sudo` itself for the steps that need it, so expect one password prompt. -sudo systemctl daemon-reload -sudo systemctl enable --now vps-backup-pull.timer -systemctl list-timers vps-backup-pull.timer -``` +**Two derivations are worth knowing, because the obvious answer is wrong for both.** The group comes from the destination rather than from `id -gn`, since `Group=` sets the process's primary group and the account's own group is usually not the one owning the backup tree. And `RequiresMountsFor=` needs the mount point rather than the destination path below it. + +**`--check` needs no root and writes nothing.** It prints both generated files, reports whether each would be created or already matches, and proves the VPS answers over SSH. Run it first. It reports "needs root to compare" rather than "already correct" when it cannot read an existing file, because an installer that claims agreement it could not verify is the failure this is written against. -**Check the install before waiting a day to find out.** `vps-backup-pull --dry-run` exercises the SSH and rsync legs, lists what the next run would transfer, and writes nothing. +Re-running is safe and is how a changed value is applied. A file that already matches is reported and left alone, and one that differs stops the run until `--force`. **The timer's hour sits behind both producers on the VPS rather than beside them**, because the VPS rotates its log and writes its archive at times of its own. Pulling before the day's archive exists fetches the previous one and reports success, which is the failure mode that looks like a working backup. `Persistent=true` covers a host that was powered off when the timer should have fired. diff --git a/ops/install.sh b/ops/install.sh new file mode 100755 index 0000000..12bdf47 --- /dev/null +++ b/ops/install.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +# Install the backup pull on this host, deriving every host-specific value from the +# environment file this repository already keeps. +# +# The pull needs three things this repository deliberately does not carry: an address, two +# destination paths, and the account to run as. All three are already known here -- +# secrets/..env holds the first three under the names the log review +# uses, and the account is whoever runs this. So they are derived rather than retyped, and +# the two vocabularies OPERATIONS.md describes cannot drift, because one is generated from +# the other. +# +# VPS_SSH_HOST -> VPS_HOST +# BACKUP_ARCHIVE_ROOT -> DEST +# LOG_ARCHIVE_ROOT -> LOG_DEST +# VPS_TRAEFIK_LOG_ARCHIVE -> REMOTE_TRAEFIK_LOG_DIR +# +# Usage: ops/install.sh [--check] [--force] +# --check validate and print what would be written, touch nothing, need no root +# --force overwrite an existing config file whose contents differ +# +# RUN IT AS THE ACCOUNT THAT WILL OWN THE BACKUP, not under sudo. That account's name and +# its SSH key are what the unit is built around, and running this under sudo would record +# root. Individual privileged steps call sudo themselves. +set -euo pipefail + +CHECK=0; FORCE=0 +while [[ $# -gt 0 ]]; do + case "$1" in + --check) CHECK=1; shift ;; + --force) FORCE=1; shift ;; + -h|--help) sed -n '2,22p' "$0"; exit 0 ;; + *) printf 'ERROR: unknown argument %s\n' "$1" >&2; exit 1 ;; + esac +done + +die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } +note() { printf ' %s\n' "$*"; } + +REPO=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +ENV_FILE=${ENV_FILE:-$REPO/secrets/local.production.env} + +[[ $EUID -ne 0 ]] || die "do not run this under sudo -- run it as the account that will own the backup; it calls sudo for the steps that need it" +[[ -f $ENV_FILE ]] || die "$ENV_FILE does not exist (ENV_FILE overrides which file is read)" + +# Sourced the same way every other script here reads it, so a value set by hand in the +# caller's shell does not quietly win over the file that is supposed to be authoritative. +set -a +# shellcheck disable=SC1090 +. "$ENV_FILE" +set +a + +[[ -n ${VPS_SSH_HOST:-} ]] || die "VPS_SSH_HOST is not set in $ENV_FILE" +[[ -n ${BACKUP_ARCHIVE_ROOT:-} ]] || die "BACKUP_ARCHIVE_ROOT is not set in $ENV_FILE" +[[ -n ${LOG_ARCHIVE_ROOT:-} ]] || die "LOG_ARCHIVE_ROOT is not set in $ENV_FILE" + +ACCOUNT=$(id -un) + +# The nearest existing ancestor of the destination, which is what both the group and the +# mount are read from. The destination itself may not exist on a first install. +ANCESTOR=$BACKUP_ARCHIVE_ROOT +while [[ ! -e $ANCESTOR && $ANCESTOR != / ]]; do ANCESTOR=$(dirname "$ANCESTOR"); done + +# The group comes from the destination rather than from `id -gn`, which is the wrong answer +# whenever the account's primary group is not the group that owns the backup tree. Group= +# sets the process's primary group, so getting it from the user would create files the +# existing tree's group cannot read, and only on the paths where a setgid bit does not +# already override it -- so it would half work, which is worse than failing. +GROUP=$(stat -c %G "$ANCESTOR") + +# The mount the destination sits on, resolved rather than guessed, because RequiresMountsFor +# has to name the mount point and not the directory below it. +MOUNT=$(findmnt -no TARGET --target "$ANCESTOR" 2>/dev/null) \ + || die "cannot resolve the mount holding $BACKUP_ARCHIVE_ROOT" + +REMOTE_TRAEFIK=${VPS_TRAEFIK_LOG_ARCHIVE:-/var/log/traefik/archive} + +printf '=== derived from %s\n' "${ENV_FILE#"$REPO"/}" +note "VPS_HOST $VPS_SSH_HOST" +note "DEST $BACKUP_ARCHIVE_ROOT" +note "LOG_DEST $LOG_ARCHIVE_ROOT" +note "remote logs $REMOTE_TRAEFIK" +note "account $ACCOUNT:$GROUP" +note "mount $MOUNT" + +ENV_DEST=/etc/vps-backup-pull.env +DROPIN_DIR=/etc/systemd/system/vps-backup-pull.service.d +DROPIN_DEST=$DROPIN_DIR/local.conf + +ENV_BODY=$(cat </dev/null; then + printf '=== %s: exists, contents need root to compare\n' "$path"; changed=1 + elif printf '%s\n' "$body" | sudo -n diff -q - "$path" >/dev/null 2>&1; then + printf '=== %s: already correct\n' "$path" + else + printf '=== %s: differs\n' "$path" + printf '%s\n' "$body" | sudo -n diff -u "$path" - || true + [[ $FORCE -eq 1 ]] || die "$path exists with different contents -- re-run with --force to replace it" + changed=1 + fi +done + +if [[ $CHECK -eq 1 ]]; then + printf '\n=== %s would contain\n' "$ENV_DEST" + printf '%s\n' "$ENV_BODY" | sed 's/^/ /' + printf '\n=== %s would contain\n' "$DROPIN_DEST" + printf '%s\n' "$DROPIN_BODY" | sed 's/^/ /' + printf '\n=== check only -- nothing written\n' + printf '=== verifying the VPS is reachable as %s\n' "$ACCOUNT" + # shellcheck disable=SC2086 + ssh ${SSH_OPTS:--o ConnectTimeout=15 -o BatchMode=yes} "$VPS_SSH_HOST" true \ + && printf ' reachable\n' \ + || die "cannot reach $VPS_SSH_HOST over SSH as $ACCOUNT (key auth only)" + exit 0 +fi + +printf '=== installing\n' +sudo install -m 755 "$REPO/ops/vps-backup-pull" /usr/local/bin/vps-backup-pull +sudo install -m 644 "$REPO/ops/vps-backup-pull.service" "$REPO/ops/vps-backup-pull.timer" \ + /etc/systemd/system/ +printf '%s\n' "$ENV_BODY" | sudo install -m 600 /dev/stdin "$ENV_DEST" +sudo mkdir -p "$DROPIN_DIR" +printf '%s\n' "$DROPIN_BODY" | sudo install -m 644 /dev/stdin "$DROPIN_DEST" +sudo systemctl daemon-reload +sudo systemctl enable --now vps-backup-pull.timer + +printf '=== verifying the installed copy, without writing a journal entry\n' +# --dry-run rather than starting the service, because a service record is the evidence that +# the timer fired, and manufacturing one here would spend that evidence to prove the install. +/usr/local/bin/vps-backup-pull --dry-run >/dev/null \ + || die "the installed script failed its dry run -- $ENV_DEST or the drop-in is wrong" +printf ' dry run clean\n' +systemctl list-timers vps-backup-pull.timer --all --no-pager +if [[ $changed -eq 1 ]]; then + printf '=== done\n' +else + printf '=== done -- configuration was already correct\n' +fi From 188d418c1e4c047b31b871708003097241a97cef Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 10:30:32 -0700 Subject: [PATCH 07/14] Use one name per directory instead of two, and delete what held them apart The pull and the secrets file had grown separate names for the same four directories: VPS_HOST/VPS_SSH_HOST, DEST/BACKUP_ARCHIVE_ROOT, LOG_DEST/LOG_ARCHIVE_ROOT, and REMOTE_TRAEFIK_LOG_DIR against VPS_TRAEFIK_LOG_ARCHIVE. I introduced the second set rather than reusing the first, on the reasoning that DEST is too generic to sit beside DEPLOY_ROOT in a shared file. That bought a bare-name collision hazard and paid for it with a duplicate vocabulary. The cost compounded rather than staying put. Two names needed a pairing table in OPERATIONS.md to say which matched which, a paragraph on why neither side could source the other's file, and a translation step in the installer. All three were machinery for a problem that did not have to exist. The pull now uses the descriptive names throughout: VPS_* is something on the VPS, *_ROOT is something on the backup host. The table, the paragraph, and the mapping are deleted rather than maintained, and install.sh copies values instead of translating them. Verified: the pull runs directly off secrets/local.production.env with no translation at all, which is the property that was missing before; it still fails closed with the new names; install.sh --check derives every value correctly; and no old name survives anywhere in the tree. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 7 +-- ops/README.md | 16 +++-- ops/install.sh | 55 ++++++++--------- ops/vps-backup-pull | 102 ++++++++++++++++---------------- ops/vps-backup-pull.env.example | 44 +++++++------- 5 files changed, 108 insertions(+), 116 deletions(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index 34d5656..338631b 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -220,12 +220,7 @@ ssh "$VPS_SSH_HOST" true && echo reachable **It is a pull rather than a push, and nothing on the VPS knows it happens.** That direction is the security property rather than an implementation detail: the backup host holds a key the VPS trusts, and the VPS holds no credential reaching any other system, so a compromise of the web server cannot walk into the backups that exist to survive it. -**The two sides use different names for the same two directories, and the pairs must agree.** The pull is configured on the backup host and writes, where this repository's `secrets/` file is read by the log review. Neither can source the other's file, because one is a `systemd` `EnvironmentFile` on another machine and the other is a shell file here, so the correspondence is stated rather than enforced, and it is the thing to check first when a review finds an empty archive. - -| The pull writes | The review reads | Holding | -| --- | --- | --- | -| `DEST` | `BACKUP_ARCHIVE_ROOT` | the encrypted archives and the plaintext `hostconfig` tree | -| `LOG_DEST` | `LOG_ARCHIVE_ROOT` | both sets of rotated logs | +**Both sides use one set of names, so there is nothing to reconcile.** The pull writes `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT` and the log review reads the same two, spelled the same way, and [`ops/install.sh`](./ops/install.sh) generates the pull's `EnvironmentFile` from this repository's `secrets/` file by copying rather than translating. That is deliberate and was not the first attempt: the two sides briefly had separate names for the same four directories, which needed a mapping table in this file and a translation step in the installer, and both disappeared when the names were unified. A value that exists once cannot disagree with itself. ```sh set -a; . secrets/local.production.env; set +a diff --git a/ops/README.md b/ops/README.md index 051f771..602f02b 100644 --- a/ops/README.md +++ b/ops/README.md @@ -19,9 +19,9 @@ Three legs, each skippable, in one direction only: | Leg | From the VPS | To the backup host | | --- | --- | --- | -| archives | the encrypted backup set | `DEST` | -| host config | the same non-secret files in plaintext | `DEST/hostconfig` | -| access logs | the rotated edge logs | `LOG_DEST` | +| archives | the encrypted backup set | `BACKUP_ARCHIVE_ROOT` | +| host config | the same non-secret files in plaintext | `BACKUP_ARCHIVE_ROOT/hostconfig` | +| access logs | the rotated edge logs | `LOG_ARCHIVE_ROOT` | **It is a pull rather than a push, and that is a security property rather than a convenience.** The backup host holds a key the VPS trusts, and the VPS holds no credential reaching anything else. A push would have to invert that, so a compromise of the web server would reach the backups that exist to survive it. @@ -42,9 +42,7 @@ ops/install.sh --check # derive, validate, print, write nothing ops/install.sh # the same, then install ``` -**Nothing is typed twice.** The address, both destinations, and the account are already known to this checkout, so `install.sh` derives them rather than asking: `VPS_SSH_HOST`, `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT` come from `secrets/..env` under the names the log review uses, the account is whoever runs the script, the group is read from the destination, and the mount is resolved with `findmnt`. That is also what keeps the two vocabularies from drifting, since one is generated from the other rather than maintained beside it. - -**Run it as the account that will own the backup, not under `sudo`.** It refuses to start as root, because that account's name and its SSH key are what the unit is built around. It calls `sudo` itself for the steps that need it, so expect one password prompt. +**Nothing is typed twice.** The address, both destinations, and the account are already known to this checkout, so `install.sh` copies them rather than asking: `VPS_SSH_HOST`, `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT` come straight from `secrets/..env`, the account is whoever runs the script, the group is read from the destination, and the mount is resolved with `findmnt`. **Two derivations are worth knowing, because the obvious answer is wrong for both.** The group comes from the destination rather than from `id -gn`, since `Group=` sets the process's primary group and the account's own group is usually not the one owning the backup tree. And `RequiresMountsFor=` needs the mount point rather than the destination path below it. @@ -54,7 +52,7 @@ Re-running is safe and is how a changed value is applied. A file that already ma **The timer's hour sits behind both producers on the VPS rather than beside them**, because the VPS rotates its log and writes its archive at times of its own. Pulling before the day's archive exists fetches the previous one and reports success, which is the failure mode that looks like a working backup. `Persistent=true` covers a host that was powered off when the timer should have fired. -**Run it as the account that owns the destination, never under `sudo`.** It authenticates with that user's SSH key and writes into a tree that user owns. Under `sudo` it uses root's identity, which the VPS does not trust, and starts mixing root-owned files into a user-owned backup tree. That is why it installs to `bin/` rather than `sbin/`, why the drop-in sets `User=`, and why the script refuses to start as root rather than warning about it. +**Run both as the account that will own the backup, never under `sudo`**, and expect one password prompt for the privileged steps the installer calls itself. The pull authenticates with that account's SSH key and writes into a tree that account owns. Under `sudo` it uses root's identity, which the VPS does not trust, and starts mixing root-owned files into a user-owned backup tree. That is why it installs to `bin/` rather than `sbin/`, why the drop-in sets `User=`, and why both scripts refuse to start as root rather than warning about it. ## Checking it, without trusting anything written down @@ -67,9 +65,9 @@ systemctl list-timers vps-backup-pull.timer --all ## Variables -Every path is a variable, so a host states its own layout rather than editing a file git owns. They are listed with their meanings in [`vps-backup-pull.env.example`](./vps-backup-pull.env.example), and the two `systemd` settings that cannot come from an environment file are in [`vps-backup-pull.service.d-local.conf.example`](./vps-backup-pull.service.d-local.conf.example). +Every path is a variable, so a host states its own layout rather than editing a file git owns. They are listed with their meanings in [`vps-backup-pull.env.example`](./vps-backup-pull.env.example), and the three `systemd` settings that cannot come from an environment file are in [`vps-backup-pull.service.d-local.conf.example`](./vps-backup-pull.service.d-local.conf.example). -**Two of them are named a second time in Blog's own `secrets/` file, and the pairs must agree.** This side writes, and the log review reads. `DEST` here is `BACKUP_ARCHIVE_ROOT` there, and `LOG_DEST` here is `LOG_ARCHIVE_ROOT` there. The correspondence, and why the two vocabularies exist rather than one, is in [`OPERATIONS.md`](../OPERATIONS.md) "Working With the VPS". +**They are the same names Blog's own `secrets/` file uses, which is the point.** `VPS_*` is something on the VPS and `*_ROOT` is something on this host, and the pull writes the two roots that the log review reads. One name per directory means the writing side and the reading side cannot disagree, and it is why `install.sh` copies rather than translates. An earlier split into two vocabularies needed a mapping table to hold it together, and unifying them deleted the table. ## This directory is the source diff --git a/ops/install.sh b/ops/install.sh index 12bdf47..871a54b 100755 --- a/ops/install.sh +++ b/ops/install.sh @@ -2,17 +2,15 @@ # Install the backup pull on this host, deriving every host-specific value from the # environment file this repository already keeps. # -# The pull needs three things this repository deliberately does not carry: an address, two -# destination paths, and the account to run as. All three are already known here -- -# secrets/..env holds the first three under the names the log review -# uses, and the account is whoever runs this. So they are derived rather than retyped, and -# the two vocabularies OPERATIONS.md describes cannot drift, because one is generated from -# the other. +# The pull needs four things this repository deliberately does not carry: an address, two +# destination paths, and the account to run as. The first three are already in +# secrets/..env under the same names the pull itself uses, so this +# copies them rather than translating them, and the account is whoever runs this. # -# VPS_SSH_HOST -> VPS_HOST -# BACKUP_ARCHIVE_ROOT -> DEST -# LOG_ARCHIVE_ROOT -> LOG_DEST -# VPS_TRAEFIK_LOG_ARCHIVE -> REMOTE_TRAEFIK_LOG_DIR +# There is deliberately no name mapping here. An earlier version of this file had one, +# because the pull and the secrets file had grown separate names for the same directories, +# and a table translating four pairs is a thing to get wrong every time one side changes. +# The names were unified instead, which deleted the table rather than maintaining it. # # Usage: ops/install.sh [--check] [--force] # --check validate and print what would be written, touch nothing, need no root @@ -72,15 +70,15 @@ GROUP=$(stat -c %G "$ANCESTOR") MOUNT=$(findmnt -no TARGET --target "$ANCESTOR" 2>/dev/null) \ || die "cannot resolve the mount holding $BACKUP_ARCHIVE_ROOT" -REMOTE_TRAEFIK=${VPS_TRAEFIK_LOG_ARCHIVE:-/var/log/traefik/archive} +VPS_TRAEFIK_ARCHIVE=${VPS_TRAEFIK_LOG_ARCHIVE:-/var/log/traefik/archive} printf '=== derived from %s\n' "${ENV_FILE#"$REPO"/}" -note "VPS_HOST $VPS_SSH_HOST" -note "DEST $BACKUP_ARCHIVE_ROOT" -note "LOG_DEST $LOG_ARCHIVE_ROOT" -note "remote logs $REMOTE_TRAEFIK" -note "account $ACCOUNT:$GROUP" -note "mount $MOUNT" +note "VPS_SSH_HOST $VPS_SSH_HOST" +note "BACKUP_ARCHIVE_ROOT $BACKUP_ARCHIVE_ROOT" +note "LOG_ARCHIVE_ROOT $LOG_ARCHIVE_ROOT" +note "VPS_TRAEFIK_LOG_ARC. $VPS_TRAEFIK_ARCHIVE" +note "account $ACCOUNT:$GROUP" +note "mount $MOUNT" ENV_DEST=/etc/vps-backup-pull.env DROPIN_DIR=/etc/systemd/system/vps-backup-pull.service.d @@ -88,13 +86,13 @@ DROPIN_DEST=$DROPIN_DIR/local.conf ENV_BODY=$(cat <..env uses, so there is one vocabulary rather than two. +# +# VPS_SSH_HOST (required) where to pull from +# BACKUP_ARCHIVE_ROOT (required) archives + hostconfig land here +# LOG_ARCHIVE_ROOT (required) both log sets land here +# VPS_ARCHIVE_DIR /var/backups/pangolin the archives ON THE VPS +# VPS_TRAEFIK_LOG_ARCHIVE /var/log/traefik/archive rotated edge access logs ON THE VPS +# VPS_BLOG_LOG_DIR /var/log/blog/legacy one-off Caddy dumps ON THE VPS +# +# The three marked required carry no default, because an address and a destination belong +# to one host. A wrong-but-valid destination is a backup nobody can find. # # Usage: vps-backup-pull [--delete] [--no-hostconfig] [--no-logs] [--dry-run] [--quiet] # --delete mirror deletions (Proxmox tracks the VPS's 14-archive window @@ -62,23 +66,21 @@ # --quiet errors and the summary only set -euo pipefail -# The three values naming THIS pair of machines have no defaults, deliberately. They are -# an address and two paths belonging to one host, and secrets/ is gitignored precisely so -# that nothing of that kind reaches a public history. They come from -# /etc/vps-backup-pull.env, and preflight refuses to run without them rather than falling -# back to something plausible -- a wrong-but-valid destination is a backup nobody can find. -VPS_HOST=${VPS_HOST:-} -DEST=${DEST:-} +# No defaults, deliberately: preflight refuses to run without them rather than falling back +# to something plausible. They come from /etc/vps-backup-pull.env, which ops/install.sh +# generates from the same names in secrets/..env. +VPS_SSH_HOST=${VPS_SSH_HOST:-} +BACKUP_ARCHIVE_ROOT=${BACKUP_ARCHIVE_ROOT:-} # Layout on the VPS, which is the same for any host running this stack, so these do default. -REMOTE_ARCHIVE_DIR=${REMOTE_ARCHIVE_DIR:-/var/backups/pangolin} +VPS_ARCHIVE_DIR=${VPS_ARCHIVE_DIR:-/var/backups/pangolin} SSH_OPTS=${SSH_OPTS:--o ConnectTimeout=15 -o BatchMode=yes} -# The log leg. Separate from DEST because these are plaintext and long-lived: the +# The log leg. Separate from BACKUP_ARCHIVE_ROOT because these are plaintext and long-lived: # VPS prunes at 400 days and this host is the copy that outlives that window, so # they must not share a directory whose retention tracks the VPS's 14 archives. -REMOTE_TRAEFIK_LOG_DIR=${REMOTE_TRAEFIK_LOG_DIR:-/var/log/traefik/archive} -REMOTE_BLOG_LOG_DIR=${REMOTE_BLOG_LOG_DIR:-/var/log/blog/legacy} -LOG_DEST=${LOG_DEST:-} +VPS_TRAEFIK_LOG_ARCHIVE=${VPS_TRAEFIK_LOG_ARCHIVE:-/var/log/traefik/archive} +VPS_BLOG_LOG_DIR=${VPS_BLOG_LOG_DIR:-/var/log/blog/legacy} +LOG_ARCHIVE_ROOT=${LOG_ARCHIVE_ROOT:-} DELETE=0; HOSTCONFIG=1; LOGS=1; DRYRUN=0; QUIET=0 LOGFILES=0 @@ -132,7 +134,7 @@ LOG_RSYNC_OPTS=("${RSYNC_OPTS[@]}") # looks like progress is printed until the configuration is known good. # # Each variable is named individually rather than reported as "configuration missing", -# which sends you to the wrong file when only one line of it is absent. LOG_DEST is +# which sends you to the wrong file when only one line of it is absent. LOG_ARCHIVE_ROOT is # required only when the log leg is running, so --no-logs still works on a host that # copies archives and nothing else. # @@ -142,23 +144,23 @@ LOG_RSYNC_OPTS=("${RSYNC_OPTS[@]}") # those instead. The unit omits User= deliberately, so this is what catches a missing # drop-in rather than the failure surfacing a day later as a permissions error. [[ $EUID -ne 0 ]] || die "refusing to run as root -- run as the account that owns the destination and holds the VPS key (systemd: set User= in the drop-in)" -[[ -n $VPS_HOST ]] || die "VPS_HOST is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" -[[ -n $DEST ]] || die "DEST is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" -[[ $LOGS -eq 0 || -n $LOG_DEST ]] || die "LOG_DEST is not set and the log leg is enabled -- set it, or pass --no-logs" +[[ -n $VPS_SSH_HOST ]] || die "VPS_SSH_HOST is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" +[[ -n $BACKUP_ARCHIVE_ROOT ]] || die "BACKUP_ARCHIVE_ROOT is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" +[[ $LOGS -eq 0 || -n $LOG_ARCHIVE_ROOT ]] || die "LOG_ARCHIVE_ROOT is not set and the log leg is enabled -- set it, or pass --no-logs" command -v rsync >/dev/null || die "rsync not installed on this host" START=$(date -u '+%Y-%m-%d %H:%M:%S UTC') say "=== VPS backup pull -- $START" -say " source: $VPS_HOST dest: $DEST" -[[ $LOGS -eq 1 ]] && say " logs: $LOG_DEST" +say " source: $VPS_SSH_HOST dest: $BACKUP_ARCHIVE_ROOT" +[[ $LOGS -eq 1 ]] && say " logs: $LOG_ARCHIVE_ROOT" [[ $DRYRUN -eq 1 ]] && say " DRY RUN -- nothing will be written" # shellcheck disable=SC2086 -ssh $SSH_OPTS "$VPS_HOST" true 2>/dev/null \ - || die "cannot reach $VPS_HOST over SSH (key auth only -- password auth is disabled there)" +ssh $SSH_OPTS "$VPS_SSH_HOST" true 2>/dev/null \ + || die "cannot reach $VPS_SSH_HOST over SSH (key auth only -- password auth is disabled there)" if [[ $DRYRUN -eq 0 ]]; then - mkdir -p "$DEST/pangolin" || die "cannot create $DEST/pangolin" - chmod 700 "$DEST" + mkdir -p "$BACKUP_ARCHIVE_ROOT/pangolin" || die "cannot create $BACKUP_ARCHIVE_ROOT/pangolin" + chmod 700 "$BACKUP_ARCHIVE_ROOT" fi # ---------------------------------------------------------------- 1. archives @@ -166,7 +168,7 @@ say say "--- archives" # shellcheck disable=SC2086 rsync "${RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - "$VPS_HOST:$REMOTE_ARCHIVE_DIR/" "$DEST/pangolin/" \ + "$VPS_SSH_HOST:$VPS_ARCHIVE_DIR/" "$BACKUP_ARCHIVE_ROOT/pangolin/" \ || die "archive rsync failed" # ---------------------------------------------------------------- 2. verify @@ -184,10 +186,10 @@ else say "--- verifying checksums" SUMS=$(mktemp) # shellcheck disable=SC2086 - ssh $SSH_OPTS "$VPS_HOST" "cd $REMOTE_ARCHIVE_DIR && sha256sum *.enc" >"$SUMS" \ + ssh $SSH_OPTS "$VPS_SSH_HOST" "cd $VPS_ARCHIVE_DIR && sha256sum *.enc" >"$SUMS" \ || die "could not read source checksums" EXPECTED=$(wc -l <"$SUMS") - if OUTPUT=$(cd "$DEST/pangolin" && sha256sum -c "$SUMS" 2>&1); then + if OUTPUT=$(cd "$BACKUP_ARCHIVE_ROOT/pangolin" && sha256sum -c "$SUMS" 2>&1); then VERIFIED=$EXPECTED say " $EXPECTED/$EXPECTED archives verified OK" else @@ -205,17 +207,17 @@ if [[ $HOSTCONFIG -eq 1 ]]; then # Ask the VPS what it considers valuable. Single source of truth: the list is # defined once, in pangolin-backup's HOST_FILES, and never copied to this host. # shellcheck disable=SC2086 - if ssh $SSH_OPTS "$VPS_HOST" 'pangolin-backup --list-files' 2>/dev/null \ + if ssh $SSH_OPTS "$VPS_SSH_HOST" 'pangolin-backup --list-files' 2>/dev/null \ | sed 's|^/||' >"$LIST" && [[ -s $LIST ]]; then WANT=$(wc -l <"$LIST") say " $WANT path(s) advertised by the VPS" - [[ $DRYRUN -eq 0 ]] && mkdir -p "$DEST/hostconfig" + [[ $DRYRUN -eq 0 ]] && mkdir -p "$BACKUP_ARCHIVE_ROOT/hostconfig" # shellcheck disable=SC2086 rsync "${RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - --files-from="$LIST" "$VPS_HOST:/" "$DEST/hostconfig/" \ + --files-from="$LIST" "$VPS_SSH_HOST:/" "$BACKUP_ARCHIVE_ROOT/hostconfig/" \ || warn "host config rsync reported errors" if [[ $DRYRUN -eq 0 ]]; then - GOT=$(find "$DEST/hostconfig" -type f | wc -l) + GOT=$(find "$BACKUP_ARCHIVE_ROOT/hostconfig" -type f | wc -l) say " $GOT file(s) on disk" [[ $GOT -eq $WANT ]] || warn "expected $WANT file(s), found $GOT -- a listed path may be missing on the VPS" fi @@ -242,24 +244,24 @@ if [[ $LOGS -eq 1 ]]; then say say "--- access logs (plaintext, and the only copy past 400 days)" if [[ $DRYRUN -eq 0 ]]; then - mkdir -p "$LOG_DEST/traefik" "$LOG_DEST/blog-legacy" || die "cannot create $LOG_DEST" + mkdir -p "$LOG_ARCHIVE_ROOT/traefik" "$LOG_ARCHIVE_ROOT/blog-legacy" || die "cannot create $LOG_ARCHIVE_ROOT" # Query strings are logged in full, so the tree is as sensitive as the log is. - chmod 700 "$LOG_DEST" + chmod 700 "$LOG_ARCHIVE_ROOT" fi # shellcheck disable=SC2086 rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - "$VPS_HOST:$REMOTE_TRAEFIK_LOG_DIR/" "$LOG_DEST/traefik/" \ + "$VPS_SSH_HOST:$VPS_TRAEFIK_LOG_ARCHIVE/" "$LOG_ARCHIVE_ROOT/traefik/" \ || warn "traefik log rsync reported errors" # shellcheck disable=SC2086 rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - "$VPS_HOST:$REMOTE_BLOG_LOG_DIR/" "$LOG_DEST/blog-legacy/" \ + "$VPS_SSH_HOST:$VPS_BLOG_LOG_DIR/" "$LOG_ARCHIVE_ROOT/blog-legacy/" \ || warn "blog legacy log rsync reported errors" if [[ $DRYRUN -eq 0 ]]; then - LOGFILES=$(find "$LOG_DEST" -type f | wc -l) + LOGFILES=$(find "$LOG_ARCHIVE_ROOT" -type f | wc -l) say " $LOGFILES log file(s) on disk" # A pull that lands nothing looks identical to a pull with nothing new, and the # difference is a broken path against a working one. Only the empty case is odd. - [[ $LOGFILES -gt 0 ]] || warn "no log files landed -- check REMOTE_TRAEFIK_LOG_DIR and REMOTE_BLOG_LOG_DIR" + [[ $LOGFILES -gt 0 ]] || warn "no log files landed -- check VPS_TRAEFIK_LOG_ARCHIVE and VPS_BLOG_LOG_DIR" fi fi @@ -269,13 +271,13 @@ if [[ $DRYRUN -eq 1 ]]; then say "=== dry run complete -- nothing changed" exit 0 fi -ARCHIVES=$(find "$DEST/pangolin" -name '*.enc' -type f | wc -l) -SIZE=$(du -sh "$DEST" 2>/dev/null | cut -f1) +ARCHIVES=$(find "$BACKUP_ARCHIVE_ROOT/pangolin" -name '*.enc' -type f | wc -l) +SIZE=$(du -sh "$BACKUP_ARCHIVE_ROOT" 2>/dev/null | cut -f1) if [[ $LOGS -eq 1 ]]; then - LOGSIZE=$(du -sh "$LOG_DEST" 2>/dev/null | cut -f1) - tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $DEST; $LOGFILES log file(s), $LOGSIZE in $LOG_DEST" + LOGSIZE=$(du -sh "$LOG_ARCHIVE_ROOT" 2>/dev/null | cut -f1) + tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; $LOGFILES log file(s), $LOGSIZE in $LOG_ARCHIVE_ROOT" else - tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $DEST; logs skipped" + tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; logs skipped" fi if [[ $VERIFY_RC -ne 0 ]]; then tell "=== COMPLETED WITH ERRORS -- see warnings above" diff --git a/ops/vps-backup-pull.env.example b/ops/vps-backup-pull.env.example index ef4ee5f..fcd8372 100644 --- a/ops/vps-backup-pull.env.example +++ b/ops/vps-backup-pull.env.example @@ -1,39 +1,35 @@ -# Copy to /etc/vps-backup-pull.env on the backup host and set for that host. -# The unit reads it with EnvironmentFile=-, so the leading - makes it optional: -# a host running the layout below needs no file at all and the script's defaults stand. +# Copy to /etc/vps-backup-pull.env on the backup host, or let ops/install.sh generate it. # -# This file configures the WRITING side, the pull itself. The reading side is -# Blog's own secrets/..env, which names the same two -# destinations so the log review can find what the pull produced. The pairs must -# agree, and OPERATIONS.md "Working With the VPS" states which pairs those are. +# These are the same variable names secrets/..env uses, deliberately. +# One name per thing: VPS_* is something on the VPS, *_ROOT is something on this host. +# The pull writes those two roots and the log review reads them, so a single vocabulary is +# what keeps the two sides pointing at the same directories with nothing to reconcile. # -# systemd parses this file itself rather than passing it to a shell. There is no -# expansion, no command substitution, and no quoting to strip, so write a bare -# value and never wrap it in quotes -- the quotes become part of the path. +# systemd parses this file itself rather than passing it to a shell. There is no expansion, +# no command substitution, and no quoting to strip, so write a bare value and never wrap it +# in quotes, or the quotes become part of the path. # Where to pull from. The administrative login, not the confined deploy account. -VPS_HOST=root@vps.example.com +VPS_SSH_HOST=root@vps.example.com # On the VPS: the encrypted archives, written by its own backup timer. -REMOTE_ARCHIVE_DIR=/var/backups/pangolin +VPS_ARCHIVE_DIR=/var/backups/pangolin -# On the VPS: the rotated edge access logs. Today's live file is deliberately not -# among them, because rotation is what makes a file eligible to be copied. -REMOTE_TRAEFIK_LOG_DIR=/var/log/traefik/archive +# On the VPS: the rotated edge access logs. Today's live file is deliberately not among +# them, because rotation is what makes a file eligible to be copied. +VPS_TRAEFIK_LOG_ARCHIVE=/var/log/traefik/archive # On the VPS: one-off Caddy container dumps kept from before rotation existed. -REMOTE_BLOG_LOG_DIR=/var/log/blog/legacy +VPS_BLOG_LOG_DIR=/var/log/blog/legacy -# On the backup host: the archives and the plaintext hostconfig tree beside them. -# Blog's secrets file names this same directory as BACKUP_ARCHIVE_ROOT. -DEST=/path/to/backup/vps +# On this host: the archives and the plaintext hostconfig tree beside them. +BACKUP_ARCHIVE_ROOT=/path/to/backup/vps -# On the backup host: both log sets. Separate from DEST, and mode 700, because these -# are plaintext and long-lived where the archives are encrypted and pruned, and +# On this host: both log sets. Separate from BACKUP_ARCHIVE_ROOT, and mode 700, because +# these are plaintext and long-lived where the archives are encrypted and pruned, and # because query strings are logged in full so the tree is as sensitive as the log. -# Blog's secrets file names this same directory as LOG_ARCHIVE_ROOT. -LOG_DEST=/path/to/backup/vps-logs +LOG_ARCHIVE_ROOT=/path/to/backup/vps-logs -# Key auth only; the VPS has password auth disabled. BatchMode makes an unusable key +# Key auth only, since the VPS has password auth disabled. BatchMode makes an unusable key # fail immediately instead of hanging a timed run on a password prompt nobody sees. SSH_OPTS=-o ConnectTimeout=15 -o BatchMode=yes From 3712bc767f4570a2f6c08230b7af8b0cd351cada Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 10:41:30 -0700 Subject: [PATCH 08/14] Describe every configuration value once, in ENVIRONMENT.md, and gate it The descriptions were spread across two .example files, three documents and a script header, so a value could be described twice and disagree with itself, and a new one could be added with no description at all. ENVIRONMENT.md is now the single description of all 28 values, grouped by where they live, because that determines who can change one, what happens when it is wrong, and whether it reaches a public history. The .example files stated the format and the scripts state the defaults, and both now point here rather than restating. The two templates collapse into one example.env at the root, covering both destinations in marked sections. `.gitignore` needed a negation for it: `*.env` matched the template, which is what the old `.example` suffixes had been dodging. Anchored, so it exempts only the root file and a real environment file anywhere is still ignored. checks/check-env-docs.py runs in CI and fails both directions: a value declared in the template, in a workflow's vars/secrets, or listed as a per-invocation knob and having no row, and a row describing something declared nowhere, which is the shape a rename leaves behind. Tested by breaking it each way rather than by trusting a clean first run. Two bugs found while writing it: - SSH_OPTS was unquoted and contains spaces. Harmless while only systemd read it; example.env is also shell-sourced, where it ran everything after the first space as a command. - The comment claiming systemd cannot strip quotes was wrong. Measured with systemd-run: it strips them. Quoting is correct for both readers. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/validate-task.yml | 6 ++ .gitignore | 11 ++- ENVIRONMENT.md | 103 +++++++++++++++++++++++++ OPERATIONS.md | 4 +- README.md | 8 +- checks/check-env-docs.py | 113 ++++++++++++++++++++++++++++ deploy/README.md | 2 +- deploy/env.example | 96 ----------------------- example.env | 110 +++++++++++++++++++++++++++ ops/README.md | 6 +- ops/install.sh | 6 +- ops/vps-backup-pull | 21 ++---- ops/vps-backup-pull.env.example | 35 --------- 13 files changed, 357 insertions(+), 164 deletions(-) create mode 100644 ENVIRONMENT.md create mode 100755 checks/check-env-docs.py delete mode 100644 deploy/env.example create mode 100644 example.env delete mode 100644 ops/vps-backup-pull.env.example diff --git a/.github/workflows/validate-task.yml b/.github/workflows/validate-task.yml index b0506c5..0d36d4c 100644 --- a/.github/workflows/validate-task.yml +++ b/.github/workflows/validate-task.yml @@ -71,6 +71,12 @@ jobs: done python3 -c 'import yaml,sys; yaml.safe_load(open("hugo.yaml"))' + # Every configuration value is described once, in ENVIRONMENT.md. + # A new value gets added wherever its author is working, and nothing else notices a + # missing row. Runs both directions: undocumented values, and rows describing nothing. + - name: Check environment docs step + run: python3 checks/check-env-docs.py + # The pin lives in the action, so validation and the deploy cannot install different generators. - name: Install Hugo step uses: ./.github/actions/install-hugo diff --git a/.gitignore b/.gitignore index 8c5d34f..076afa5 100644 --- a/.gitignore +++ b/.gitignore @@ -30,13 +30,16 @@ __pycache__/ # Host-specific values: deploy roots, base URLs, container names, and uids. # Each names one particular machine rather than the project. # The whole directory is ignored so a value added later lands ignored by default. -# `deploy/env.example` is the committed template and sits outside the directory. -# The last pattern is the backstop for one written outside the directory, matching the -# `..env` shape those files are named for rather than a single literal name. -# `deploy/env.example` does not end in `.env`, so it is unaffected. +# `example.env` is the committed template and sits at the repository root. +# The `*.env` pattern is the backstop for a real environment file written outside the +# directory, matching the `..env` shape those files are named for +# rather than a single literal name. It also matches the template, so the template is +# negated on the line after it, anchored so it only exempts the one at the root. Order +# matters: a negation placed before its pattern does nothing. secrets/ **/secrets *.env +!/example.env # The working copies of the host channel described in OPERATIONS.md. # They carry server internals, and the host's own backup is what makes them durable. diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md new file mode 100644 index 0000000..6cc1584 --- /dev/null +++ b/ENVIRONMENT.md @@ -0,0 +1,103 @@ +# ENVIRONMENT.md + +Every configuration value this repository reads or writes, described once. [`OPERATIONS.md`](./OPERATIONS.md) is the procedure and this is the reference the procedure points at, so a value is explained here and named elsewhere. + +**A description lives here and nowhere else.** The `.example` files state the format, the scripts state the defaults, and this file states what a value means. [`checks/check-env-docs.py`](./checks/check-env-docs.py) fails if a value is declared or consumed anywhere and has no row below, or has a row and is declared nowhere, so the two stay in step without depending on anyone remembering. + +**Values are grouped by where they live rather than by what reads them**, because where a value lives determines who can change it, what happens when it is wrong, and whether it reaches a public history. + +## The mechanism + +A local run reads one file. `secrets/..env` is sourced with `set -a`, selected by `ENV_FILE`, and defaults to `secrets/local.production.env`. The whole `secrets/` directory is gitignored, so a value naming a machine never reaches the published history, and [`example.env`](./example.env) is the tracked template that documents the shape. + +Two consequences of `set -a` are worth stating because both have surprised someone. Sourcing overwrites a variable the caller exported first, so exporting `DEPLOY_ROOT` by hand does not switch environments and only `ENV_FILE` does. And a named file that does not exist is a hard failure rather than a fall-through, because on a host serving two sites the ambient value is the other site's root. + +CI reads no file. The deploy workflow resolves the same values from the GitHub Environment, which is why the shapes have to match even though the sources do not. + +## Repository environment files + +Held in `secrets/..env`, one file per environment. Template: [`example.env`](./example.env). + +| Value | Names | Notes | +| --- | --- | --- | +| `DEPLOY_ROOT` | where a release is written, and what the container mounts read-only at `/srv/blog` | The first argument to `make-release.sh` wins over it. | +| `HUGO_BASEURL` | the site base URL | Baked into the canonical tag, the feed links, and every absolute permalink. Must be set for anything that is not production, or a mirror serves pages pointing at production and every gate still passes. | +| `CADDY_APPDATA` | the container's persistent state root, deliberately outside `DEPLOY_ROOT` | Holds `config/` with the bootstrap Caddyfile and `data/` with Caddy state. A release writes neither. Nothing reads this value, so it is recorded to keep a rebuild from depending on memory. | +| `CADDY_CONTAINER` | the container serving this environment | A release needs no restart, because Caddy reloads in process. Restarting is the remedy when the watcher dies, which it does silently after one failed load. | +| `EXPECT_SITE_ENV` | the environment that must answer, compared against the `X-Blog-Env` header the bundle stamps | A proxy rule aimed at the wrong container returns a healthy 200 under the right hostname, so the check refuses to start rather than proving nothing. | +| `PANGOLIN_ACCESS_TOKEN_ID` | the resource access token's id, for an environment behind the auth gate | Set both or neither. Leave both unset for a site that is public. | +| `PANGOLIN_ACCESS_TOKEN` | the token itself | Read by `check-live-urls.sh`. Staging keeps its gate on because it serves a byte-identical copy of the public site. | +| `CAPTURE_ROOT` | the provenance capture, holding the WordPress exports, the crawl of the old platform, and the inventories derived from it | `checks/build-redirects.py` takes it as its one argument. Environment-independent, so it belongs in the default file only. Nothing sources it. | +| `VPS_SSH_HOST` | the VPS administrative login | Not the deploy account. See "Two credentials" below. Environment-independent. | +| `VPS_TRAEFIK_LOG` | today's live access log on the VPS, still being appended to | Never pulled, because rotation is what makes a file eligible. An analysis covering today reads it over SSH. Nothing sources it. | +| `VPS_TRAEFIK_LOG_ARCHIVE` | the rotated access logs on the VPS, and the source of the off-host copy | Also read by the pull, below. | +| `VPS_COMMS_DIR` | the two agent channel files on the VPS | Nothing sources it, and the transfer commands in `OPERATIONS.md` are spelled out rather than using it. See "The one place indirection is wrong" below. | +| `BACKUP_ARCHIVE_ROOT` | the off-host encrypted archives and the plaintext `hostconfig` tree beside them | Written by the pull, read by a rebuild. | +| `LOG_ARCHIVE_ROOT` | the off-host copy of the rotated logs | Written by the pull, read by the log review. | + +Three more are named in the template but commented out, because CI resolves them from the GitHub Environment and a local run deploys to a path and needs none of them: `DEPLOY_SSH_HOST`, `DEPLOY_SSH_USER`, `DEPLOY_SSH_KNOWN_HOSTS`. They are listed there so the local file and the environment describe the same shape. + +## The backup host + +Held in `/etc/vps-backup-pull.env`, read by `vps-backup-pull` through the unit's `EnvironmentFile`. Template: [`example.env`](./example.env). [`ops/install.sh`](./ops/install.sh) generates it by copying from the repository environment file, which is why the four shared names are spelled identically in both. + +| Value | Names | Notes | +| --- | --- | --- | +| `VPS_SSH_HOST` | where to pull from | Required. No default. | +| `BACKUP_ARCHIVE_ROOT` | where the archives and host config land | Required. No default. | +| `LOG_ARCHIVE_ROOT` | where both log sets land | Required unless `--no-logs`. No default. Mode 700, because query strings are logged in full. | +| `VPS_ARCHIVE_DIR` | the encrypted archives on the VPS | Defaults to the documented layout. | +| `VPS_TRAEFIK_LOG_ARCHIVE` | the rotated edge access logs on the VPS | Defaults to the documented layout. | +| `VPS_BLOG_LOG_DIR` | one-off Caddy container dumps on the VPS, kept from before rotation existed | Defaults to the documented layout. | +| `SSH_OPTS` | the SSH options the transfer uses | `BatchMode` makes an unusable key fail immediately rather than hanging a timed run on a password prompt nobody sees. | + +**The three marked required carry no default on purpose.** An address and a destination belong to one host, and a wrong-but-valid destination is a backup nobody can find, so the pull names what is missing and refuses to run rather than falling back to something plausible. + +**`systemd` parses this file itself rather than passing it to a shell**, so there is no expansion and no command substitution, and a `$` or a backtick is a literal character. It does strip matching quotes, verified rather than assumed, so a value containing spaces is quoted and arrives without them. That matters because [`example.env`](./example.env) is also sourced by a shell for the other destination, where an unquoted value would run everything after the first space as a command. + +## The GitHub Environments + +Held on the `production` and `staging` environments. The deploy workflow reads no file. + +| Value | Kind | Names | +| --- | --- | --- | +| `HUGO_BASEURL` | variable | the base URL, used twice: the site is built with it and `check-live-urls.sh` is pointed at it | +| `DEPLOY_SSH_HOST` | variable | the deploy endpoint | +| `DEPLOY_SSH_USER` | variable | the confined deploy account | +| `DEPLOY_SSH_KNOWN_HOSTS` | variable | the pinned host key. A variable rather than a secret, deliberately, since it is public by nature | +| `DEPLOY_SSH_PRIVATE_KEY` | secret | the deploy key, held behind an `rrsync` forced command | +| `PANGOLIN_ACCESS_TOKEN_ID` | secret | as above, for an environment behind the gate | +| `PANGOLIN_ACCESS_TOKEN` | secret | as above | + +**`HUGO_BASEURL` being read twice is the trap worth knowing.** A wrong value bakes the wrong address into every canonical tag and then runs the full URL contract against that same wrong address, so the deploy verifies itself and passes. + +**A host rebuild regenerates the SSH host keys and the pinned value stops matching**, which fails every deploy closed and blocks the rollback path at the same moment a rebuild makes both matter. Replace `DEPLOY_SSH_KNOWN_HOSTS` on **both** environments before the first deploy after a rebuild. + +Two repository-level secrets are unrelated to deployment and exist for the merge bot: `CODEGEN_APP_CLIENT_ID` and `CODEGEN_APP_PRIVATE_KEY`. + +## Per-invocation knobs + +Set on the command line for one run rather than stored anywhere. + +| Value | Effect | +| --- | --- | +| `ENV_FILE` | which environment file to source. Defaults to `secrets/local.production.env` | +| `REQUIRE_BROTLI=1` | fail rather than shipping gzip-only. CI sets it | +| `NO_LINK_DEST=1` | full copy instead of hard-linking from the previous release | +| `KEEP_RELEASES` | how many releases `make-release.sh` leaves behind | +| `EXPECT_RELEASE` | the release id `check-live-urls.sh` requires the live site to report, which is what makes a rollback verifiable rather than merely exiting zero | + +## Two credentials to the VPS, and why they are separate + +`DEPLOY_SSH_USER` reaches a confined account behind an `rrsync` forced command that can write one release tree and read nothing else. `VPS_SSH_HOST` is the ordinary administrative login used for reading logs, reading the archive directory, and moving the channel files. Reaching for the deploy account to read a log fails in a way that reads like an outage, and reaching for the admin account to deploy grants far more than the deploy needs. + +## The one place indirection is wrong + +The two channel transfers under [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides" spell out the host and directory rather than using `VPS_SSH_HOST` and `VPS_COMMS_DIR`. The permission allowlist matches the text of a command rather than what it expands to, so substituting the variables turns an allowed transfer into one that prompts, while looking like a tidy-up that changed nothing. The same rule is why neither may be chained behind `cd` or `&&`. + +## Rules + +- **One name per thing.** A value that appears on two sides is spelled identically on both, so neither side needs translating into the other. +- **No value naming a machine reaches git.** Not in a script default, not in a unit, not in a template. The `.example` files carry placeholders, and the real values live in `secrets/` or on the host. +- **A description belongs here and a reference belongs everywhere else.** A `.example` file says what the format is, and this file says what the value means. +- **Nothing sources some of these, and that is recorded rather than hidden.** A value kept only so a rebuild does not depend on memory is still worth holding, but a reader should not have to discover that no code reads it. diff --git a/OPERATIONS.md b/OPERATIONS.md index 338631b..d637fd5 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -153,7 +153,7 @@ HUGO_BASEURL= deploy/make-release.sh "$(git rev-parse -- checks/check-live-urls.sh ``` -The deploy root and the base URL are the only host-specific values. A local run reads them from an untracked file under `secrets/`, one per environment, copied from [`deploy/env.example`](./deploy/env.example), and CI passes both explicitly. The whole `secrets/` directory is gitignored, so no address, path, or container name belonging to one machine reaches the published history. +The deploy root and the base URL are the only host-specific values. A local run reads them from an untracked file under `secrets/`, one per environment, copied from [`example.env`](./example.env), and CI passes both explicitly. The whole `secrets/` directory is gitignored, so no address, path, or container name belonging to one machine reaches the published history. **Always set `HUGO_BASEURL` for anything that is not production.** The base URL is baked into the canonical tag, the feed links, and every absolute permalink, so a mirror built without it serves pages that all point back at the production address. Nothing downstream catches this, because the pages render at the right paths and the build gate passes. The effective value is printed on every build for that reason. @@ -220,7 +220,7 @@ ssh "$VPS_SSH_HOST" true && echo reachable **It is a pull rather than a push, and nothing on the VPS knows it happens.** That direction is the security property rather than an implementation detail: the backup host holds a key the VPS trusts, and the VPS holds no credential reaching any other system, so a compromise of the web server cannot walk into the backups that exist to survive it. -**Both sides use one set of names, so there is nothing to reconcile.** The pull writes `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT` and the log review reads the same two, spelled the same way, and [`ops/install.sh`](./ops/install.sh) generates the pull's `EnvironmentFile` from this repository's `secrets/` file by copying rather than translating. That is deliberate and was not the first attempt: the two sides briefly had separate names for the same four directories, which needed a mapping table in this file and a translation step in the installer, and both disappeared when the names were unified. A value that exists once cannot disagree with itself. +**Both sides use one set of names, so there is nothing to reconcile.** The pull writes `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT` and the log review reads the same two, spelled the same way, and [`ops/install.sh`](./ops/install.sh) generates the pull's `EnvironmentFile` from this repository's `secrets/` file by copying rather than translating. Every value is described once, in [`ENVIRONMENT.md`](./ENVIRONMENT.md), and [`checks/check-env-docs.py`](./checks/check-env-docs.py) fails if one is declared without a description or described without existing. ```sh set -a; . secrets/local.production.env; set +a diff --git a/README.md b/README.md index 1e46680..9450866 100644 --- a/README.md +++ b/README.md @@ -153,8 +153,9 @@ flowchart LR | [`checks/`][checks] | the URL contract and the gates that enforce it | | [`deploy/`][deploy] | the release script, the web-server config, and the redirect maps | | [`ops/`][ops] | the pull that copies the server's backups and access logs off it, and its schedule | +| [`ENVIRONMENT.md`][environment] | every configuration value, described once | -Deploy paths, environment variables, and the server layout are documented in [OPERATIONS.md][operations]. +Every configuration value is described in [ENVIRONMENT.md][environment]. The deploy procedure and the server layout are in [OPERATIONS.md][operations]. ## Questions or Issues @@ -180,7 +181,7 @@ deploy/make-release.sh checks/check-live-urls.sh "$HUGO_BASEURL" ``` -The deploy root and the base URL come from an untracked file per environment under `secrets/`, named `..env`, copied from [deploy/env.example][env-example] and selected with `ENV_FILE`. `secrets/local.production.env` is the one read when `ENV_FILE` is unset. The whole `secrets/` directory is gitignored, so host-specific values stay out of the published history. +The deploy root and the base URL come from an untracked file per environment under `secrets/`, named `..env`, copied from [example.env][env-example] and selected with `ENV_FILE`. `secrets/local.production.env` is the one read when `ENV_FILE` is unset. The whole `secrets/` directory is gitignored, so host-specific values stay out of the published history. ## 3rd Party Tools @@ -214,10 +215,11 @@ Licensed under the [MIT License][license]\ [commits-link]: https://github.com/ptr727/Blog/commits [deploy]: ./deploy/ [ops]: ./ops/ +[environment]: ./ENVIRONMENT.md [deploy-readme]: ./deploy/README.md [discussions-link]: https://github.com/ptr727/Blog/discussions [issues-link]: https://github.com/ptr727/Blog/issues -[env-example]: ./deploy/env.example +[env-example]: ./example.env [history]: ./HISTORY.md [hugo-config]: ./hugo.yaml [license]: ./LICENSE diff --git a/checks/check-env-docs.py b/checks/check-env-docs.py new file mode 100755 index 0000000..922fe92 --- /dev/null +++ b/checks/check-env-docs.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +"""Fail if a configuration value is declared or consumed anywhere but not described in +ENVIRONMENT.md, or described there and declared nowhere. + +ENVIRONMENT.md is the single description of every configuration value. A new value gets +added wherever its author is working, which is rarely the doc, and no linter notices a +missing paragraph. This does. + +Both directions matter and they catch different mistakes: + + undocumented a value exists and nobody wrote down what it means. + unused a value is described but declared nowhere. Usually a rename that updated + the code and left the prose, which is worse than an omission because it + reads as current. + +Read-only. Exit 1 on any finding. +""" +from __future__ import annotations + +import re +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +DOC = REPO / "ENVIRONMENT.md" + +# The one template, whose keys are the declared configuration surface. +TEMPLATES = [REPO / "example.env"] + +# Workflow references. `vars.X` and `secrets.X` are the GitHub Environment surface, and a +# value added there is exactly as undocumented as one added to a template. +WORKFLOWS = sorted((REPO / ".github" / "workflows").glob("*.yml")) + +# Set per invocation rather than stored, so they appear in no template and would otherwise +# be invisible to this check. Listed here because the doc has a table for them, and a knob +# nobody documented is the same failure as an undocumented file value. +KNOBS = {"ENV_FILE", "REQUIRE_BROTLI", "NO_LINK_DEST", "KEEP_RELEASES", "EXPECT_RELEASE"} + +# Names that look like configuration to the patterns above but are not. +# ENVIRONMENT and RELEASE_ID are computed inside the workflow and passed down, and +# GITHUB_* is the runner's own namespace. +IGNORE = {"ENVIRONMENT", "RELEASE_ID", "SSH_TRANSPORT"} + +DECL = re.compile(r"^([A-Z][A-Z0-9_]*)=", re.M) +COMMENTED_DECL = re.compile(r"^#\s*([A-Z][A-Z0-9_]*)=", re.M) +GH_REF = re.compile(r"\b(?:vars|secrets)\.([A-Z][A-Z0-9_]*)\b") +# A row is `| `NAME` | ...`, and the backticks are what separate a described value from a +# mention of one in a sentence. The trailing `=value` is optional because a knob is +# documented as REQUIRE_BROTLI=1, which names the value that switches it on. +DOC_ROW = re.compile(r"^\|\s*`([A-Z][A-Z0-9_]*)(?:=[^`]*)?`", re.M) +# Values the doc names in prose rather than in a table row, which is how the three +# commented-out template keys and the two bot secrets are covered. +DOC_INLINE = re.compile(r"`([A-Z][A-Z0-9_]{2,})(?:=[^`]*)?`") + + +def main() -> int: + if not DOC.exists(): + print(f"ERROR: {DOC.name} does not exist", file=sys.stderr) + return 1 + + doc_text = DOC.read_text(encoding="utf-8") + documented_rows = set(DOC_ROW.findall(doc_text)) + documented_any = documented_rows | set(DOC_INLINE.findall(doc_text)) + + declared: dict[str, set[str]] = {} + + def note(name: str, where: str) -> None: + if name not in IGNORE: + declared.setdefault(name, set()).add(where) + + for path in TEMPLATES: + if not path.exists(): + print(f"ERROR: template {path} is missing", file=sys.stderr) + return 1 + text = path.read_text(encoding="utf-8") + rel = path.relative_to(REPO).as_posix() + for name in DECL.findall(text): + note(name, rel) + # A commented-out key is still a declared value: it documents the shape a + # deployment has to supply from somewhere else. + for name in COMMENTED_DECL.findall(text): + note(name, rel) + + for path in WORKFLOWS: + rel = path.relative_to(REPO).as_posix() + for name in GH_REF.findall(path.read_text(encoding="utf-8")): + note(name, rel) + + for name in KNOBS: + note(name, "per-invocation knob") + + undocumented = sorted(n for n in declared if n not in documented_any) + # Only table rows count as "described", so a value the doc merely mentions in passing + # is not treated as having a description it can be removed against. + unused = sorted(n for n in documented_rows if n not in declared) + + for name in undocumented: + where = ", ".join(sorted(declared[name])) + print(f"undocumented: {name} declared in {where} but not described in {DOC.name}") + for name in unused: + print(f"unused: {name} described in {DOC.name} but declared nowhere") + + total = len(undocumented) + len(unused) + if total: + print(f"\n{total} finding(s). Describe the value in {DOC.name}, or remove the row.") + return 1 + + print(f"{len(declared)} configuration value(s), all described in {DOC.name}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/deploy/README.md b/deploy/README.md index 43124fb..7fe3b01 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -57,7 +57,7 @@ checks/check-live-urls.sh "$HUGO_BASEURL" ``` The deploy root and the base URL are the only host-specific values, and they pair per -environment. Copy [`env.example`](./env.example) to `secrets/local.production.env`, which is the +environment. Copy [`example.env`](../example.env) to `secrets/local.production.env`, which is the file read when `ENV_FILE` is unset, and add `secrets/..env` for each further environment. A single environment therefore needs `secrets/local.production.env` and nothing else, since a differently named file is read only when `ENV_FILE` names it. `secrets/` is diff --git a/deploy/env.example b/deploy/env.example deleted file mode 100644 index f9aa36f..0000000 --- a/deploy/env.example +++ /dev/null @@ -1,96 +0,0 @@ -# Copy to secrets/..env and set for this host. -# Every value here names a machine rather than the project, so secrets/ is gitignored whole. -# CI sets the deploy values from environment secrets and reads no file. -# -# One file per environment, named for the server it describes and the environment on it, -# with both words spelled out, and selected by ENV_FILE: -# secrets/local.production.env the default, read when ENV_FILE is unset -# secrets/local.staging.env ENV_FILE=secrets/local.staging.env deploy/make-release.sh -# secrets/vps.production.env ENV_FILE=secrets/vps.production.env deploy/make-release.sh -# secrets/vps.staging.env ENV_FILE=secrets/vps.staging.env deploy/make-release.sh -# -# The file is sourced with `set -a`, which overwrites a variable the caller exported first. -# Selecting the file is therefore how an environment is chosen. -# The first argument to make-release.sh is how its root is overridden. -# A named file that does not exist is a hard failure rather than a fall-through. -# -# Naming convention: the prefix names whatever owns the value, not whatever reads it. -# HUGO_ is fixed by Hugo, which maps HUGO_ onto its own config natively. -# DEPLOY_ is the release tooling, which writes the deploy root. -# CADDY_ is the container, which owns state the release never touches. -# PANGOLIN_ is the proxy, which owns the credential that opens its auth gate. - -# Written by every release, and mounted read-only by the container at /srv/blog. -# The first argument to make-release.sh wins over this value. -DEPLOY_ROOT=/path/to/deploy/root - -# Must be set for anything that is not production. -# The base URL is baked into the canonical tag, the feed links, and every absolute permalink. -# A mirror built without it serves pages pointing back at production, and every gate still passes. -HUGO_BASEURL=https://blog.example.com/ - -# The container's persistent state root, deliberately outside DEPLOY_ROOT. -# Two directories hang off it, and a release writes neither: -# /config mounted at /config, holding the bootstrap Caddyfile -# /data mounted at /data, holding Caddy state and the reload autosave -# No script reads this, so it is recorded to keep a rebuild from depending on memory. -CADDY_APPDATA=/path/to/container/appdata - -# The container serving this environment. -# A release needs no restart, because Caddy reloads its config in process. -# Restarting is the remedy when the watcher dies, which it does silently after one failed load. -# -# Environments are named production and staging, spelled out, with no prod or stage anywhere. -# The name is compared, by EXPECT_SITE_ENV below and by the deploy. -# A spelling that differs by environment fails a deploy for a reason that reads like an outage. -CADDY_CONTAINER=blog-production - -# A resource access token, read by check-live-urls.sh, for an environment behind the auth gate. -# Staging keeps its gate on, because it serves a byte-identical copy of the public site. -# Set both or neither, and leave both unset for a site that is public. -PANGOLIN_ACCESS_TOKEN_ID= -PANGOLIN_ACCESS_TOKEN= - -# The environment that must answer, compared against the X-Blog-Env header the bundle stamps. -# A proxy rule aimed at the wrong container returns a healthy 200 under the right hostname. -# Checking the URL contract against that proves nothing, so the check refuses to start. -EXPECT_SITE_ENV=production - -# Read by the deploy workflow, which resolves them from the GitHub Environment rather than a file. -# They are named here so the local file and the environment describe the same shape. -# A local run deploys to a path and needs none of them. -#DEPLOY_SSH_HOST= -#DEPLOY_SSH_USER= -#DEPLOY_SSH_KNOWN_HOSTS= - -# The provenance capture, holding the WordPress exports, the crawl of the old platform, and the inventories derived from it. -# checks/build-redirects.py takes this directory as its one argument and rebuilds deploy/maps/ from it. -# Environment-independent, unlike every value above, so keep it in the default file, secrets/local.production.env, and drop it from any per-environment copy of this template. -# Nothing sources this value, so it is recorded to keep a rebuild from depending on memory. -CAPTURE_ROOT=/path/to/blog-capture - -# The off-host copy of the VPS access log, which OPERATIONS.md "Log Review" reads and vps-backup-pull writes. -# The encrypted host archives sit in a sibling directory and belong to the host rather than to this repo, so they are not named here. -# Environment-independent, like CAPTURE_ROOT above, so keep it in secrets/local.production.env and drop it from any per-environment copy. -# Nothing sources this value either. It is recorded because the alternative is a path that lives only in a gitignored session note and in the pull script's own defaults, which is how a review procedure ends up unable to say where the data it reviews is kept. -LOG_ARCHIVE_ROOT=/path/to/vps-logs - -# The off-host copy of the encrypted host archives and the plaintext hostconfig tree beside them. -# The plaintext copy is what makes the VPS's own configuration readable without the encryption key, which is not held here and must not be. -BACKUP_ARCHIVE_ROOT=/path/to/vps-archives - -# The VPS administrative connection, which is NOT the deploy connection above. -# DEPLOY_SSH_USER reaches a confined account behind an rrsync forced command that can write one release tree and read nothing. -# This one is the ordinary admin login, used for reading logs, reading the archive directory, and moving the channel files. -# They are separate values because they are separate credentials with different blast radii, and reaching for the wrong one either fails confusingly or grants far more than the task needed. -# Environment-independent, like CAPTURE_ROOT above: there is one such host, not one per environment. -VPS_SSH_HOST=root@vps.example.com - -# Where the VPS keeps the access log, whose rotated files LOG_ARCHIVE_ROOT is the off-host copy of. -# Today's live file is deliberately never pulled, because it is still being appended to and a copy is a torn prefix. -# So an analysis covering today reads VPS_TRAEFIK_LOG over SSH and everything older from LOG_ARCHIVE_ROOT. -VPS_TRAEFIK_LOG=/var/log/traefik/access.log -VPS_TRAEFIK_LOG_ARCHIVE=/var/log/traefik/archive - -# The two agent channel files, described under "The Channel Between the Two Sides". -VPS_COMMS_DIR=/srv/agent-comms diff --git a/example.env b/example.env new file mode 100644 index 0000000..6f214c0 --- /dev/null +++ b/example.env @@ -0,0 +1,110 @@ +# The one template for every configuration value this repository reads or writes. +# +# WHAT EACH VALUE MEANS IS IN ENVIRONMENT.md, which is the one place it is described. +# This file states the shape and a placeholder. Add a value here and describe it there, +# or checks/check-env-docs.py fails. +# +# It fills two destinations, marked below, because a value belongs to whichever machine +# holds it. Copy the section you need rather than the whole file. +# +# secrets/..env on a workstation, one file per environment +# /etc/vps-backup-pull.env on the backup host, or let ops/install.sh write it +# +# A value appearing in both sections is spelled the same way in both, deliberately. One +# name per thing means the side that writes and the side that reads cannot disagree, which +# is also why ops/install.sh copies values across rather than translating them. +# +# Naming convention: the prefix names whatever owns the value, not whatever reads it. +# HUGO_ is fixed by Hugo, DEPLOY_ is the release tooling, CADDY_ is the container, +# PANGOLIN_ is the proxy, VPS_ is the server, and a *_ROOT is a directory on this host. + +# ============================================================================= +# secrets/..env +# ============================================================================= +# One file per environment, named for the server it describes and the environment on it, +# with both words spelled out, and selected by ENV_FILE: +# secrets/local.production.env the default, read when ENV_FILE is unset +# secrets/local.staging.env ENV_FILE=secrets/local.staging.env deploy/make-release.sh +# secrets/vps.production.env ENV_FILE=secrets/vps.production.env deploy/make-release.sh +# secrets/vps.staging.env ENV_FILE=secrets/vps.staging.env deploy/make-release.sh +# +# Sourced with `set -a`, which overwrites a variable the caller exported first. A named +# file that does not exist is a hard failure rather than a fall-through. The whole +# secrets/ directory is gitignored, so no value naming a machine reaches this history. + +# Where a release is written. The first argument to make-release.sh wins over it. +DEPLOY_ROOT=/path/to/deploy/root + +# The site base URL. Must be set for anything that is not production. +HUGO_BASEURL=https://blog.example.com/ + +# The container's persistent state root, outside DEPLOY_ROOT. Nothing reads it. +CADDY_APPDATA=/path/to/container/appdata + +# The container serving this environment. +CADDY_CONTAINER=blog-production + +# The environment that must answer, compared against the X-Blog-Env header. +EXPECT_SITE_ENV=production + +# Resource access token for an environment behind the auth gate. Set both or neither. +PANGOLIN_ACCESS_TOKEN_ID= +PANGOLIN_ACCESS_TOKEN= + +# Read by the deploy workflow, which resolves them from the GitHub Environment rather than +# a file. Named here so the local file and the environment describe the same shape. +# A local run deploys to a path and needs none of them. +#DEPLOY_SSH_HOST= +#DEPLOY_SSH_USER= +#DEPLOY_SSH_KNOWN_HOSTS= + +# Environment-independent, so these belong in the default file only. + +# The provenance capture, holding the exports and the crawl of the old platform. +CAPTURE_ROOT=/path/to/blog-capture + +# The VPS administrative login, NOT the confined deploy account. +VPS_SSH_HOST=root@vps.example.com + +# Today's live access log on the VPS, read over SSH and never pulled. +VPS_TRAEFIK_LOG=/var/log/traefik/access.log + +# The two agent channel files on the VPS. +VPS_COMMS_DIR=/srv/agent-comms + +# ============================================================================= +# Both destinations +# ============================================================================= +# These name the off-host copy, so the pull writes them and the log review reads them. + +# Off-host archives and the plaintext hostconfig tree beside them. +BACKUP_ARCHIVE_ROOT=/path/to/backup/vps + +# Off-host copy of the rotated logs. Mode 700, since query strings are logged in full. +LOG_ARCHIVE_ROOT=/path/to/backup/vps-logs + +# The rotated access logs on the VPS, and the source of that copy. +VPS_TRAEFIK_LOG_ARCHIVE=/var/log/traefik/archive + +# ============================================================================= +# /etc/vps-backup-pull.env +# ============================================================================= +# On the backup host. Also needs VPS_SSH_HOST and the three values above. +# +# systemd parses this file itself rather than passing it to a shell, so there is no +# expansion and no command substitution: a $ or a backtick is a literal character. It does +# strip matching quotes, which is why a value containing spaces is quoted and arrives +# without them. +# +# VPS_SSH_HOST, BACKUP_ARCHIVE_ROOT and LOG_ARCHIVE_ROOT have no defaults in the pull. An +# address and a destination belong to one host, and a wrong-but-valid destination is a +# backup nobody can find, so it names what is missing and refuses to run. + +# The layout on the VPS, the same for any host running this stack. +VPS_ARCHIVE_DIR=/var/backups/pangolin +VPS_BLOG_LOG_DIR=/var/log/blog/legacy + +# Key auth only, since the VPS has password auth disabled. +# Quoted because it contains spaces: this file is sourced by a shell for the secrets/ +# half, where a bare value would run everything after the first space as a command. +SSH_OPTS="-o ConnectTimeout=15 -o BatchMode=yes" diff --git a/ops/README.md b/ops/README.md index 602f02b..c5ad1e2 100644 --- a/ops/README.md +++ b/ops/README.md @@ -8,8 +8,8 @@ Tooling that runs on the **backup host**, not on the web server and not in CI. O | `vps-backup-pull` | `/usr/local/bin/vps-backup-pull` | | `vps-backup-pull.service` | `/etc/systemd/system/` | | `vps-backup-pull.timer` | `/etc/systemd/system/` | -| `vps-backup-pull.env.example` | `/etc/vps-backup-pull.env` | | `vps-backup-pull.service.d-local.conf.example` | `/etc/systemd/system/vps-backup-pull.service.d/local.conf` | +| [`example.env`](../example.env) | `/etc/vps-backup-pull.env` | **The last two are required, not optional, and `install.sh` generates both.** Nothing in this directory names a machine, so the address, the destination paths, and the account are supplied at install time from values this repository already holds. A missing value stops the pull with the name of what is missing rather than falling back to something plausible, since a wrong-but-valid destination is a backup nobody can find. The two `.example` files document the format and are not the install path. @@ -65,9 +65,9 @@ systemctl list-timers vps-backup-pull.timer --all ## Variables -Every path is a variable, so a host states its own layout rather than editing a file git owns. They are listed with their meanings in [`vps-backup-pull.env.example`](./vps-backup-pull.env.example), and the three `systemd` settings that cannot come from an environment file are in [`vps-backup-pull.service.d-local.conf.example`](./vps-backup-pull.service.d-local.conf.example). +Every path is a variable, so a host states its own layout rather than editing a file git owns. [`example.env`](../example.env) lists them and [`ENVIRONMENT.md`](../ENVIRONMENT.md) describes them. The three `systemd` settings that cannot come from an environment file are in [`vps-backup-pull.service.d-local.conf.example`](./vps-backup-pull.service.d-local.conf.example). -**They are the same names Blog's own `secrets/` file uses, which is the point.** `VPS_*` is something on the VPS and `*_ROOT` is something on this host, and the pull writes the two roots that the log review reads. One name per directory means the writing side and the reading side cannot disagree, and it is why `install.sh` copies rather than translates. An earlier split into two vocabularies needed a mapping table to hold it together, and unifying them deleted the table. +**They are the same names Blog's own `secrets/` file uses, which is the point.** `VPS_*` is something on the VPS and `*_ROOT` is something on this host, and the pull writes the two roots that the log review reads. One name per directory means the writing side and the reading side cannot disagree, and it is why `install.sh` copies rather than translates. Every value is described in [`ENVIRONMENT.md`](../ENVIRONMENT.md). ## This directory is the source diff --git a/ops/install.sh b/ops/install.sh index 871a54b..9f3ac27 100755 --- a/ops/install.sh +++ b/ops/install.sh @@ -7,10 +7,8 @@ # secrets/..env under the same names the pull itself uses, so this # copies them rather than translating them, and the account is whoever runs this. # -# There is deliberately no name mapping here. An earlier version of this file had one, -# because the pull and the secrets file had grown separate names for the same directories, -# and a table translating four pairs is a thing to get wrong every time one side changes. -# The names were unified instead, which deleted the table rather than maintaining it. +# There is no name mapping here, because both sides spell every shared value the same way. +# Keep it that way: a translation table is a thing to get wrong every time one side changes. # # Usage: ops/install.sh [--check] [--force] # --check validate and print what would be written, touch nothing, need no root diff --git a/ops/vps-backup-pull b/ops/vps-backup-pull index 753481d..21476e4 100755 --- a/ops/vps-backup-pull +++ b/ops/vps-backup-pull @@ -19,7 +19,7 @@ # it. A copy found on a host is an installed artifact, not a place to make a change. # # Paths below are read from /etc/vps-backup-pull.env, so a host states its own layout -# there rather than editing this file. See ops/vps-backup-pull.env.example. +# there rather than editing this file. See example.env and ENVIRONMENT.md. # # Why a PULL and not a push from the VPS: it keeps the VPS free of any credential # reaching another system, which is the one security rule in CONFIG.md item 0. @@ -40,19 +40,8 @@ # ciphertext it would make the encryption decorative. It is in 1Password. # # Every path is an environment variable, read from /etc/vps-backup-pull.env, so a host -# states its own layout there rather than editing a file git owns. VPS_* names something on -# the VPS and *_ROOT names something on this host, and these are the same names -# secrets/..env uses, so there is one vocabulary rather than two. -# -# VPS_SSH_HOST (required) where to pull from -# BACKUP_ARCHIVE_ROOT (required) archives + hostconfig land here -# LOG_ARCHIVE_ROOT (required) both log sets land here -# VPS_ARCHIVE_DIR /var/backups/pangolin the archives ON THE VPS -# VPS_TRAEFIK_LOG_ARCHIVE /var/log/traefik/archive rotated edge access logs ON THE VPS -# VPS_BLOG_LOG_DIR /var/log/blog/legacy one-off Caddy dumps ON THE VPS -# -# The three marked required carry no default, because an address and a destination belong -# to one host. A wrong-but-valid destination is a backup nobody can find. +# states its own layout there rather than editing a file git owns. The variables are listed +# in example.env and described in ENVIRONMENT.md. # # Usage: vps-backup-pull [--delete] [--no-hostconfig] [--no-logs] [--dry-run] [--quiet] # --delete mirror deletions (Proxmox tracks the VPS's 14-archive window @@ -144,8 +133,8 @@ LOG_RSYNC_OPTS=("${RSYNC_OPTS[@]}") # those instead. The unit omits User= deliberately, so this is what catches a missing # drop-in rather than the failure surfacing a day later as a permissions error. [[ $EUID -ne 0 ]] || die "refusing to run as root -- run as the account that owns the destination and holds the VPS key (systemd: set User= in the drop-in)" -[[ -n $VPS_SSH_HOST ]] || die "VPS_SSH_HOST is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" -[[ -n $BACKUP_ARCHIVE_ROOT ]] || die "BACKUP_ARCHIVE_ROOT is not set -- see /etc/vps-backup-pull.env (template: ops/vps-backup-pull.env.example)" +[[ -n $VPS_SSH_HOST ]] || die "VPS_SSH_HOST is not set -- see /etc/vps-backup-pull.env (template: example.env)" +[[ -n $BACKUP_ARCHIVE_ROOT ]] || die "BACKUP_ARCHIVE_ROOT is not set -- see /etc/vps-backup-pull.env (template: example.env)" [[ $LOGS -eq 0 || -n $LOG_ARCHIVE_ROOT ]] || die "LOG_ARCHIVE_ROOT is not set and the log leg is enabled -- set it, or pass --no-logs" command -v rsync >/dev/null || die "rsync not installed on this host" diff --git a/ops/vps-backup-pull.env.example b/ops/vps-backup-pull.env.example deleted file mode 100644 index fcd8372..0000000 --- a/ops/vps-backup-pull.env.example +++ /dev/null @@ -1,35 +0,0 @@ -# Copy to /etc/vps-backup-pull.env on the backup host, or let ops/install.sh generate it. -# -# These are the same variable names secrets/..env uses, deliberately. -# One name per thing: VPS_* is something on the VPS, *_ROOT is something on this host. -# The pull writes those two roots and the log review reads them, so a single vocabulary is -# what keeps the two sides pointing at the same directories with nothing to reconcile. -# -# systemd parses this file itself rather than passing it to a shell. There is no expansion, -# no command substitution, and no quoting to strip, so write a bare value and never wrap it -# in quotes, or the quotes become part of the path. - -# Where to pull from. The administrative login, not the confined deploy account. -VPS_SSH_HOST=root@vps.example.com - -# On the VPS: the encrypted archives, written by its own backup timer. -VPS_ARCHIVE_DIR=/var/backups/pangolin - -# On the VPS: the rotated edge access logs. Today's live file is deliberately not among -# them, because rotation is what makes a file eligible to be copied. -VPS_TRAEFIK_LOG_ARCHIVE=/var/log/traefik/archive - -# On the VPS: one-off Caddy container dumps kept from before rotation existed. -VPS_BLOG_LOG_DIR=/var/log/blog/legacy - -# On this host: the archives and the plaintext hostconfig tree beside them. -BACKUP_ARCHIVE_ROOT=/path/to/backup/vps - -# On this host: both log sets. Separate from BACKUP_ARCHIVE_ROOT, and mode 700, because -# these are plaintext and long-lived where the archives are encrypted and pruned, and -# because query strings are logged in full so the tree is as sensitive as the log. -LOG_ARCHIVE_ROOT=/path/to/backup/vps-logs - -# Key auth only, since the VPS has password auth disabled. BatchMode makes an unusable key -# fail immediately instead of hanging a timed run on a password prompt nobody sees. -SSH_OPTS=-o ConnectTimeout=15 -o BatchMode=yes From df5a95679b850b1778ac789174e7b8f61f6f0d80 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 10:49:06 -0700 Subject: [PATCH 09/14] Put the ops scripts under the shell gate the other two already pass Two shell scripts were added to a repo that shellcheck-and-shfmt gates its shell, and neither was in the list, so CI would have said nothing about them indefinitely. Both now pass the gate unchanged in behavior: - shfmt -w reformatted them. install.sh takes tabs from .editorconfig [*.sh]; vps-backup-pull carries no extension so it takes the [*] default of four spaces. Formatting only, proven by diffing the dry-run output of the old and new scripts rather than by reading the diff: identical except rsync's own byte counters, which vary per run. - Two shellcheck findings are suppressed with the reasoning at the site. SC2329 on cleanup(), which the EXIT trap invokes and shellcheck cannot see. SC2029 on the checksum ssh, where the local expansion of VPS_ARCHIVE_DIR is the intent and the *.enc glob stays quoted for the remote shell. Running shfmt -w through Docker writes as root and takes ownership of the files it touches. The workflow comment now says to use -d, which is what CI runs and what avoids it. The installed copy predates the reformat, so it is no longer byte- identical to the committed one. Normalizing it through shfmt and diffing shows the only content difference is the two directives, so the running backup is correct, and re-running ops/install.sh restores the identity. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/validate-task.yml | 6 +- TODO.md | 6 +- ops/install.sh | 113 +++++++++------ ops/vps-backup-pull | 212 ++++++++++++++++------------ 4 files changed, 198 insertions(+), 139 deletions(-) diff --git a/.github/workflows/validate-task.yml b/.github/workflows/validate-task.yml index 0d36d4c..20eb390 100644 --- a/.github/workflows/validate-task.yml +++ b/.github/workflows/validate-task.yml @@ -53,11 +53,13 @@ jobs: run: docker run --rm --pull=always -v "$PWD":/check --workdir /check mstruebing/editorconfig-checker:latest # The shell clean-compile is shellcheck at default severity plus `shfmt -d`, both reporting nothing. - # The formatter reads .editorconfig, which pins these scripts to tabs. + # The formatter reads .editorconfig, which pins the .sh files to tabs. + # ops/vps-backup-pull carries no extension, so it takes the [*] default of four spaces instead. + # Use `-d` rather than `-w` here and locally: the container writes as root and would take ownership of the tree. - name: Lint shell scripts step run: | set -Eeuo pipefail - scripts=(checks/check-live-urls.sh deploy/make-release.sh) + scripts=(checks/check-live-urls.sh deploy/make-release.sh ops/vps-backup-pull ops/install.sh) docker run --rm --pull=always -v "$PWD":/mnt --workdir /mnt \ koalaman/shellcheck:stable "${scripts[@]}" docker run --rm --pull=always -v "$PWD":/mnt --workdir /mnt \ diff --git a/TODO.md b/TODO.md index f08980a..13ea461 100644 --- a/TODO.md +++ b/TODO.md @@ -55,8 +55,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returned 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently. **That one file is deleted and the archive reads 58**, and the general case now has a mechanism: the host side keeps an append-only `RECONCILE.md` **inside the archive directory**, so the pull carries it alongside the data it explains rather than relying on a note in a channel file nobody rereads. It records what a file contained rather than what it was called, and it will be counted among the pulled log files. - **Read a date from a line's `StartUTC` rather than from the filename holding it.** That is the durable form of the lesson, and it is in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" as well. - **An off-host copy also predates the timer.** The directory mtimes on the backup host are copy times where the file mtimes are the VPS's, and they show a pull at 2026-08-08 03:31 UTC that the service journal has no record of, because the script was run directly rather than through `systemd`. So a journal with one entry is not evidence of one copy. The whole set was audited both ways afterwards and nothing else had diverged: logs identical, 38 archives shared and identical in size, four newer on the VPS because they postdate the pull, four older retained off-host because the pull passes no `--delete`. - - **Both halves were exercised 2026-08-08 between 16:18 and 16:22 UTC, deliberately without running the service.** `systemctl start` would have written the second journal entry that 2026-08-09 is supposed to prove, so the transport was exercised with `--dry-run` instead and the timer was read rather than triggered: `NEXT` is 2026-08-09 09:08:45 UTC inside the 15-minute randomized window, `LAST` is `-`, and the journal still holds exactly one entry. **If a second entry exists before that time, someone ran it by hand and the schedule is still unproven.** The dry run reached the VPS over SSH and all three legs planned cleanly. Pending for the first timed run: five encrypted archives dated 2026-08-08, plus `RECONCILE.md` **and** `MANIFEST.txt`, so the log-file count rises by two non-log files rather than the one recorded above. No rotated access log is pending, which is correct, because the 00:00 UTC rotation that produces `access.log-2026-08-08` has not happened yet. - - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is committed at [`ops/vps-backup-pull`](./ops/vps-backup-pull) and is not installed**, so the running copy still lacks it until `ops/install.sh` is run. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. + - **Both halves were exercised 2026-08-08 between 16:18 and 16:22 UTC, deliberately without running the service.** `systemctl start` would have written the second journal entry that 2026-08-09 is supposed to prove, so the transport was exercised with `--dry-run` instead and the timer was read rather than triggered: `LAST` is `-` and the journal still holds exactly one entry, with `NEXT` inside the 15-minute randomized window after 09:00 UTC. **Read `NEXT` rather than remembering it**, because `systemctl enable` redraws that offset: it moved from 09:08:45 to 09:00:12 UTC when the unit was installed. **If a second entry exists before that time, someone ran it by hand and the schedule is still unproven.** The dry run reached the VPS over SSH and all three legs planned cleanly. Pending for the first timed run: five encrypted archives dated 2026-08-08, plus `RECONCILE.md` **and** `MANIFEST.txt`, so the log-file count rises by two non-log files rather than the one recorded above. No rotated access log is pending, which is correct, because the 00:00 UTC rotation that produces `access.log-2026-08-08` has not happened yet. + - **`--dry-run` named nothing, which made it a connectivity test wearing a preview's name.** `RSYNC_OPTS` carried only `-a --human-readable --info=stats1`, so a dry run printed transfer totals and not one filename, and "what will tomorrow's run bring" was unanswerable by the flag that exists to answer it. Fixed by adding `--itemize-changes` alongside `--dry-run`, which is how the pending set above was read. **The fix is committed at [`ops/vps-backup-pull`](./ops/vps-backup-pull) and installed 2026-08-08**, verified byte-identical to the committed copy. Installing it is a maintainer step, and it stacks with the unshipped change [#53][issue-53] already owes the VPS canonical at `/usr/local/share/pangolin-maint/vps-pull.sh`. - **The VPS's older copy of the script is not a source, and reconciling the two is [#53][issue-53].** The committed copy carries the whole access-log leg, the `tell()` fix and the `VERIFIED` counter. The VPS's carries an install block and a no-sudo rationale that this side lacked, now folded in. Neither direction is a safe overwrite, so #53 is a merge rather than a copy, and copying the VPS's over the committed one would delete the log pull that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" runs on. - **The header's own install command pointed at `/usr/local/sbin/vps-backup-pull`, which nothing runs.** `vps-backup-pull.service` runs `/usr/local/bin/vps-backup-pull` and `/usr/local/sbin/` is empty, so following the instruction would have written a second copy nobody executes while `scp` and `chmod` both reported success and the timer went on running the old one. The canonical had already corrected this to a `sudo install` into `bin/`, and states the reason `bin/` is deliberate. **The stale block is replaced in the patch copy** with the canonical's wording plus an explicit refusal to run that `scp` until the divergence above is reconciled. It is another reason a plain overwrite in either direction is the wrong merge. - **The outward pass ran end to end on 2026-08-08 traffic and found nothing to add.** 9,996 edge requests, 9,285 of them this repo's own deploy gate. Of the 711 that remain, 101 were site-host 404s across 73 distinct paths, and every one was a scanner shape. Only `/` and `/robots.txt` intersect the URL contract at all, and both are explained rather than open: `/robots.txt` 404ed until the 15:47:17 deploy and has answered 200 since 15:48:45, and `/` 404ed twice at the edge, below. **No legacy content URL 404ed, so `checks/golden-urls.txt` needs no addition from this run**, which is the expected result and is recorded because an unrecorded clean pass is indistinguishable from a pass nobody ran. @@ -77,7 +77,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions -- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **One step remains and it is one command:** `ops/install.sh`, run as the account that owns the backup. It derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env` and installs everything, so nothing is typed twice and the two vocabularies cannot drift. `ops/install.sh --check` proves the derivation without root and writes nothing, and it has been run: every value matches the running configuration. The install itself needs a password this session could not supply. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. +- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **Installed 2026-08-08 with `ops/install.sh`**, which derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env`, so nothing is typed twice. Verified after the fact rather than from the installer's own output: `systemd` resolves `User=pieter`, `Group=users` and `RequiresMountsFor=/data/backup` from the drop-in, and the environment file is `600 root:root`. **The committed script was reformatted for the shell gate after that install, so re-run `ops/install.sh` to restore byte-identity.** The difference is the formatting plus two `shellcheck` directives, confirmed by normalizing the running copy through `shfmt` and diffing, so the running backup is correct meanwhile. The root guard was exercised and refused. **The journal still holds exactly one entry and the timer's `LAST` is still `-`**, so installing did not spend the evidence that 2026-08-09 is the first scheduled run. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. - **Where the rest of the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy were an operational asset built from another agent's instructions, and they existed only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. - **The pull itself is resolved and the reasoning is kept because it applies to everything still listed here.** What made it urgent was measured: the copy protected everywhere was the VPS's older one, while the copy that actually ran, carrying the log leg the review depends on, was in no snapshot and no repository. Committing it is what closed that, not the backup host's own off-site copy, which never reached the script. - **The directory holding it is named as though it were disposable.** `~/vps-backup-pull-patch` reads as a patch staged against a source, and there is no source: it is the most complete copy of the script in existence. A directory named for a temporary artifact is the one a cleanup deletes, and nothing here would notice until a restore produced the wrong script. diff --git a/ops/install.sh b/ops/install.sh index 9f3ac27..65ddb4c 100755 --- a/ops/install.sh +++ b/ops/install.sh @@ -19,17 +19,33 @@ # root. Individual privileged steps call sudo themselves. set -euo pipefail -CHECK=0; FORCE=0 +CHECK=0 +FORCE=0 while [[ $# -gt 0 ]]; do - case "$1" in - --check) CHECK=1; shift ;; - --force) FORCE=1; shift ;; - -h|--help) sed -n '2,22p' "$0"; exit 0 ;; - *) printf 'ERROR: unknown argument %s\n' "$1" >&2; exit 1 ;; - esac + case "$1" in + --check) + CHECK=1 + shift + ;; + --force) + FORCE=1 + shift + ;; + -h | --help) + sed -n '2,22p' "$0" + exit 0 + ;; + *) + printf 'ERROR: unknown argument %s\n' "$1" >&2 + exit 1 + ;; + esac done -die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } +die() { + printf 'ERROR: %s\n' "$*" >&2 + exit 1 +} note() { printf ' %s\n' "$*"; } REPO=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) @@ -45,9 +61,9 @@ set -a . "$ENV_FILE" set +a -[[ -n ${VPS_SSH_HOST:-} ]] || die "VPS_SSH_HOST is not set in $ENV_FILE" +[[ -n ${VPS_SSH_HOST:-} ]] || die "VPS_SSH_HOST is not set in $ENV_FILE" [[ -n ${BACKUP_ARCHIVE_ROOT:-} ]] || die "BACKUP_ARCHIVE_ROOT is not set in $ENV_FILE" -[[ -n ${LOG_ARCHIVE_ROOT:-} ]] || die "LOG_ARCHIVE_ROOT is not set in $ENV_FILE" +[[ -n ${LOG_ARCHIVE_ROOT:-} ]] || die "LOG_ARCHIVE_ROOT is not set in $ENV_FILE" ACCOUNT=$(id -un) @@ -65,8 +81,8 @@ GROUP=$(stat -c %G "$ANCESTOR") # The mount the destination sits on, resolved rather than guessed, because RequiresMountsFor # has to name the mount point and not the directory below it. -MOUNT=$(findmnt -no TARGET --target "$ANCESTOR" 2>/dev/null) \ - || die "cannot resolve the mount holding $BACKUP_ARCHIVE_ROOT" +MOUNT=$(findmnt -no TARGET --target "$ANCESTOR" 2>/dev/null) || + die "cannot resolve the mount holding $BACKUP_ARCHIVE_ROOT" VPS_TRAEFIK_ARCHIVE=${VPS_TRAEFIK_LOG_ARCHIVE:-/var/log/traefik/archive} @@ -82,7 +98,8 @@ ENV_DEST=/etc/vps-backup-pull.env DROPIN_DIR=/etc/systemd/system/vps-backup-pull.service.d DROPIN_DEST=$DROPIN_DIR/local.conf -ENV_BODY=$(cat </dev/null; then - printf '=== %s: exists, contents need root to compare\n' "$path"; changed=1 - elif printf '%s\n' "$body" | sudo -n diff -q - "$path" >/dev/null 2>&1; then - printf '=== %s: already correct\n' "$path" - else - printf '=== %s: differs\n' "$path" - printf '%s\n' "$body" | sudo -n diff -u "$path" - || true - [[ $FORCE -eq 1 ]] || die "$path exists with different contents -- re-run with --force to replace it" - changed=1 - fi + path=${pair%%|*} + body=${pair#*|} + if [[ ! -e $path ]]; then + printf '=== %s: will be created\n' "$path" + changed=1 + elif ! sudo -n true 2>/dev/null; then + printf '=== %s: exists, contents need root to compare\n' "$path" + changed=1 + elif printf '%s\n' "$body" | sudo -n diff -q - "$path" >/dev/null 2>&1; then + printf '=== %s: already correct\n' "$path" + else + printf '=== %s: differs\n' "$path" + printf '%s\n' "$body" | sudo -n diff -u "$path" - || true + [[ $FORCE -eq 1 ]] || die "$path exists with different contents -- re-run with --force to replace it" + changed=1 + fi done if [[ $CHECK -eq 1 ]]; then - printf '\n=== %s would contain\n' "$ENV_DEST" - printf '%s\n' "$ENV_BODY" | sed 's/^/ /' - printf '\n=== %s would contain\n' "$DROPIN_DEST" - printf '%s\n' "$DROPIN_BODY" | sed 's/^/ /' - printf '\n=== check only -- nothing written\n' - printf '=== verifying the VPS is reachable as %s\n' "$ACCOUNT" - # if/else rather than `A && B || C`, which runs C when B fails as well as when A does. - # shellcheck disable=SC2086 - if ssh ${SSH_OPTS:--o ConnectTimeout=15 -o BatchMode=yes} "$VPS_SSH_HOST" true; then - printf ' reachable\n' - else - die "cannot reach $VPS_SSH_HOST over SSH as $ACCOUNT (key auth only)" - fi - exit 0 + printf '\n=== %s would contain\n' "$ENV_DEST" + printf '%s\n' "$ENV_BODY" | sed 's/^/ /' + printf '\n=== %s would contain\n' "$DROPIN_DEST" + printf '%s\n' "$DROPIN_BODY" | sed 's/^/ /' + printf '\n=== check only -- nothing written\n' + printf '=== verifying the VPS is reachable as %s\n' "$ACCOUNT" + # if/else rather than `A && B || C`, which runs C when B fails as well as when A does. + # shellcheck disable=SC2086 + if ssh ${SSH_OPTS:--o ConnectTimeout=15 -o BatchMode=yes} "$VPS_SSH_HOST" true; then + printf ' reachable\n' + else + die "cannot reach $VPS_SSH_HOST over SSH as $ACCOUNT (key auth only)" + fi + exit 0 fi printf '=== installing\n' sudo install -m 755 "$REPO/ops/vps-backup-pull" /usr/local/bin/vps-backup-pull sudo install -m 644 "$REPO/ops/vps-backup-pull.service" "$REPO/ops/vps-backup-pull.timer" \ - /etc/systemd/system/ + /etc/systemd/system/ printf '%s\n' "$ENV_BODY" | sudo install -m 600 /dev/stdin "$ENV_DEST" sudo mkdir -p "$DROPIN_DIR" printf '%s\n' "$DROPIN_BODY" | sudo install -m 644 /dev/stdin "$DROPIN_DEST" @@ -162,12 +183,12 @@ sudo systemctl enable --now vps-backup-pull.timer printf '=== verifying the installed copy, without writing a journal entry\n' # --dry-run rather than starting the service, because a service record is the evidence that # the timer fired, and manufacturing one here would spend that evidence to prove the install. -/usr/local/bin/vps-backup-pull --dry-run >/dev/null \ - || die "the installed script failed its dry run -- $ENV_DEST or the drop-in is wrong" +/usr/local/bin/vps-backup-pull --dry-run >/dev/null || + die "the installed script failed its dry run -- $ENV_DEST or the drop-in is wrong" printf ' dry run clean\n' systemctl list-timers vps-backup-pull.timer --all --no-pager if [[ $changed -eq 1 ]]; then - printf '=== done\n' + printf '=== done\n' else - printf '=== done -- configuration was already correct\n' + printf '=== done -- configuration was already correct\n' fi diff --git a/ops/vps-backup-pull b/ops/vps-backup-pull index 21476e4..5a93279 100755 --- a/ops/vps-backup-pull +++ b/ops/vps-backup-pull @@ -71,25 +71,52 @@ VPS_TRAEFIK_LOG_ARCHIVE=${VPS_TRAEFIK_LOG_ARCHIVE:-/var/log/traefik/archive} VPS_BLOG_LOG_DIR=${VPS_BLOG_LOG_DIR:-/var/log/blog/legacy} LOG_ARCHIVE_ROOT=${LOG_ARCHIVE_ROOT:-} -DELETE=0; HOSTCONFIG=1; LOGS=1; DRYRUN=0; QUIET=0 +DELETE=0 +HOSTCONFIG=1 +LOGS=1 +DRYRUN=0 +QUIET=0 LOGFILES=0 while [[ $# -gt 0 ]]; do - case "$1" in - --delete) DELETE=1; shift ;; - --no-hostconfig) HOSTCONFIG=0; shift ;; - --no-logs) LOGS=0; shift ;; - --dry-run) DRYRUN=1; shift ;; - --quiet) QUIET=1; shift ;; - -h|--help) - awk 'NR>1 && /^#/{sub(/^# ?/,""); print; next} NR>1{exit}' "$0"; exit 0 ;; - *) echo "unknown argument: $1" >&2; exit 2 ;; - esac + case "$1" in + --delete) + DELETE=1 + shift + ;; + --no-hostconfig) + HOSTCONFIG=0 + shift + ;; + --no-logs) + LOGS=0 + shift + ;; + --dry-run) + DRYRUN=1 + shift + ;; + --quiet) + QUIET=1 + shift + ;; + -h | --help) + awk 'NR>1 && /^#/{sub(/^# ?/,""); print; next} NR>1{exit}' "$0" + exit 0 + ;; + *) + echo "unknown argument: $1" >&2 + exit 2 + ;; + esac done -say() { [[ $QUIET -eq 1 ]] || printf '%s\n' "$*"; } +say() { [[ $QUIET -eq 1 ]] || printf '%s\n' "$*"; } warn() { printf 'WARNING: %s\n' "$*" >&2; } -die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } +die() { + printf 'ERROR: %s\n' "$*" >&2 + exit 1 +} # The closing summary, which prints even under --quiet. It went through say() before, so --quiet # silenced it too and delivered neither half of the "errors and the summary only" this script's @@ -101,8 +128,14 @@ tell() { printf '%s\n' "$*"; } # Declared up front and cleaned up by ONE trap. Both are conditional -- --dry-run # skips the checksum step, so a trap referencing an unset $SUMS would fail under # `set -u` at exit and mask the real exit status. -SUMS=""; LIST="" -cleanup() { [[ -n $SUMS ]] && rm -f "$SUMS"; [[ -n $LIST ]] && rm -f "$LIST"; return 0; } +SUMS="" +LIST="" +# shellcheck disable=SC2329 # invoked by the EXIT trap below, which shellcheck cannot see +cleanup() { + [[ -n $SUMS ]] && rm -f "$SUMS" + [[ -n $LIST ]] && rm -f "$LIST" + return 0 +} trap cleanup EXIT RSYNC_OPTS=(-a --human-readable) @@ -134,7 +167,7 @@ LOG_RSYNC_OPTS=("${RSYNC_OPTS[@]}") # drop-in rather than the failure surfacing a day later as a permissions error. [[ $EUID -ne 0 ]] || die "refusing to run as root -- run as the account that owns the destination and holds the VPS key (systemd: set User= in the drop-in)" [[ -n $VPS_SSH_HOST ]] || die "VPS_SSH_HOST is not set -- see /etc/vps-backup-pull.env (template: example.env)" -[[ -n $BACKUP_ARCHIVE_ROOT ]] || die "BACKUP_ARCHIVE_ROOT is not set -- see /etc/vps-backup-pull.env (template: example.env)" +[[ -n $BACKUP_ARCHIVE_ROOT ]] || die "BACKUP_ARCHIVE_ROOT is not set -- see /etc/vps-backup-pull.env (template: example.env)" [[ $LOGS -eq 0 || -n $LOG_ARCHIVE_ROOT ]] || die "LOG_ARCHIVE_ROOT is not set and the log leg is enabled -- set it, or pass --no-logs" command -v rsync >/dev/null || die "rsync not installed on this host" @@ -144,12 +177,12 @@ say " source: $VPS_SSH_HOST dest: $BACKUP_ARCHIVE_ROOT" [[ $LOGS -eq 1 ]] && say " logs: $LOG_ARCHIVE_ROOT" [[ $DRYRUN -eq 1 ]] && say " DRY RUN -- nothing will be written" # shellcheck disable=SC2086 -ssh $SSH_OPTS "$VPS_SSH_HOST" true 2>/dev/null \ - || die "cannot reach $VPS_SSH_HOST over SSH (key auth only -- password auth is disabled there)" +ssh $SSH_OPTS "$VPS_SSH_HOST" true 2>/dev/null || + die "cannot reach $VPS_SSH_HOST over SSH (key auth only -- password auth is disabled there)" if [[ $DRYRUN -eq 0 ]]; then - mkdir -p "$BACKUP_ARCHIVE_ROOT/pangolin" || die "cannot create $BACKUP_ARCHIVE_ROOT/pangolin" - chmod 700 "$BACKUP_ARCHIVE_ROOT" + mkdir -p "$BACKUP_ARCHIVE_ROOT/pangolin" || die "cannot create $BACKUP_ARCHIVE_ROOT/pangolin" + chmod 700 "$BACKUP_ARCHIVE_ROOT" fi # ---------------------------------------------------------------- 1. archives @@ -157,8 +190,8 @@ say say "--- archives" # shellcheck disable=SC2086 rsync "${RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - "$VPS_SSH_HOST:$VPS_ARCHIVE_DIR/" "$BACKUP_ARCHIVE_ROOT/pangolin/" \ - || die "archive rsync failed" + "$VPS_SSH_HOST:$VPS_ARCHIVE_DIR/" "$BACKUP_ARCHIVE_ROOT/pangolin/" || + die "archive rsync failed" # ---------------------------------------------------------------- 2. verify # rsync verifies its own transfers, but this re-reads what actually landed on @@ -169,50 +202,53 @@ VERIFY_RC=0 # its own summary and never reaches that line. VERIFIED=0 if [[ $DRYRUN -eq 1 ]]; then - say " (dry run -- checksum verification skipped)" + say " (dry run -- checksum verification skipped)" else - say - say "--- verifying checksums" - SUMS=$(mktemp) - # shellcheck disable=SC2086 - ssh $SSH_OPTS "$VPS_SSH_HOST" "cd $VPS_ARCHIVE_DIR && sha256sum *.enc" >"$SUMS" \ - || die "could not read source checksums" - EXPECTED=$(wc -l <"$SUMS") - if OUTPUT=$(cd "$BACKUP_ARCHIVE_ROOT/pangolin" && sha256sum -c "$SUMS" 2>&1); then - VERIFIED=$EXPECTED - say " $EXPECTED/$EXPECTED archives verified OK" - else - VERIFY_RC=1 - printf '%s\n' "$OUTPUT" | grep -v ': OK$' >&2 || true - warn "checksum verification FAILED -- the local copy does not match the VPS" - fi + say + say "--- verifying checksums" + SUMS=$(mktemp) + # SC2086: SSH_OPTS is a word list and must split. + # SC2029: VPS_ARCHIVE_DIR expanding here is the point, since the configured path is + # this side's. The *.enc glob stays quoted so the remote shell expands it instead. + # shellcheck disable=SC2086,SC2029 + ssh $SSH_OPTS "$VPS_SSH_HOST" "cd $VPS_ARCHIVE_DIR && sha256sum *.enc" >"$SUMS" || + die "could not read source checksums" + EXPECTED=$(wc -l <"$SUMS") + if OUTPUT=$(cd "$BACKUP_ARCHIVE_ROOT/pangolin" && sha256sum -c "$SUMS" 2>&1); then + VERIFIED=$EXPECTED + say " $EXPECTED/$EXPECTED archives verified OK" + else + VERIFY_RC=1 + printf '%s\n' "$OUTPUT" | grep -v ': OK$' >&2 || true + warn "checksum verification FAILED -- the local copy does not match the VPS" + fi fi # ---------------------------------------------------------------- 3. hostconfig if [[ $HOSTCONFIG -eq 1 ]]; then - say - say "--- host config (plaintext, readable without the key)" - LIST=$(mktemp) - # Ask the VPS what it considers valuable. Single source of truth: the list is - # defined once, in pangolin-backup's HOST_FILES, and never copied to this host. - # shellcheck disable=SC2086 - if ssh $SSH_OPTS "$VPS_SSH_HOST" 'pangolin-backup --list-files' 2>/dev/null \ - | sed 's|^/||' >"$LIST" && [[ -s $LIST ]]; then - WANT=$(wc -l <"$LIST") - say " $WANT path(s) advertised by the VPS" - [[ $DRYRUN -eq 0 ]] && mkdir -p "$BACKUP_ARCHIVE_ROOT/hostconfig" + say + say "--- host config (plaintext, readable without the key)" + LIST=$(mktemp) + # Ask the VPS what it considers valuable. Single source of truth: the list is + # defined once, in pangolin-backup's HOST_FILES, and never copied to this host. # shellcheck disable=SC2086 - rsync "${RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - --files-from="$LIST" "$VPS_SSH_HOST:/" "$BACKUP_ARCHIVE_ROOT/hostconfig/" \ - || warn "host config rsync reported errors" - if [[ $DRYRUN -eq 0 ]]; then - GOT=$(find "$BACKUP_ARCHIVE_ROOT/hostconfig" -type f | wc -l) - say " $GOT file(s) on disk" - [[ $GOT -eq $WANT ]] || warn "expected $WANT file(s), found $GOT -- a listed path may be missing on the VPS" + if ssh $SSH_OPTS "$VPS_SSH_HOST" 'pangolin-backup --list-files' 2>/dev/null | + sed 's|^/||' >"$LIST" && [[ -s $LIST ]]; then + WANT=$(wc -l <"$LIST") + say " $WANT path(s) advertised by the VPS" + [[ $DRYRUN -eq 0 ]] && mkdir -p "$BACKUP_ARCHIVE_ROOT/hostconfig" + # shellcheck disable=SC2086 + rsync "${RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ + --files-from="$LIST" "$VPS_SSH_HOST:/" "$BACKUP_ARCHIVE_ROOT/hostconfig/" || + warn "host config rsync reported errors" + if [[ $DRYRUN -eq 0 ]]; then + GOT=$(find "$BACKUP_ARCHIVE_ROOT/hostconfig" -type f | wc -l) + say " $GOT file(s) on disk" + [[ $GOT -eq $WANT ]] || warn "expected $WANT file(s), found $GOT -- a listed path may be missing on the VPS" + fi + else + warn "could not get the file list from the VPS (old pangolin-backup without --list-files?) -- skipping host config" fi - else - warn "could not get the file list from the VPS (old pangolin-backup without --list-files?) -- skipping host config" - fi fi # ---------------------------------------------------------------- 4. access logs @@ -230,47 +266,47 @@ fi # is a torn prefix that the next run would fetch again; the rotation at 00:00 UTC # is what makes a file eligible. Read the live file over SSH when analysing today. if [[ $LOGS -eq 1 ]]; then - say - say "--- access logs (plaintext, and the only copy past 400 days)" - if [[ $DRYRUN -eq 0 ]]; then - mkdir -p "$LOG_ARCHIVE_ROOT/traefik" "$LOG_ARCHIVE_ROOT/blog-legacy" || die "cannot create $LOG_ARCHIVE_ROOT" - # Query strings are logged in full, so the tree is as sensitive as the log is. - chmod 700 "$LOG_ARCHIVE_ROOT" - fi - # shellcheck disable=SC2086 - rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - "$VPS_SSH_HOST:$VPS_TRAEFIK_LOG_ARCHIVE/" "$LOG_ARCHIVE_ROOT/traefik/" \ - || warn "traefik log rsync reported errors" - # shellcheck disable=SC2086 - rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ - "$VPS_SSH_HOST:$VPS_BLOG_LOG_DIR/" "$LOG_ARCHIVE_ROOT/blog-legacy/" \ - || warn "blog legacy log rsync reported errors" - if [[ $DRYRUN -eq 0 ]]; then - LOGFILES=$(find "$LOG_ARCHIVE_ROOT" -type f | wc -l) - say " $LOGFILES log file(s) on disk" - # A pull that lands nothing looks identical to a pull with nothing new, and the - # difference is a broken path against a working one. Only the empty case is odd. - [[ $LOGFILES -gt 0 ]] || warn "no log files landed -- check VPS_TRAEFIK_LOG_ARCHIVE and VPS_BLOG_LOG_DIR" - fi + say + say "--- access logs (plaintext, and the only copy past 400 days)" + if [[ $DRYRUN -eq 0 ]]; then + mkdir -p "$LOG_ARCHIVE_ROOT/traefik" "$LOG_ARCHIVE_ROOT/blog-legacy" || die "cannot create $LOG_ARCHIVE_ROOT" + # Query strings are logged in full, so the tree is as sensitive as the log is. + chmod 700 "$LOG_ARCHIVE_ROOT" + fi + # shellcheck disable=SC2086 + rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ + "$VPS_SSH_HOST:$VPS_TRAEFIK_LOG_ARCHIVE/" "$LOG_ARCHIVE_ROOT/traefik/" || + warn "traefik log rsync reported errors" + # shellcheck disable=SC2086 + rsync "${LOG_RSYNC_OPTS[@]}" -e "ssh $SSH_OPTS" \ + "$VPS_SSH_HOST:$VPS_BLOG_LOG_DIR/" "$LOG_ARCHIVE_ROOT/blog-legacy/" || + warn "blog legacy log rsync reported errors" + if [[ $DRYRUN -eq 0 ]]; then + LOGFILES=$(find "$LOG_ARCHIVE_ROOT" -type f | wc -l) + say " $LOGFILES log file(s) on disk" + # A pull that lands nothing looks identical to a pull with nothing new, and the + # difference is a broken path against a working one. Only the empty case is odd. + [[ $LOGFILES -gt 0 ]] || warn "no log files landed -- check VPS_TRAEFIK_LOG_ARCHIVE and VPS_BLOG_LOG_DIR" + fi fi # ---------------------------------------------------------------- summary say if [[ $DRYRUN -eq 1 ]]; then - say "=== dry run complete -- nothing changed" - exit 0 + say "=== dry run complete -- nothing changed" + exit 0 fi ARCHIVES=$(find "$BACKUP_ARCHIVE_ROOT/pangolin" -name '*.enc' -type f | wc -l) SIZE=$(du -sh "$BACKUP_ARCHIVE_ROOT" 2>/dev/null | cut -f1) if [[ $LOGS -eq 1 ]]; then - LOGSIZE=$(du -sh "$LOG_ARCHIVE_ROOT" 2>/dev/null | cut -f1) - tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; $LOGFILES log file(s), $LOGSIZE in $LOG_ARCHIVE_ROOT" + LOGSIZE=$(du -sh "$LOG_ARCHIVE_ROOT" 2>/dev/null | cut -f1) + tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; $LOGFILES log file(s), $LOGSIZE in $LOG_ARCHIVE_ROOT" else - tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; logs skipped" + tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; logs skipped" fi if [[ $VERIFY_RC -ne 0 ]]; then - tell "=== COMPLETED WITH ERRORS -- see warnings above" - exit 1 + tell "=== COMPLETED WITH ERRORS -- see warnings above" + exit 1 fi say " Restore needs the key from 1Password; it is deliberately not stored here." exit 0 From 7dece2a5c8889a2bd46179be96527270733c9ee3 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 10:50:48 -0700 Subject: [PATCH 10/14] Record the pull as installed from the committed copy, byte for byte The reinstall after the shell-gate reformat restores byte-identity between /usr/local/bin/vps-backup-pull and ops/vps-backup-pull, so the running backup is verifiably the reviewed one rather than merely equivalent to it. It also exercised the installer's idempotent path for the first time: both generated config files reported already correct, only the script was replaced, and the summary took the unchanged branch. So applying a changed value means running the installer again rather than editing anything on the host. The timer's LAST is still `-` and the journal still holds one entry, so 2026-08-09 remains the first scheduled run, which is the whole point of never having started the service by hand. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 13ea461..28dda38 100644 --- a/TODO.md +++ b/TODO.md @@ -77,7 +77,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions -- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **Installed 2026-08-08 with `ops/install.sh`**, which derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env`, so nothing is typed twice. Verified after the fact rather than from the installer's own output: `systemd` resolves `User=pieter`, `Group=users` and `RequiresMountsFor=/data/backup` from the drop-in, and the environment file is `600 root:root`. **The committed script was reformatted for the shell gate after that install, so re-run `ops/install.sh` to restore byte-identity.** The difference is the formatting plus two `shellcheck` directives, confirmed by normalizing the running copy through `shfmt` and diffing, so the running backup is correct meanwhile. The root guard was exercised and refused. **The journal still holds exactly one entry and the timer's `LAST` is still `-`**, so installing did not spend the evidence that 2026-08-09 is the first scheduled run. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. +- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **Installed 2026-08-08 with `ops/install.sh`**, which derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env`, so nothing is typed twice. Verified after the fact rather than from the installer's own output: `systemd` resolves `User=pieter`, `Group=users` and `RequiresMountsFor=/data/backup` from the drop-in, and the environment file is `600 root:root`. The running script is byte-identical to the committed one. Re-running the installer after the shell-gate reformat also exercised its idempotent path, which reported both config files already correct and replaced only the script, so a changed value is applied by running it again rather than by editing anything on the host. The root guard was exercised and refused. **The journal still holds exactly one entry and the timer's `LAST` is still `-`**, so installing did not spend the evidence that 2026-08-09 is the first scheduled run. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. - **Where the rest of the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy were an operational asset built from another agent's instructions, and they existed only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. - **The pull itself is resolved and the reasoning is kept because it applies to everything still listed here.** What made it urgent was measured: the copy protected everywhere was the VPS's older one, while the copy that actually ran, carrying the log leg the review depends on, was in no snapshot and no repository. Committing it is what closed that, not the backup host's own off-site copy, which never reached the script. - **The directory holding it is named as though it were disposable.** `~/vps-backup-pull-patch` reads as a patch staged against a source, and there is no source: it is the most complete copy of the script in existence. A directory named for a temporary artifact is the one a cleanup deletes, and nothing here would notice until a restore produced the wrong script. From 899ee494939366670add0dd0fc99f2ad90622f55 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 11:03:17 -0700 Subject: [PATCH 11/14] Correct the release TODO.md named twice and disagreed with itself The State table said production serves 20260808-154717 and a bullet twelve lines below said 20260808-041050. The header on the live site says 154717, so the bullet was describing the first production deploy while reading as current. Rewritten to state what is served now, with the 9/9 host-side verification kept and attributed to the release it actually covered. Found by Copilot on PR #62. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 28dda38..29ba0f6 100644 --- a/TODO.md +++ b/TODO.md @@ -29,7 +29,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y ## Next, in dependency order - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. -- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` serves release `20260808-041050`, answering `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07. The host side verified it independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": +- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` serves release `20260808-154717`, answering `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, read from the `X-Blog-Release` header rather than from a pipeline's exit code. The host side verified the first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. From 5e3ef216c424ac52b7d89836ff243438b48f62fc Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 11:10:46 -0700 Subject: [PATCH 12/14] Fix four defects Copilot raised as low confidence, all four real Suppressed findings carry the real defects often enough that reading them is the rule here. All four reproduced. - A failing du aborted the run silently. `set -e` with `pipefail` makes the assignment fatal and 2>/dev/null hid the cause, so a pull that copied everything exited 1 at the summary with an empty journal. Reproduced against a missing path: no output, exit 1. The size is cosmetic, so du now reports to stderr and the run carries on with the transfer already done. - The generated /etc/vps-backup-pull.env still wrote SSH_OPTS unquoted, the same defect fixed in example.env one commit earlier and missed in the file that generates it. Harmless while only systemd reads it, and broken the moment a shell sources the same content. - The dry-run early exit printed its summary through say(), so `--quiet --dry-run` printed nothing at all, contradicting the flag's own promise of errors and the summary. It uses tell() now, which is what the summary path already did. - The header cited CONFIG.md as though it were in this repository. It is /root/CONFIG.md on the VPS, confirmed present there, so the reference now says which host holds it and points at the local equivalent. Co-Authored-By: Claude Opus 5 (1M context) --- ops/install.sh | 2 +- ops/vps-backup-pull | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/ops/install.sh b/ops/install.sh index 65ddb4c..c86aa15 100755 --- a/ops/install.sh +++ b/ops/install.sh @@ -108,7 +108,7 @@ VPS_TRAEFIK_LOG_ARCHIVE=$VPS_TRAEFIK_ARCHIVE VPS_BLOG_LOG_DIR=${VPS_BLOG_LOG_DIR:-/var/log/blog/legacy} BACKUP_ARCHIVE_ROOT=$BACKUP_ARCHIVE_ROOT LOG_ARCHIVE_ROOT=$LOG_ARCHIVE_ROOT -SSH_OPTS=-o ConnectTimeout=15 -o BatchMode=yes +SSH_OPTS="-o ConnectTimeout=15 -o BatchMode=yes" EOF ) diff --git a/ops/vps-backup-pull b/ops/vps-backup-pull index 5a93279..a6d0abd 100755 --- a/ops/vps-backup-pull +++ b/ops/vps-backup-pull @@ -22,8 +22,10 @@ # there rather than editing this file. See example.env and ENVIRONMENT.md. # # Why a PULL and not a push from the VPS: it keeps the VPS free of any credential -# reaching another system, which is the one security rule in CONFIG.md item 0. -# This host already has an SSH key trusted by the VPS; the VPS has none for here. +# reaching another system. That rule is item 0 of the VPS's own /root/CONFIG.md, which is +# on that host rather than in this repository, and OPERATIONS.md "Working With the VPS" +# states the same reasoning here. The backup host holds a key the VPS trusts, and the VPS +# holds none for here. # # What it fetches: # pangolin/ the encrypted archives -- the actual backup. Contains config/, @@ -293,13 +295,18 @@ fi # ---------------------------------------------------------------- summary say if [[ $DRYRUN -eq 1 ]]; then - say "=== dry run complete -- nothing changed" + # tell() rather than say(), because this IS the summary for a dry run, and --quiet + # promises errors and the summary. say() would make `--quiet --dry-run` print nothing. + tell "=== dry run complete -- nothing changed" exit 0 fi ARCHIVES=$(find "$BACKUP_ARCHIVE_ROOT/pangolin" -name '*.enc' -type f | wc -l) -SIZE=$(du -sh "$BACKUP_ARCHIVE_ROOT" 2>/dev/null | cut -f1) +# `set -e` plus `pipefail` makes a failing du fatal, and 2>/dev/null then hides why, so a +# pull that copied everything exits 1 at the summary with an empty journal. The size is +# cosmetic, so let du report to stderr and carry on with the transfer already done. +SIZE=$(du -sh "$BACKUP_ARCHIVE_ROOT" | cut -f1) || SIZE="size unavailable" if [[ $LOGS -eq 1 ]]; then - LOGSIZE=$(du -sh "$LOG_ARCHIVE_ROOT" 2>/dev/null | cut -f1) + LOGSIZE=$(du -sh "$LOG_ARCHIVE_ROOT" | cut -f1) || LOGSIZE="size unavailable" tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; $LOGFILES log file(s), $LOGSIZE in $LOG_ARCHIVE_ROOT" else tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; logs skipped" From e7e6b158c9def4de8af932c90ff48556a14bfb2c Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 11:18:54 -0700 Subject: [PATCH 13/14] Fix five more suppressed findings, all five real Second round of low-confidence findings, and the ratio held: every one reproduced. - ops/install.sh took ENV_FILE verbatim, so a relative name resolved against the caller's directory. deploy/make-release.sh in this same repo normalizes against the repo and refuses traversal, and the installer is the one that writes to /etc, so it now applies the same rule. Verified from /tmp and against a traversing name. - The header said paths are read from /etc/vps-backup-pull.env, which the script never opens. systemd loads it through EnvironmentFile, so a direct run picks up nothing unless the caller sources it first. - The header told the reader to schedule via a user timer rather than root's, while this PR ships a system timer. The account is what matters and the drop-in sets it, so the guidance now says that. - check-env-docs.py claimed to cover a value "declared or consumed anywhere". It reads example.env, workflow vars/secrets refs, and the KNOBS list. Widening it is not the fix, because a script-local name and a configuration value are the same shape, so the docstring now states the boundary and why it is there. - ops/README.md said a matching file is left alone. Both generated files are rewritten every run; the comparison is a report and the --force guard, not a skip. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-env-docs.py | 10 ++++++++-- ops/README.md | 2 +- ops/install.sh | 9 +++++++++ ops/vps-backup-pull | 13 ++++++++----- 4 files changed, 26 insertions(+), 8 deletions(-) diff --git a/checks/check-env-docs.py b/checks/check-env-docs.py index 922fe92..e2ed0f5 100755 --- a/checks/check-env-docs.py +++ b/checks/check-env-docs.py @@ -1,6 +1,12 @@ #!/usr/bin/env python3 -"""Fail if a configuration value is declared or consumed anywhere but not described in -ENVIRONMENT.md, or described there and declared nowhere. +"""Fail if a configuration value is declared without a description in ENVIRONMENT.md, or +described there and declared nowhere. + +The declared surface is three things, and nothing else: the keys in example.env, the +vars.X and secrets.X a workflow references, and the KNOBS list below. A variable a script +merely reads is NOT in scope, because a script-local name and a configuration value are the +same shape and no pattern separates them, so widening this would report the difference as +findings nobody can clear. A new knob therefore has to be added to KNOBS by hand. ENVIRONMENT.md is the single description of every configuration value. A new value gets added wherever its author is working, which is rarely the doc, and no linter notices a diff --git a/ops/README.md b/ops/README.md index c5ad1e2..6d16119 100644 --- a/ops/README.md +++ b/ops/README.md @@ -48,7 +48,7 @@ ops/install.sh # the same, then install **`--check` needs no root and writes nothing.** It prints both generated files, reports whether each would be created or already matches, and proves the VPS answers over SSH. Run it first. It reports "needs root to compare" rather than "already correct" when it cannot read an existing file, because an installer that claims agreement it could not verify is the failure this is written against. -Re-running is safe and is how a changed value is applied. A file that already matches is reported and left alone, and one that differs stops the run until `--force`. +Re-running is safe and is how a changed value is applied. Both generated files are rewritten every run, so the comparison is a report and a guard rather than a skip: a file that already matches says so, and one that differs stops the run until `--force`. **The timer's hour sits behind both producers on the VPS rather than beside them**, because the VPS rotates its log and writes its archive at times of its own. Pulling before the day's archive exists fetches the previous one and reports success, which is the failure mode that looks like a working backup. `Persistent=true` covers a host that was powered off when the timer should have fired. diff --git a/ops/install.sh b/ops/install.sh index c86aa15..9c6a51e 100755 --- a/ops/install.sh +++ b/ops/install.sh @@ -50,6 +50,15 @@ note() { printf ' %s\n' "$*"; } REPO=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) ENV_FILE=${ENV_FILE:-$REPO/secrets/local.production.env} +# Same rule deploy/make-release.sh applies, because the two read the same files and a name +# that means different things depending on where you stood is worse here: this one installs. +# A relative name resolves against the repo, so it means the same from any directory, and +# traversal is refused rather than resolved, since a relative name is meant to reach secrets/. +case "$ENV_FILE" in +/*) ;; +*..*) die "ENV_FILE must not traverse: $ENV_FILE" ;; +*) ENV_FILE="$REPO/$ENV_FILE" ;; +esac [[ $EUID -ne 0 ]] || die "do not run this under sudo -- run it as the account that will own the backup; it calls sudo for the steps that need it" [[ -f $ENV_FILE ]] || die "$ENV_FILE does not exist (ENV_FILE overrides which file is read)" diff --git a/ops/vps-backup-pull b/ops/vps-backup-pull index a6d0abd..f829c91 100755 --- a/ops/vps-backup-pull +++ b/ops/vps-backup-pull @@ -11,15 +11,18 @@ # RUN IT AS YOUR NORMAL USER, NOT WITH SUDO. It authenticates to the VPS with that # user's SSH key and writes to a tree that user owns. Under sudo it uses root's # identity, which the VPS does not trust, and starts mixing root-owned files into a -# user-owned backup tree. That is why it lives in bin/ rather than sbin/, and why -# the unit sets User=. Same applies when scheduling it: use the user's crontab or a -# user systemd timer, not root's. +# user-owned backup tree. That is why it lives in bin/ rather than sbin/, and why the +# shipped unit is a system timer that sets User= in a drop-in rather than running as root. +# A user timer or that account's crontab works equally well. What matters is the account, +# not which scheduler owns it. # # The source is ops/vps-backup-pull in the Blog repository. Edit it there and install # it. A copy found on a host is an installed artifact, not a place to make a change. # -# Paths below are read from /etc/vps-backup-pull.env, so a host states its own layout -# there rather than editing this file. See example.env and ENVIRONMENT.md. +# Paths below come from the environment, and this script reads nothing but the environment. +# Under the timer, systemd loads /etc/vps-backup-pull.env through the unit's EnvironmentFile. +# Run by hand, nothing loads it for you, so source an environment file first or the required +# values are unset. See example.env and ENVIRONMENT.md. # # Why a PULL and not a push from the VPS: it keeps the VPS free of any credential # reaching another system. That rule is item 0 of the VPS's own /root/CONFIG.md, which is From dea3bbdd8d7904bf50d32d79e61e36a70323a865 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 11:25:22 -0700 Subject: [PATCH 14/14] Refuse a root path that would chmod 700 the filesystem Both roots are chmod 700'd and written into, so a wrong value is damage to the host rather than a failed backup. `/` is the case that matters: `chmod 700 /` locks every other account out, and nothing downstream would have refused it. The preflight only checked that the value was non-empty. Now checked: absolute, not `/`, and no trailing slash. A relative value is refused for the reason the deploy tooling refuses one, since it means a different directory depending on where the caller stood, and under the timer that is systemd's working directory. LOG_ARCHIVE_ROOT is checked only when the log leg runs, so --no-logs is unaffected. Also fixes --check, which died on the first differing file. It is documented as validate-and-print-touching-nothing, so stopping there hid the second file and the reachability test behind it. The --force requirement belongs to the install path and now applies only there. Raised by Copilot on PR #62. Co-Authored-By: Claude Opus 5 (1M context) --- ops/install.sh | 7 ++++++- ops/vps-backup-pull | 15 +++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/ops/install.sh b/ops/install.sh index 9c6a51e..f05f365 100755 --- a/ops/install.sh +++ b/ops/install.sh @@ -157,7 +157,12 @@ for pair in "$ENV_DEST|$ENV_BODY" "$DROPIN_DEST|$DROPIN_BODY"; do else printf '=== %s: differs\n' "$path" printf '%s\n' "$body" | sudo -n diff -u "$path" - || true - [[ $FORCE -eq 1 ]] || die "$path exists with different contents -- re-run with --force to replace it" + # --force is an install-path guard, not a reporting one. --check exists to show what + # would happen, so stopping at the first differing file would hide the second one and + # the reachability test behind it. + if [[ $CHECK -eq 0 && $FORCE -eq 0 ]]; then + die "$path exists with different contents -- re-run with --force to replace it" + fi changed=1 fi done diff --git a/ops/vps-backup-pull b/ops/vps-backup-pull index f829c91..b5b01eb 100755 --- a/ops/vps-backup-pull +++ b/ops/vps-backup-pull @@ -174,6 +174,21 @@ LOG_RSYNC_OPTS=("${RSYNC_OPTS[@]}") [[ -n $VPS_SSH_HOST ]] || die "VPS_SSH_HOST is not set -- see /etc/vps-backup-pull.env (template: example.env)" [[ -n $BACKUP_ARCHIVE_ROOT ]] || die "BACKUP_ARCHIVE_ROOT is not set -- see /etc/vps-backup-pull.env (template: example.env)" [[ $LOGS -eq 0 || -n $LOG_ARCHIVE_ROOT ]] || die "LOG_ARCHIVE_ROOT is not set and the log leg is enabled -- set it, or pass --no-logs" + +# Both roots are chmod 700'd and written into, so a wrong value here is not a failed backup, +# it is damage to the host. `/` is the case that matters: `chmod 700 /` locks every other +# account out of the filesystem, and nothing downstream would refuse it. A relative value is +# refused for the same reason it is refused in the deploy tooling, since it means a different +# directory depending on where the caller stood, and under the timer that is systemd's cwd. +check_root() { + local name=$1 value=$2 + [[ $value == /* ]] || die "$name must be an absolute path, got: $value" + [[ $value != "/" ]] || die "$name must not be / -- this directory is chmod 700'd and written into" + [[ $value != */ ]] || die "$name must not end in a slash, got: $value" +} +check_root BACKUP_ARCHIVE_ROOT "$BACKUP_ARCHIVE_ROOT" +[[ $LOGS -eq 0 ]] || check_root LOG_ARCHIVE_ROOT "$LOG_ARCHIVE_ROOT" + command -v rsync >/dev/null || die "rsync not installed on this host" START=$(date -u '+%Y-%m-%d %H:%M:%S UTC')