From 00db1580f206ac08d3f4c91ec699d3da537064c0 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Thu, 6 Aug 2026 20:28:19 -0700 Subject: [PATCH] Document the periodic log review, in both directions Real traffic is the only source that finds what every check here is blind to. The URL contract proves the 328 URLs someone thought to list and the 917 redirects derived from the export, and it cannot know about a URL nobody recorded, because the crawl that produced the lists is the crawl the lists are checked against. A visitor following a sixteen-year-old link is the one reader who tests that. This was discussed when the migration was designed and had survived only as a one-line first-week task, which is not a periodic process. Both directions are recorded, because they are the same two the media checks read and they are blind to each other for the same reason. Outward finds a non-200 and names content that may be missing, which is the one with an action: add the URL to the golden list and add a redirect. Inward subtracts every URL that has ever answered 200 from the set the site builds, and names content no reader has reached. That one is deliberately slow, since a post can go a year without a visit and still be worth keeping, and its first concrete use is the 97 carried media files the old platform never published: if nothing requests them across a year, that settles whether carrying them is preservation or clutter, which no reasoning from this repo can settle. The three tiers are named, because a request crosses the proxy before it reaches the site and no single log answers both questions. Traefik, and Pangolin's Traefik on the VPS, see every request reaching the host including the ones aimed at names this site does not serve. Pangolin sees what the auth gate rejected. Caddy sees only what arrived. So a 404 count taken from Caddy alone is a floor rather than a total, since a request the edge refused is a reader who found nothing just as surely and appears in no Caddy log. Two parsing properties are stated because both silently produce nothing. The Caddy access log is `format console`, so a line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON, and a parser assuming one object per line reads zero records. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, without which every request appears to come from one internal address. Retention is recorded as the unconfigured prerequisite it is. The containers use Docker's `json-file` driver with no rotation and no size limit, so the log grows without bound and is discarded when the container is recreated. The first is a disk risk and the second silently resets the window the inward pass depends on being long. It belongs to the host rather than to this repo, the same split Retention and Who Owns What already describe for release pruning, so it is recorded and dated to the cutover rather than changed here. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 35 +++++++++++++++++++++++++++++++++++ TODO.md | 3 ++- 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index 5ef1031..1c35786 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -171,6 +171,41 @@ The script asserts both halves of that rather than assuming them. It fails when **Nothing prunes on the deploy path, and that is what keeps the deploy key's capability small.** A prune racing a deploy could take the rollback target, where a lingering release only costs disk. This is also why the key needs no delete capability, which is the property "Server Hardening" depends on. The count and the timer belong to the host, so this section records what the host declares rather than holding a second copy of it. See "Who Owns What". +## Log Review + +**Real traffic is the only source that finds what every check here is blind to.** The URL contract proves the 328 URLs someone thought to list and the 917 redirects derived from the export. It cannot know about a URL nobody recorded, because the crawl that produced the lists is the same crawl the lists are checked against. A visitor following a sixteen-year-old link is the one reader who tests that. + +Review runs in both directions, which are the same two the media checks read and have the same blind spots for the same reason. + +| Direction | Question | Signal | Cadence | +| --- | --- | --- | --- | +| Outward | What did someone ask for that is not here? | non-200 responses | daily for the first week after cutover, then monthly | +| Inward | What is here that nobody has ever asked for? | URLs absent from every 200 | quarterly at the earliest, and a long tail by nature | + +**The outward pass is the one with an action.** A 404 on a path shaped like real content means the golden list missed a URL: add it to [`checks/golden-urls.txt`](./checks/golden-urls.txt) and add a redirect, per that file's own maintenance rules. Expect the raw counts to be dominated by scanners probing for `wp-login.php`, `.env`, and `.git/config`, which is noise from a site that used to run WordPress and should be filtered by shape rather than investigated. + +**The inward pass answers a question nothing else can.** Subtracting every URL that has ever returned 200 from the set the site builds names the content no reader has reached. It is slow evidence and deliberately so, since a post can go a year without a visit and still be worth keeping. Its first concrete use is the 97 carried media files that no page links and that the old platform never published, recorded in [`checks/README.md`](./checks/README.md): if nothing requests them across a year, that settles whether carrying them is preservation or clutter, and no reasoning from the repository alone can settle it. + +### The log is three tiers, and each is blind to something + +A request crosses the proxy before it reaches the site, so no single log answers both questions. + +| Tier | Sees | Cannot see | +| --- | --- | --- | +| Traefik, or Pangolin's Traefik on the VPS | every request reaching the host, including unknown hostnames, TLS failures, and traffic aimed at names this site does not serve | which release answered, or anything about the site's own routing | +| Pangolin, on the VPS only | requests the auth gate rejected | anything on the local mirrors, which have no gate | +| Caddy, per environment | path, status, and the `X-Blog-Release` that answered | anything the tiers above rejected, which never arrives | + +**A 404 count taken from Caddy alone is therefore a floor, not a total.** A request the edge refused is a reader who found nothing just as surely, and it appears in no Caddy log. Read the edge for what never arrived and Caddy for what arrived and failed, and treat the two as one answer. + +Two properties of the Caddy side worth knowing before parsing it. Its access log is `format console`, so each line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON itself, and a parser that assumes one object per line reads nothing. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, which is the same setting "Serving" describes as a security boundary — without it every request in the log appears to come from one internal address and the inward pass cannot distinguish a reader from a health check. + +### Retention is the prerequisite, and it is not configured + +**The containers log to Docker's `json-file` driver with no rotation and no size limit.** `/etc/docker/daemon.json` sets only a storage driver, so the built-in defaults apply: the file grows without bound, and it is discarded entirely when the container is recreated. Both halves defeat a periodic review — the first is a disk risk on a VPS, and the second silently resets the window that the inward pass depends on being long. + +Nothing here fixes that, because the log driver belongs to the host rather than to this repository, the same split "Retention" and "Who Owns What" describe for release pruning. **Settle it before the first review is expected to mean anything**, since a review reads whatever survived and reports confidently on it either way. + ## Who Owns What The site and the server it runs on are maintained separately, so the boundary is written down rather than inferred. This repo owns the artifact and what proves it correct; the host owns where a release may be written and what happens to it afterwards. diff --git a/TODO.md b/TODO.md index ba83389..7810bf5 100644 --- a/TODO.md +++ b/TODO.md @@ -31,7 +31,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. - **Deploy production once, to a name that is not the live one.** The production environment is configured and its Pangolin resource is deliberately disabled, so nothing has ever run against it. Validate there before the record moves. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. -- Watch server logs for 404s daily for the first week, because real traffic finds what the golden list missed. Append anything new to `checks/golden-urls.txt` and add a redirect. +- **Configure log retention before the cutover, or the review below reads whatever happened to survive.** The containers use Docker's `json-file` driver with no rotation and no size limit, so the log grows without bound and is discarded when the container is recreated. It belongs to the host rather than to this repo, like release pruning. +- Watch server logs for non-200s daily for the first week, then monthly, because real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site.