From d73e49a62eac43b49d78df7c9499b751d74d9e02 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 10:13:17 -0700 Subject: [PATCH 1/4] Filter the log by the header, and say a hand probe has to carry it (#81) * Filter the log by the header, and say a hand probe has to carry it The outward pass told a reader to separate our traffic from real visitors by user agent, which is the coincidence X-Blog-Check was added to replace: the CI runner's curl and the host's own curl are byte-identical, so only the client address separates them and the CI half rotates every run. The header is the mechanism now, and its value carries provenance, so a run is identifiable rather than merely excludable. The gap worth recording is that check-live-urls.sh sends it and a bare curl does not, so an interactive probe lands in the visitor set unless whoever runs it passes -H. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window, measured on the host side. The user-agent method stays, scoped to what it can still read: the days logged before the field existed. Kept as sub-bullets under the first filter rather than as four more bold paragraphs, because the section opens by saying it is four filters and a reader has to be able to count them. Co-Authored-By: Claude Opus 5 (1M context) * Show real tag values, and say the field depends on the edge logging it The examples were angle-bracket placeholders, and the tag is validated: exactly one slash, letters, digits, dot, underscore, hyphen. Pasting one of those literally is refused, so the section now shows values that work and says the shape is enforced rather than conventional. The larger omission is that the filter depends on something this repository does not own. `request_X-Blog-Check` appears only because the edge is configured to log that header, so an absent field means either an untagged request or a capture that stopped, and the log alone does not distinguish them. Reading a day of absence as a day of real traffic is the wrong answer that shape produces. Both found by review, which is also the general form the host side named: before believing a negative, establish that the check could have produced a positive. Co-Authored-By: Claude Opus 5 (1M context) * Put a pasteable value in the pasteable command The prose examples were corrected and the curl one-liner was not, which is the one a reader actually copies. It now carries proxmox/media-dev. ENVIRONMENT.md keeps its angle brackets deliberately. There they describe the enforced grammar and the value the script derives under Actions, neither of which anyone types, and the settable example beside them is already a real one. Found by Copilot review on #81, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) * Claim only what the tagging actually covers The lead said every synthetic request announces itself, and the bullet below it said a hand curl sends nothing unless someone adds the header. The section contradicted itself in the direction that flatters the filter, which is the direction that produces a wrong answer nobody checks. It now says the scripted checks send it on every request they make, and that the filter is only as complete as the tagging is. Found by Copilot review on #81, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) * Say which half of the tag carries the purpose "Naming the purpose rather than the host" implied the whole value is a purpose label, when proxmox is the source half and only the id half is free. A reader following that would have written something like media-dev/probe, inverting the grammar the section just described. Found by Copilot review on #81, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- OPERATIONS.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index 22b83b8..efcb837 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -265,7 +265,15 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access The outward pass is four filters over the edge log, and each one exists because skipping it produced a wrong answer once. -**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. Filter on user agent: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0` and the 711 that remained are the entire real dataset. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. +**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. + +The mechanism is the `X-Blog-Check` request header, which the scripted checks send on every request they make, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the filter. It is only as complete as the tagging is, which is the first bullet below. Its value is a source and an id rather than a boolean, so a run is identifiable rather than merely excludable, and real values look like `github/31322640628-1` from CI, `vps/smoke` from the host side, and `proxmox/media-dev` from here. The shape is enforced by `check-live-urls.sh`, which takes exactly one `/` and only letters, digits, `.`, `_`, `-`, so a placeholder written with angle brackets is a description rather than something to paste. + +- **The field exists only because the edge is configured to log that header**, which is the host side's to hold and not this repository's. An absent field therefore has two meanings, an untagged request or a capture that stopped, and they are not distinguishable from the log alone. Confirm the capture is live before reading a day's absence as a day of real traffic. + +- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/media-dev"`. The source half stays `proxmox`, which is where the probe came from, and the id half is where the purpose goes. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. +- **Absence is not proof of a human**, since a scanner sends no header either, so this pairs with the scanner-shape filters below rather than replacing them. The field is forgeable and must never reach auth, rate limiting, robots handling, or caching. +- **Before 2026-08-09 the log carries no such field**, and user agent is the only key for those days: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0`, leaving 711 real ones. That key is a coincidence rather than a rule, since the CI runner's curl and the host's are byte-identical and only the rotating client address separates them, which is why the header exists. **A referer does not implicate this site unless it points somewhere else.** The rule worth applying is that a 404 carrying a referer is a broken link and a 404 without one is a typed or probed address, and it fails on scanners, which set `Referer` to the request URL itself. Every one of the 36 referer-bearing site-host 404s on 2026-08-08 was self-referential, so the unrefined rule reported three dozen broken links on a site that had none. Discard the matches before counting, and **normalize the scheme rather than comparing it**, because a scanner reaching an HTTPS site routinely sends an `http://` referer for the same address. Comparing against the request's own scheme therefore matches nothing and leaves every false positive in place: on the 2026-08-08 data the naive form kept all 36 where the normalized form kept none. From 77f5783bc8212bcd4672572ea7ec9793bb16f1ad Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 10:41:13 -0700 Subject: [PATCH 2/4] State the negative-result trap as a demand for evidence (#82) * State the negative-result trap as a demand for evidence The trap said a query matching nothing reads as a clean result, which is true and names no action. The host side put the general form better: before believing a negative, establish that the check could have produced a positive. That is the same rule with a step in it, and the step is what was missing every time this cost something. Four costumes are now listed under it, because seeing them as one failure is the whole value: a query that cannot see its target, including a journal grep run from an account not in adm or systemd-journal, where the lines exist and are simply not shown to it a rule naming a target that does not exist, which is the .gitattributes pin for two files this repository has never carried a filter whose precondition is unstated, which is reading an absent X-Blog-Check as a visitor when the field exists only because the edge is configured to log it a tool reporting success having done nothing, which is git restore-mtime v2022.12 printing a count and processing none of it No count is claimed for them. An earlier draft said six false passes and the list does not add to six, some were near-misses caught in time, and one has not happened at all. Two State rows are re-measured while here, since both had gone false today. Production serves a new release and the hard-linking result now exists, 1,052 of 3,275 with every linked media file at 644. The backup timer fired unattended at 09:11:01 UTC, proven by both halves of the standard rather than either alone. The serving release id is no longer repeated outside the State table. A value that moves with every deploy should exist in one place, and that duplicate had already gone stale twice in a day. Co-Authored-By: Claude Opus 5 (1M context) * Match the file on a code literal and a tool name The file mode was plain text beside literals the same table wraps in code, and the tool was named by its subcommand invocation where the two nearby mentions use the binary name. Both now read the same way throughout. The distinction is worth keeping straight rather than merely consistent: git-restore-mtime is the tool, and git restore-mtime is one of the two ways it resolves. This sentence is about the tool. Found by Copilot review on #82. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- TODO.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/TODO.md b/TODO.md index 055fe2f..7628b51 100644 --- a/TODO.md +++ b/TODO.md @@ -19,8 +19,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical | | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | -| VPS production | serving release `20260809-030521` at `blog.insanegenius.net` as of 2026-08-09, deployed from `main` after the promotion and read back from the live `X-Blog-Release` header rather than from the run's exit code. `200` unauthenticated, `X-Robots-Tag: noindex, nofollow` still set for the rehearsal, and the live check's 1,253 URLs verified against the running site, which is the 328 + 917 + 8 subset described in the URL contract row rather than a different contract. That release is **the first built with the mtimes restored**, so the first whose static tree can arrive as hard links, and therefore the first where a bad file mode could ride one forward into every later release. The shared-inode count is server-side, so ask the host for it rather than inferring it here. **M7a, the first production deploy, was 2026-08-08 on release `20260808-154717`**, which established the unauthenticated `200`, the `.net` sitemap line in `/robots.txt`, and the gallery fix. DNS for the public name is still on the old platform | -| Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has run once by hand, the outward pass only: it read 2026-08-08 traffic and found nothing to add to the URL contract, the inward pass has not run, and neither pass is on the cadence **Recurring operations** sets | +| VPS production | serving release `20260809-163217` at `blog.insanegenius.net` as of 2026-08-09, deployed from `main` and read back from the live `X-Blog-Release` header rather than from the run's exit code. `200` unauthenticated, `X-Robots-Tag: noindex, nofollow` still set for the rehearsal, and the live check's 1,253 URLs verified against the running site, which is the 328 + 917 + 8 subset described in the URL contract row rather than a different contract. **Hard-linking works and the host measured it**: 1,052 of 3,275 files shared with the previous release, the linked set being exactly the restored generation, and all 1,048 linked media files at mode `644`. A release now costs 11.4 MB incremental against 585 MB for a full copy. Staging reproduced the same 1,052 independently. **M7a, the first production deploy, was 2026-08-08 on release `20260808-154717`**, which established the unauthenticated `200`, the `.net` sitemap line in `/robots.txt`, and the gallery fix. DNS for the public name is still on the old platform | +| Operations | the backup half is proven on its schedule; the review half is not. **The off-host log pull fired unattended on 2026-08-09 at 09:11:01 UTC**, inside its randomized window, proven by both halves of the standard: a second journal entry and `LAST` moving off `-`. It copied more than the hand run, 52 archives against 42. The periodic log review has run once by hand, the outward pass only: it read 2026-08-08 traffic and found nothing to add to the URL contract, the inward pass has not run, and neither pass is on the cadence **Recurring operations** sets | ## Blocked on the maintainer @@ -30,7 +30,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y ## Next, in dependency order - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. -- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` answers `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, with the serving release read from the `X-Blog-Release` header rather than from a pipeline's exit code. **It serves `20260809-030521` today; M7a itself was `20260808-154717`**, and the State table above is the current value, so read it there rather than from this paragraph. The host side verified that first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": +- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` answers `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, with the serving release read from the `X-Blog-Release` header rather than from a pipeline's exit code. **M7a itself was `20260808-154717`. What it serves today is in the State table above and is deliberately not repeated here**, since a release id moves with every deploy and a second copy of it goes stale by the afternoon. The host side verified that first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. @@ -93,7 +93,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions -- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **Installed 2026-08-08 with `ops/install.sh`**, which derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env`, so nothing is typed twice. Verified after the fact rather than from the installer's own output: `systemd` resolves `User=pieter`, `Group=users` and `RequiresMountsFor=/data/backup` from the drop-in, and the environment file is `600 root:root`. The running script is byte-identical to the committed one. Re-running the installer after the shell-gate reformat also exercised its idempotent path, which reported both config files already correct and replaced only the script, so a changed value is applied by running it again rather than by editing anything on the host. The root guard was exercised and refused. **The journal still holds exactly one entry and the timer's `LAST` is still `-`**, so installing did not spend the evidence that 2026-08-09 is the first scheduled run. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. +- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **Installed 2026-08-08 with `ops/install.sh`**, which derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env`, so nothing is typed twice. Verified after the fact rather than from the installer's own output: `systemd` resolves `User=pieter`, `Group=users` and `RequiresMountsFor=/data/backup` from the drop-in, and the environment file is `600 root:root`. The running script is byte-identical to the committed one. Re-running the installer after the shell-gate reformat also exercised its idempotent path, which reported both config files already correct and replaced only the script, so a changed value is applied by running it again rather than by editing anything on the host. The root guard was exercised and refused. Installing did not spend the evidence for the first scheduled run, which then arrived: **the timer fired unattended on 2026-08-09 at 09:11:01 UTC**, with a second journal entry and `LAST` off `-`. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. - **Where the rest of the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy were an operational asset built from another agent's instructions, and they existed only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. - **The pull itself is resolved and the reasoning is kept because it applies to everything still listed here.** What made it urgent was measured: the copy protected everywhere was the VPS's older one, while the copy that actually ran, carrying the log leg the review depends on, was in no snapshot and no repository. Committing it is what closed that, not the backup host's own off-site copy, which never reached the script. - **The directory holding it is named as though it were disposable.** `~/vps-backup-pull-patch` reads as a patch staged against a source, and there is no source: it is the most complete copy of the script in existence. A directory named for a temporary artifact is the one a cleanup deletes, and nothing here would notice until a restore produced the wrong script. @@ -157,7 +157,11 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **Do not name any workflow `build-*-task.yml`** while the repo declares `source-only`, since `detect` is literally `["no build-*-task.yml"]`. - **Do not edit `.markdownlint-cli2.jsonc`, `repo-config/configure.sh`, or the two ruleset payloads.** They are carried verbatim and byte-matched against the hub. Scope a glob in the workflow instead. A reviewer finding a real defect in one of them is answered by declining locally and filing it at the hub, never by editing the file to satisfy the review. - **The hub's `main` can promote while a convergence pull request is open**, so ground truth moves underneath work that was correct when it started. It happened twice in one session on 2026-08-03, and the second time added drift the branch could not have known about. Re-run the audit against the hub ref actually carried before claiming convergence, and name that ref in the change, or the claim ages into a false one. -- **A query that matches nothing reads as a clean result.** It has cost three separate false passes: a review-thread poll that could not see suppressed findings, a reviewer filter written in the wrong API's login form, and an audit loop whose `jq` path had moved. Each returned empty, and empty looked like nothing to report. Assert the query matched before reading what it returned, which is what `jq -e` and a non-empty check are for. +- **Before believing a negative, establish that the check could have produced a positive.** An empty result and a clean result are the same bytes, so the question is never "did it find anything" but "could it have". The instances below are the same failure in different costumes, some of which cost a false pass and one of which has not happened yet. It is stated as a demand for evidence rather than as a warning to be careful because *be careful* has no step to perform and *prove the check can fire* does. + - **A query that cannot see its target.** A review-thread poll blind to suppressed findings, a reviewer filter written in the wrong API's login form, an audit loop whose `jq` path had moved, and a journal grep for `Started` from an account not in `adm` or `systemd-journal`, where the lines exist and are not shown to it. Assert the query matched before reading what it returned, which is what `jq -e` and a non-empty check are for. + - **A rule naming a target that does not exist.** `.gitattributes` pinned two paths this repository has never carried, and the comment above them claimed the case was covered, so the one file that needed the pin sat unpinned twenty lines up. [`checks/check-eol-pins.py`](./checks/check-eol-pins.py) gates that file now; the shape is not gated anywhere. + - **A filter whose precondition is unstated.** The outward pass reads an absent `X-Blog-Check` as a visitor, and the field exists only because the edge is configured to log it, so a stopped capture reads as a day of real traffic. Recorded in [`OPERATIONS.md`](./OPERATIONS.md) beside the filter. + - **A tool that reports success having done nothing.** `git-restore-mtime` v2022.12 printed `1,052 files to be processed`, processed none, and exited 0 for four releases. The remedy is the same in kind: assert the outcome rather than the exit status, which is what both release paths now do. - **The hub authors `scripts/pr_review.py`, and hand-rolling the review loop re-discovers its bugs.** One `status` call reports rounds, head coverage, unresolved threads, suppressed findings across every round, and whether a request was ever picked up. `wait` runs the backoff in-process, so a review wait costs one turn rather than one per poll. It is read-only by design and the mutations stay explicit, so fetch and run it rather than reimplementing it. Its README documents the traps below as the reason it exists. - **A review request can sit forever without being picked up, which looks exactly like patience.** Copilot raises a `copilot_work_started` timeline event within about half a minute of accepting; a request that never draws one is not slow, it is inert, and elapsed time cannot tell them apart. The event is REST-only. Recover by clearing the request with `union: false` and an empty `botIds`, then requesting again, after reading the pending set so a human reviewer is not dropped. - **The Copilot reviewer's login differs by API, and a wrong-form filter reads as a clean review.** REST reports `copilot-pull-request-reviewer[bot]`, GraphQL omits the suffix. A filter written in the other form matches nothing, and an empty result is indistinguishable from no findings. Assert the filter matched before trusting what it returned. From 2c8d4beed1fc98f302a86c2f1f66c6170f1a23f7 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 11:07:13 -0700 Subject: [PATCH 3/4] Add X-Blog-Check to the keep-list this document already describes (#84) * Add X-Blog-Check to the keep-list this document already describes The access-log section said request headers are dropped except Referer and User-Agent. The host added X-Blog-Check to that allowlist when the header was introduced, which is the only reason request_X-Blog-Check appears at all, so the outward pass and the log description contradicted each other in the same file. The keep-list is named as the precondition rather than left as trivia. A header absent from it does not appear in the log, which reads exactly like a request that never sent one, and that is the negative-result trap this file now states as a rule. The reason the list is an allowlist is unchanged and still the point: it is what keeps the Pangolin resource access token out of a file that is retained and copied off the host. Found by Copilot review on #83. Co-Authored-By: Claude Opus 5 (1M context) * Say the log omits headers rather than that headers are dropped "Request headers are dropped except" reads as Traefik stripping them from the request. That would mean the Pangolin access token never reaches the site, and the staging gate works, so a reader who took it that way would be left with two facts that cannot both hold. It now says the log records three and omits the rest from the record rather than from the request, which still arrives intact. Found by Copilot review on #84. Co-Authored-By: Claude Opus 5 (1M context) * Name the log keys, since that is what a reader greps for The sentence named the headers where the rest of the section names the JSON keys, so it described the policy without giving the thing anyone parsing the file actually needs. The three now appear as request_Referer, request_User-Agent and request_X-Blog-Check, which is the form already used in the jq examples above and below it. Found by Copilot review on #84, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- OPERATIONS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index efcb837..fe3d680 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -299,7 +299,7 @@ Widen `== 404` to `>= 400` for the whole non-200 sweep the table above describes ### Retention Is the Prerequisite, and It Belongs to the Host -**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=` traffic is visible as itself. Request headers are dropped except `Referer` and `User-Agent`, which is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. +**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=` traffic is visible as itself. The log records only three request headers, which appear as the keys `request_Referer`, `request_User-Agent`, and `request_X-Blog-Check`, and omits the rest from the record rather than stripping them from the request, which still arrives intact. That omission is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. That keep-list is the host's to hold and is the precondition the outward pass depends on: a header absent from it does not appear in the log at all, which is indistinguishable from a request that never sent one. **That log rotates and is eventually deleted, on a schedule the host sets and can change.** The window is long, and it is finite, so anything the inward pass depends on has to be copied off the host before the archive ages out. Read the current retention from the host rather than from this file, because a number written here is a number nothing checks. From 7a40d5deda89aad93ef5caf013e88bea4893d71d Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 11:26:55 -0700 Subject: [PATCH 4/4] Drop a count that only stayed true by being dated (#85) * Drop a count that only stayed true by being dated The HUGO_BASEURL item said a wrong base URL would have run 1,245 requests at the live site. That was the live check size on 2026-08-07 and the sentence is a counterfactual about that day, so it was defensible, and leaving it meant a reader validating a deploy met 1,245 and 1,253 in one file with nothing connecting them. The number was never load-bearing there. The point is that the check would have been aimed at the old platform, which the sentence makes without it. Same reasoning as removing the duplicated release id: a value that has to be dated to stay true is better removed than dated, because the next reader has no way to know which vintage they are holding. Found by Copilot review on #83, as a suppressed comment, having been declined once on the grounds that the counterfactual was accurate. Co-Authored-By: Claude Opus 5 (1M context) * Stop the live check miscounting itself in its own comments Two comments said the script runs 1,245 requests. It runs 1,253: the 1,245-URL contract plus the eight media probes added when the live media check landed. The file describing what it does was the last thing to know what it does. Neither states a number now. One says it runs one request per URL checked and the other says one request before the rest, which is what each comment was actually about, and neither goes stale when a list grows. The three other occurrences in the repository are correct and are left alone. 1,245 is the URL contract, 328 render plus 917 redirect, and 1,253 is what the live check requests because it adds eight media URLs. The blog post contrasting 1,245 with the sitemap 111, the audit report quoting a 2026-08-04 run, and the parity note about a build baked with the wrong host all mean the contract and all say so correctly. Found by Copilot review on #85, which caught the PR description claiming 1,245 appeared nowhere when it had only been grepped out of two files. Co-Authored-By: Claude Opus 5 (1M context) * Say every request carries it, since a URL is not one request The comment claimed one invocation per URL checked. A render check is one curl, a redirect check is three, and the media check reaches six, so the count was wrong in the direction that understates the exposure the comment exists to explain. It now says every request would carry the token, which is the actual reason the credential goes in a config file rather than on a command line, and which no list or code path can make stale. Found by Copilot review on #85, as a suppressed comment, on wording I had introduced one commit earlier to remove a different stale count. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- TODO.md | 2 +- checks/check-live-urls.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/TODO.md b/TODO.md index 7628b51..86634c0 100644 --- a/TODO.md +++ b/TODO.md @@ -31,7 +31,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. - **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` answers `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, with the serving release read from the `X-Blog-Release` header rather than from a pipeline's exit code. **M7a itself was `20260808-154717`. What it serves today is in the State table above and is deliberately not repeated here**, since a release id moves with every deploy and a second copy of it goes stale by the afternoon. The host side verified that first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": - - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** + - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run the whole live check at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. - **`robots.txt` is decided, built, and deployed, 2026-08-08.** The site emits one, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. Verified from the served bytes on release `20260808-154717`: `/robots.txt` answers 200 advertising `https://blog.insanegenius.net/sitemap.xml`, `/robots.txt/` 301s to it, and `sitemap.xml` carries 312 `.net` URLs and zero `.com`. The record below is kept because the reasoning is what the next decision about crawl directives will need. diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index e37f12d..feb66b3 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -114,7 +114,7 @@ echo "==> tagging requests X-Blog-Check: $CHECK_TAG" # A resource access token opens the proxy's auth gate. # It goes into a curl config file because bash cannot export an array to the parallel checks. -# A command line is also world-readable in ps output, and this runs 1,245 of them. +# A command line is also world-readable in ps output, and every request would carry it. if [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] && [ -n "${PANGOLIN_ACCESS_TOKEN:-}" ]; then # Same hazard as CHECK_TAG above and the same reason, but a narrower rule, because the # grammar of a credential is the issuer's to define and not this script's. Only the @@ -269,7 +269,7 @@ export BASE FAILED CURLRC CHECKRC echo "==> $BASE" -# One request before the 1,245, because an auth gate turns a bad credential into a total failure. +# One request before the rest, because an auth gate turns a bad credential into a total failure. # Otherwise the output reads as a vanished site rather than a wrong token. # Transport failures are separated from HTTP ones, since a name that does not resolve otherwise # reports as a status code and gets diagnosed as a credential or a symlink.