From cd9285a9fea0b7f662391a0efe3c236b963473e9 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 19:45:23 -0700 Subject: [PATCH 1/6] Record the promotion, and retire a done item the file still lists as next (#71) State drifts silently, so this is the state after the 2026-08-09 promotion rather than new work. The mtime restore was listed under "Next, in dependency order" and has been in deploy-site-task.yml since #65. It is marked done, with the measurement kept, and with the part that is still ahead named: the next production deploy is the first to exercise it, and therefore the first where a badly moded file can ride a link into every later release. The live media check landed before it for that reason. A Branches row records that main carries every commit on develop with an empty content diff, and that nothing published, since a promotion that publishes nothing looks identical to one that failed to. ProjectTemplate#633 is added to the hub table, and "Owed to the hub" no longer reads "Nothing": porting the line-ending gate into the hub's scripts/ is offered there and is owed only if the hub wants that shape. It is marked do-not-start, because the open design question changes the code rather than merely delaying it. Three traps, each hit this session and none of them mechanically preventable: a review existing on the head is not a finished review, which is how a green report went out over an open finding a rule naming a target that does not exist reads as coverage, which is what hid the unpinned script a gate is only as good as its matcher, and a wrong matcher fails quietly, which is why check-eol-pins.py now compares itself against git check-attr rather than against the documentation And one new item: checks/README.md is the authority for its directory since #66 and names one of the three gates in it. Co-authored-by: Claude Opus 5 (1M context) --- TODO.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/TODO.md b/TODO.md index e5dc8ec..7d86384 100644 --- a/TODO.md +++ b/TODO.md @@ -12,7 +12,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Media integrity | closed for the files themselves. The conversion's only loss was 19 images, inside five galleries, restored from the capture, and the 98 files with no page links are adjudicated rather than unknown, with `ORPHANED_MEDIA` failing if that count moves either way. How media was *presented* was a separate loss, found later and now gated | | URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated | | Deploy shape | done. Proven on two local mirrors and on the VPS, by hand and by pipeline | -| CI workflows | green. Validation runs on every pull request and feeds the required check | +| CI workflows | green. Validation runs on every pull request and feeds the required check. Three gates live in `checks/`: the URL contract, the environment-docs pair, and the line-ending pins | +| Branches | converged 2026-08-09. [#68][pr-68] promoted `develop` to `main` as a merge commit, `main` carries every commit on `develop`, and the content diff is empty. Nothing published: no run fired on `main` and `1.0.11` is still the newest release, which is the release model behaving as specified | | GitHub repo | public, both rulesets active, `configure.sh check` exits 0 | | Release pipeline | proven end to end. `1.0.17-g4b2def3ee9` is the newest, a prerelease from `develop` | | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical | @@ -41,10 +42,12 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **Restore file mtimes in CI so `--link-dest` links. The media check it waited on now exists, so this is unblocked.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. +- **~~Restore file mtimes in CI so `--link-dest` links.~~ Done, [#65][pr-65], and on `main` since the 2026-08-09 promotion.** `deploy-site-task.yml` installs `git-restore-mtime` and runs `git restore-mtime static` before the upload. Measured with two independent clones each way: 0 of 1791 files linked without the restore, 1052 with, and that 1052 corroborates from three directions, being what Hugo reports as static files, what links on a locally built release, and the file count in `static/`. **The next production deploy is the first to exercise it**, so it is also the first where a badly moded file can ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. The reasoning is kept below because it is what the next change to the deploy's linking behavior will need. + - The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **Publish a release from `main`, once the pipeline has soaked.** `1.0.11` is the newest release from `main` and was cut on 2026-08-01, ahead of every deploy change, so the next one is the first that would describe a site actually serving its public address. The mechanism is proven and is not what this waits on: it waits on the switchover being trusted rather than merely green, which is what the log review under **Recurring operations** establishes and no gate can. A release cut before that names a state that has not held yet. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. +- **[`checks/README.md`](./checks/README.md) names one of the three gates in its own directory.** It is written as the URL contract's document, titled `URL Parity Gate`, and `check-env-docs.py` and `check-eol-pins.py` appear in it nowhere. Each carries its reasoning in its own docstring, which is where the detail belongs, but [#66][pr-66] made a directory's README the authority on that directory and this one currently describes a subset of it without saying so. The small fix is a section naming what else lives there and what each gate answers; the alternative is to state in the README that it is scoped to the contract alone. Either resolves it, and the present state is the one that reads as complete while not being. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site. ## Recurring operations @@ -65,7 +68,9 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y ## Owed to the hub -Nothing. The spec update this repo owed the hub has landed: [ProjectTemplate#560][hub-type-pr] authored the `hugo` type, the `self-hosted` target, the `deploy-ssh` mechanism, guarantees D4.6 and D5.6, and a reference leaf pair, all measured from what this repo actually runs rather than from the prediction the intake carried. [#456][hub-issue] and [#558][hub-spec-issue] are closed with it. +**One thing, and it is conditional.** [ProjectTemplate#633][issue-633] reports that the hub's `repo_gate.py --check eol` compares `.gitattributes` and `.editorconfig` to each other and never to the tree, and offers to port this repo's [`checks/check-eol-pins.py`](./checks/check-eol-pins.py) into the hub's `scripts/` with tests in the `test_repo_gate.py` style. That offer is owed only if the hub wants that shape. **Do not start it before the hub rules**, because the open design question changes the code: the dead-pattern direction cannot be lifted unmodified, since a carried baseline forward-declares pins for files its consumers will have and the template itself does not, which is correct there and a finding anywhere else. + +The spec update this repo owed the hub has landed: [ProjectTemplate#560][hub-type-pr] authored the `hugo` type, the `self-hosted` target, the `deploy-ssh` mechanism, guarantees D4.6 and D5.6, and a reference leaf pair, all measured from what this repo actually runs rather than from the prediction the intake carried. [#456][hub-issue] and [#558][hub-spec-issue] are closed with it. **It is on the hub's `develop` and not on `main`, so it is not ground truth yet.** The registry entry that reclassifies this repo to `types: ["hugo", "source-only"]` with both publish targets sits on the same unpromoted branch. Until the hub promotes, this repo stays `source-only` for audit purposes, and the anticipatory evaluation of the nine `hugo` checks is in [the audit report](./reports/Blog/audit.md). @@ -109,6 +114,7 @@ Three findings are open at the hub, recorded here rather than only in the issues | [#550][issue-550] | Nothing detects a repo missing from the registry, which is how this repo stayed invisible. Three other repos are still absent. | | [#552][issue-552] | The audit flags any carried `AGENTS.md` naming the template repo, and the byte-locked `Fleet Bootstrap` section names it. Carrying the canonical correctly cannot pass, and it is the one finding the current run cannot clear. | | [#597][issue-597] | Filed from here, after a mandatory pre-merge gate in `OPERATIONS.md` was skipped on [#40][pr-40]. The ruling is that a verification a runner cannot perform needs a declared destination, not a better per-repo pointer. Answered by [#598][pr-598], which this repo owes work against once it is ground truth. | +| [#633][issue-633] | Filed from here, 2026-08-09. The hub's `eol` gate is document-to-document, so it reported `0 issue(s)` against the exact tree carrying both defects [#69][pr-69] fixed. Where a repo sets `[*] end_of_line = lf`, as this one does, that gate is vacuously satisfied by any pin at all, including one naming a file that does not exist. Open on the hub's side; the conditional work it may ask of this repo is under **Owed to the hub**. | **What [#598][pr-598] will ask of this repo, once it is ground truth.** It is merged to the hub's `develop` and not to `main`, so it binds nothing yet, per the trap below about reading `main` as ground truth. It makes `Local Verification` a sixth declared `OPERATIONS.md` heading, leading the file as the only pre-merge one. This repo's `OPERATIONS.md` carries **13 level-two headings and matches none of the five declared today**, so the work is a rename and reorder rather than new prose: `Local Verification Before a Pull Request` becomes `Local Verification`, and `Backup and Restore` is one word from the declared `Backup and Recovery`. Both are near-misses rather than absences, which is the shape a heading check will mostly find in a repo that wrote its operational document before the spec declared headings. Do not start until the hub promotes it. @@ -147,6 +153,9 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **The Copilot reviewer's login differs by API, and a wrong-form filter reads as a clean review.** REST reports `copilot-pull-request-reviewer[bot]`, GraphQL omits the suffix. A filter written in the other form matches nothing, and an empty result is indistinguishable from no findings. Assert the filter matched before trusting what it returned. - **A Copilot review hides findings in the review body, where the thread API cannot see them.** The `reviewThreads` query returns line threads only, so a review carrying `Suppressed comments (N)` in a `
` block reports zero unresolved while real findings sit unread. Read the review body itself, not just the threads, before calling a review loop finished. - **`gh pr merge --delete-branch` on a `develop -> main` promotion deletes `develop`.** Use a plain `gh pr merge --merge`. +- **A review that exists on the head is not a finished review.** On [#68][pr-68] `pr_review.py wait` returned after one second with `review_on_head=yes unresolved=0 suppressed=0`, an independent thread poll agreed at zero, and the pull request was reported green. Copilot posted a third finding *after* both reads, and the maintainer found it. The re-request had matched a review still being written, so every query was correct and every query was early. Wait and re-sweep all three surfaces, line threads, issue-level comments and the review bodies, before calling a loop finished. A very short `waited=` is the tell. +- **A rule naming a target that does not exist reads as coverage.** `.gitattributes` pinned `deploy/blog-deploy-shell` and `deploy/authorized_keys`, neither ever tracked in this repo, and the comment above them claimed the extensionless-shebang case was handled. It was not, and the one real instance sat unpinned twenty lines up, invisible precisely because the file looked thorough. [`checks/check-eol-pins.py`](./checks/check-eol-pins.py) gates this file now, in both directions. The generalisation is not gated anywhere: a config that names a path, a job, or a host that is absent will read as protection for as long as nobody checks. +- **A gate is only as good as the matcher underneath it, and a wrong matcher fails quietly.** `check-eol-pins.py` shipped with two defects in one function, `git ls-files` pathspec not sharing gitattributes glob semantics and then a leading `/` being stripped before anchoring was decided. Both were found by review rather than by the gate, and each would have made it report a confident wrong answer. [#70][pr-70] fixed them and it now checks its own matching against `git check-attr` on every run, which is the general move: where a check predicts what a tool will do, compare it against the tool rather than against the documentation. ## Reference @@ -182,6 +191,11 @@ The deploy root is deliberately absent from this table. The rsync destination is [migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md [pr-30]: https://github.com/ptr727/Blog/pull/30 [pr-40]: https://github.com/ptr727/Blog/pull/40 +[pr-65]: https://github.com/ptr727/Blog/pull/65 +[pr-66]: https://github.com/ptr727/Blog/pull/66 +[pr-68]: https://github.com/ptr727/Blog/pull/68 +[pr-69]: https://github.com/ptr727/Blog/pull/69 +[pr-70]: https://github.com/ptr727/Blog/pull/70 @@ -196,5 +210,6 @@ The deploy root is deliberately absent from this table. The rsync destination is [issue-554]: https://github.com/ptr727/ProjectTemplate/issues/554 [issue-563]: https://github.com/ptr727/ProjectTemplate/issues/563 [issue-597]: https://github.com/ptr727/ProjectTemplate/issues/597 +[issue-633]: https://github.com/ptr727/ProjectTemplate/issues/633 [pr-553]: https://github.com/ptr727/ProjectTemplate/pull/553 [pr-598]: https://github.com/ptr727/ProjectTemplate/pull/598 From 555e97a67aea481d8c355df20fc41259f6a71166 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 20:33:41 -0700 Subject: [PATCH 2/6] Ask for the deploy's missing ref gate, and refresh three rows that moved (#72) * Ask for the deploy's missing ref gate, and refresh three rows that moved The release workflow gates its branch in both directions: a dispatch from anything but main or develop is refused, and validate-release fails loudly if the default branch carries a prerelease suffix or a non-default branch carries none. The deploy has half of that. assert-ref refuses production from any ref but main, and staging accepts any ref at all, so a staging deploy can be dispatched from main, which is what happened here today. Recorded with the part that makes it more than a one-line change: the comment above that job argues for the present behavior, so the code and the reason move together or the file ends up contradicting itself. The full-ref comparison stays, for the reason already written there. Three state rows had moved and are re-measured rather than adjusted: the release row, now 1.0.54 from main and 1.0.54-gcd9285a9fe from develop, each pinned to its own branch tip the production row, now release 20260809-030521, read back from the live X-Blog-Release header and the note that this is the first production release built with the mtimes restored, so it is the first that can arrive as hard links and the first where a bad mode could ride one forward. The shared-inode count is the host's to report, so it is asked for rather than inferred. Requested by the maintainer. Co-Authored-By: Claude Opus 5 (1M context) * Stop the State table contradicting itself, and retire the release hold Review caught three contradictions, all of them mine and all the same mistake: new state was prepended to old prose instead of replacing it, so each cell asserted both readings at once. The Branches row said "Nothing published" and named 1.0.11 as newest, while the row directly above it named 1.0.54. Both were written today. What is actually true is narrower and is what the row now says: the merge published nothing, because a human merge never auto-publishes, and the releases were cut afterwards by explicit dispatch. The VPS production row said "The description below is the M7a record" with that record inlined immediately after it in the same cell, so the sentence pointed at nothing a reader could locate. The row now leads with what is served today and names M7a as history at the end. The Next item still said production serves 20260808-154717 in the present tense. It now names both and points at the State table as the current value, so the number lives in one place. Two more of the same class that review did not reach: The "Publish a release from main, once the pipeline has soaked" item is done and is retired. The soak reasoning is kept deliberately, because it was overridden rather than satisfied and the same argument returns at the next release: 1.0.54 names a working pipeline and a rehearsal host, not a completed cutover. The robots.txt item's reference to 20260808-154717 is left alone. It reads "verified from the served bytes on release ...", which is a past measurement and is still true. Co-Authored-By: Claude Opus 5 (1M context) * Correct a gate this repo does not have, and a directional reference The item claimed publish-release.yml carries a validate-release job that fails when a branch's version classification disagrees with it. It does not. There is no validate-release anywhere under .github/ in this repo: that is a build-repo construct in the fleet contract, and I described the contract as though it were the implementation. What is actually there is one guard, an Assert dispatch ref step refusing any ref but main or develop. Everything else is derived from the ref with nothing to disagree with it. NBGV reads publicReleaseRefSpec from version.json to give main a clean X.Y.Z and every other branch a -g suffix, and the release's prerelease flag is computed as ref_name is not main. main cannot cut a prerelease because no input exists that could ask it to. That correction makes the item's argument better rather than weaker, so the argument is rewritten rather than patched. The deploy is not lacking symmetry with the release. It differs in kind: environment is an independent dispatch input, so the ref and the target are two values that can disagree, and only one of the two disagreements is caught today. A workflow taking its target as a separate input has to check it; one deriving its target from the ref has nothing to check. Also: the Branches row pointed at "the row above" for the release versions, which sit two rows below it. Both found by review, as suppressed comments carrying no thread. Co-Authored-By: Claude Opus 5 (1M context) * Reconcile the two URL totals, and stop calling the validate job absent Two numbers in the State table disagreed without saying why. The URL contract row implies 328 + 917 = 1,245, and the production row claimed a 1,253-URL contract. Both are right and they count different things: build gate 328 render, 917 redirect, 778 legacy image URLs, against files on disk live check 328 + 917 + 8 curated media = 1,253, against a running server Verified by counting the lists rather than by arithmetic on the prose: golden-urls.txt is 328, redirect-urls.txt is 917, golden-media-live.txt is 8. The URL contract row now states that two gates read different subsets and why the live one carries 8 images rather than 778, and the production row names its total as that subset rather than as a second contract. The 1,245 under the HUGO_BASEURL item is deliberately left alone. It is a counterfactual about 2026-08-07, when the live check was 1,245, so updating it to today's number would make a true sentence false. Separately, "exactly one guard" was wrong: publish-release.yml also runs the reusable validate job the pull request runs, so a dispatch cannot release a ref that fails validation. It is now "exactly one ref guard", with the validate job named and the distinction stated, since it gates the sources rather than the branch-to-classification relationship the item is about. Both found by review, as suppressed comments carrying no thread. Co-Authored-By: Claude Opus 5 (1M context) * Name the deploy that exercised the mtime restore, since it has happened The item said "the next production deploy is the first to exercise it". That was true when written and stopped being true an hour later in the same session, when 20260809-030521 deployed. It now names that release and dates it. The sentence also carried the only forward-looking part worth keeping, so that is stated as the open half rather than dropped: the shared-inode count is server-side, the host had been measuring 0 across every pipeline release, and roughly 1052 of 1791 is what the change predicts. It is asked in the channel rather than inferred from this side, because nothing here can read an inode on that host. Swept for the rest of the class and found none: no other future-tense claim in the file describes something already done. Found by review, as a suppressed comment carrying no thread. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- TODO.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/TODO.md b/TODO.md index 7d86384..e556aaf 100644 --- a/TODO.md +++ b/TODO.md @@ -10,16 +10,16 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | --- | --- | | Content and media | done. Carries every migrated post and page, with the media hash-verified against the export tar | | Media integrity | closed for the files themselves. The conversion's only loss was 19 images, inside five galleries, restored from the capture, and the 98 files with no page links are adjudicated rather than unknown, with `ORPHANED_MEDIA` failing if that count moves either way. How media was *presented* was a separate loss, found later and now gated | -| URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated | +| URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated. **Two gates read different subsets of that, which is why two totals appear in this file**: the build gate proves all three against files on disk, and the live check requests 328 + 917 + 8 curated media = **1,253** against a running server, since proving 778 images over the wire on every deploy buys nothing the build gate has not already proven | | Deploy shape | done. Proven on two local mirrors and on the VPS, by hand and by pipeline | | CI workflows | green. Validation runs on every pull request and feeds the required check. Three gates live in `checks/`: the URL contract, the environment-docs pair, and the line-ending pins | -| Branches | converged 2026-08-09. [#68][pr-68] promoted `develop` to `main` as a merge commit, `main` carries every commit on `develop`, and the content diff is empty. Nothing published: no run fired on `main` and `1.0.11` is still the newest release, which is the release model behaving as specified | +| Branches | converged 2026-08-09. [#68][pr-68] promoted `develop` to `main` as a merge commit, `main` carries every commit on `develop`, and the content diff is empty. **The merge itself published nothing**, which is the release model behaving as specified: a human merge never auto-publishes, and the releases named two rows down were cut afterwards by explicit dispatch | | GitHub repo | public, both rulesets active, `configure.sh check` exits 0 | -| Release pipeline | proven end to end. `1.0.17-g4b2def3ee9` is the newest, a prerelease from `develop` | +| Release pipeline | proven end to end and cut from both branches 2026-08-09. `1.0.54` is the newest release, from `main` and marked Latest, and `1.0.54-gcd9285a9fe` the newest prerelease, from `develop`. Each tag is pinned to its own branch tip, and the clean-versus-suffixed classification is the gate working rather than a coincidence | | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical | | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | -| VPS production | **M7a done 2026-08-08.** Serving release `20260808-154717` at `blog.insanegenius.net`, answering `200` unauthenticated, deployed from `main` by pipeline with the 1,245-URL contract verified against the live site. `/robots.txt` answers 200 carrying a `.net` sitemap line, and the gallery fix is live. DNS for the public name is still on the old platform | +| VPS production | serving release `20260809-030521` at `blog.insanegenius.net` as of 2026-08-09, deployed from `main` after the promotion and read back from the live `X-Blog-Release` header rather than from the run's exit code. `200` unauthenticated, `X-Robots-Tag: noindex, nofollow` still set for the rehearsal, and the live check's 1,253 URLs verified against the running site, which is the 328 + 917 + 8 subset described in the URL contract row rather than a different contract. That release is **the first built with the mtimes restored**, so the first whose static tree can arrive as hard links, and therefore the first where a bad file mode could ride one forward into every later release. The shared-inode count is server-side, so ask the host for it rather than inferring it here. **M7a, the first production deploy, was 2026-08-08 on release `20260808-154717`**, which established the unauthenticated `200`, the `.net` sitemap line in `/robots.txt`, and the gallery fix. DNS for the public name is still on the old platform | | Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has run once by hand, the outward pass only: it read 2026-08-08 traffic and found nothing to add to the URL contract, the inward pass has not run, and neither pass is on the cadence **Recurring operations** sets | ## Blocked on the maintainer @@ -30,7 +30,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y ## Next, in dependency order - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. -- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` serves release `20260808-154717`, answering `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, read from the `X-Blog-Release` header rather than from a pipeline's exit code. The host side verified the first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": +- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` answers `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, with the serving release read from the `X-Blog-Release` header rather than from a pipeline's exit code. **It serves `20260809-030521` today; M7a itself was `20260808-154717`**, and the State table above is the current value, so read it there rather than from this paragraph. The host side verified that first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. @@ -42,10 +42,15 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **~~Restore file mtimes in CI so `--link-dest` links.~~ Done, [#65][pr-65], and on `main` since the 2026-08-09 promotion.** `deploy-site-task.yml` installs `git-restore-mtime` and runs `git restore-mtime static` before the upload. Measured with two independent clones each way: 0 of 1791 files linked without the restore, 1052 with, and that 1052 corroborates from three directions, being what Hugo reports as static files, what links on a locally built release, and the file count in `static/`. **The next production deploy is the first to exercise it**, so it is also the first where a badly moded file can ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. The reasoning is kept below because it is what the next change to the deploy's linking behavior will need. +- **~~Restore file mtimes in CI so `--link-dest` links.~~ Done, [#65][pr-65], and on `main` since the 2026-08-09 promotion.** `deploy-site-task.yml` installs `git-restore-mtime` and runs `git restore-mtime static` before the upload. Measured with two independent clones each way: 0 of 1791 files linked without the restore, 1052 with, and that 1052 corroborates from three directions, being what Hugo reports as static files, what links on a locally built release, and the file count in `static/`. **Release `20260809-030521` is the first production deploy to exercise it**, done 2026-08-09, so it is also the first where a badly moded file can ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. **The shared-inode count is the open half**: it is server-side, the host had been measuring 0 across every pipeline release, and roughly 1052 of 1791 is what the change predicts. Asked in the channel rather than inferred from this side. The reasoning is kept below because it is what the next change to the deploy's linking behavior will need. - The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. -- **Publish a release from `main`, once the pipeline has soaked.** `1.0.11` is the newest release from `main` and was cut on 2026-08-01, ahead of every deploy change, so the next one is the first that would describe a site actually serving its public address. The mechanism is proven and is not what this waits on: it waits on the switchover being trusted rather than merely green, which is what the log review under **Recurring operations** establishes and no gate can. A release cut before that names a state that has not held yet. +- **~~Publish a release from `main`, once the pipeline has soaked.~~ Done 2026-08-09, with the maintainer lifting the soak condition explicitly.** `1.0.54` is cut from `main` and marked Latest, and `1.0.54-gcd9285a9fe` from `develop`, each tag pinned to its own branch tip. Before this, `1.0.11` had been the newest from `main` since 2026-08-01, predating every deploy change. **The soak reasoning is kept because it was not satisfied, it was overridden**, and the same argument returns at the next release: what a release from `main` cannot yet claim is a switchover that has been *trusted* rather than merely green, which the log review under **Recurring operations** establishes and no gate can. `1.0.54` therefore names a working pipeline and a rehearsal host, not a completed cutover. +- **Gate the deploy ref by environment in both directions, so a deploy cannot do what a release structurally cannot.** [`deploy-site.yml`](./.github/workflows/deploy-site.yml)'s `assert-ref` job refuses production from any ref but `main`, and **staging accepts any ref at all**. Add the missing arm so staging accepts only `develop`: `develop` deploys staging, `main` deploys production, and neither can reach the other's environment. + - **The release workflow is safe by derivation rather than by a gate, which is the distinction to get right before copying anything from it.** [`publish-release.yml`](./.github/workflows/publish-release.yml) has exactly one **ref** guard, an `Assert dispatch ref step` refusing any ref but `main` or `develop`. It also runs the reusable `validate` job that the pull request runs, so a dispatch cannot release a ref that fails validation, but that gates the *sources* rather than the branch-to-classification relationship this item is about. Everything else follows from the ref with nothing to disagree with it: NBGV reads `publicReleaseRefSpec` in [`version.json`](./version.json) to give `main` a clean `X.Y.Z` and every other branch a `-g` suffix, and the release's `prerelease:` is computed as `github.ref_name != 'main'`. **`main` cannot cut a prerelease because no input exists that could ask it to.** This repo has no `validate-release` job — that is a build-repo construct in the fleet contract, not something shipped here, and an earlier draft of this item claimed it did. + - **The deploy differs in kind, which is why it needs the gate the release does not.** `environment` is an independent `workflow_dispatch` input, so the ref and the environment are two values that can disagree, and today only one of the two disagreements is caught. That is the whole argument: not that the deploy should imitate the release, but that a workflow taking the target as a separate input has to check it, where one deriving the target from the ref has nothing to check. + - **This reverses a deliberate decision, so the comment above the job has to change with the code.** It currently reads "Staging deploys from any ref, since proving a branch before it merges is what staging is for", which is the argument for the present behavior and would be left contradicting the gate. Whoever makes the change states the new reason instead: staging is the rehearsal for what `develop` will promote, so a deploy from a feature branch proves something that is not what ships. Requested by the maintainer 2026-08-09, after a staging deploy was dispatched from `main` here. + - Keep the existing comparison style, against the full `github.ref` rather than `ref_name`, for the reason already recorded there: a tag and a branch can share a short name, so a tag called `develop` would satisfy a `ref_name` test while pointing at an arbitrary commit. The new arm inherits that hazard exactly and is not a special case. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. - **[`checks/README.md`](./checks/README.md) names one of the three gates in its own directory.** It is written as the URL contract's document, titled `URL Parity Gate`, and `check-env-docs.py` and `check-eol-pins.py` appear in it nowhere. Each carries its reasoning in its own docstring, which is where the detail belongs, but [#66][pr-66] made a directory's README the authority on that directory and this one currently describes a subset of it without saying so. The small fix is a section naming what else lives there and what each gate answers; the alternative is to state in the README that it is scoped to the contract alone. Either resolves it, and the present state is the one that reads as complete while not being. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site. From ed695a546bdfb88d540437fa85bf199243894c2b Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 05:54:53 -0700 Subject: [PATCH 3/6] Use the maintained git-restore-mtime, and assert it actually restored (#75) The Ubuntu package is git-tools v2022.12, and that release shells out to `git whatchanged`. Current git refuses to run that without `--i-still-use-this`, a flag there is no way to pass through the tool, so it failed twelve times inside one step, restored nothing, and exited 0. Every release since #65 has been a full copy while CI reported success, which the VPS agent measured from the other end as 0 of 3,275 shared inodes. Upstream fixed it: MestreLion/git-tools 91dc541 replaced whatchanged with `git log`, released as v2025.08. chetan/git-restore-mtime-action v2.3 vendors that exact version, verified by reading the vendored script at the pinned SHA -- `__version__ = "2025.08"` and no whatchanged anywhere in it. Taking the action rather than the tarball also drops the apt round trip the step's own comment called out as a thing that can fail on its own, and it pins by SHA like every other action here. The assertion is the half that matters, because the failure mode was a step that exited 0 having done nothing. A restored file cannot be newer than the commit it was dated from, so no file under static/ may be newer than HEAD's own commit time. A checkout necessarily happens after the commit it checks out, so an unrestored tree always breaches that bound and a restored one never does. It calibrates from the repository, so nothing goes stale as content moves. Counting distinct mtime days was written first and is wrong. git restores the LAST COMMIT time and static/ arrived in a bulk import, so a correctly restored tree here has exactly ONE distinct day and that check would have failed the good case. It was caught only because it was run against a real restored clone instead of being trusted, which is the same discipline this whole change exists to enforce. Measured on two clones of this repository, the assertion run verbatim: restored ok margin 651427s unrestored ERROR margin -33196s And the property the deploy actually needs: two independent clones, restored, produce byte-identical path+mtime sets, so --link-dest can match. 1,052 files updated by the new script, the same 1,052 as before. Closes #74. Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/deploy-site-task.yml | 40 ++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/.github/workflows/deploy-site-task.yml b/.github/workflows/deploy-site-task.yml index 5964693..f0ca4a9 100644 --- a/.github/workflows/deploy-site-task.yml +++ b/.github/workflows/deploy-site-task.yml @@ -77,12 +77,12 @@ jobs: # One update for the job, because each one is a network round trip that can fail on its # own. REQUIRE_BROTLI later makes a missing brotli fatal, so this keeps the build from - # failing, and git-restore-mtime is what the next step runs. + # failing. - name: Install build tools step run: | set -Eeuo pipefail sudo apt-get update - sudo apt-get install --yes --no-install-recommends brotli git-restore-mtime + sudo apt-get install --yes --no-install-recommends brotli # Git stores no mtimes, so a checkout stamps every file with the moment it was written. # The deploy uploads with --link-dest against the previous release, and a file only links @@ -100,12 +100,42 @@ jobs: # began with, so a media file that acquires a bad one stays present, correctly named and # unreadable, through every later release. The live check is what notices that, by # requesting images and failing on the 403. - # `git restore-mtime`, the subcommand form, because the package installs into git's - # exec-path at /usr/lib/git-core rather than onto PATH, so the bare name does not resolve. + # The action rather than the Ubuntu package, because the package is git-tools v2022.12 and + # that release shells out to `git whatchanged`, which current git refuses to run without + # `--i-still-use-this` - a flag there is no way to pass through. It failed twelve times in + # one step, restored nothing, and exited 0, so every release since #65 was a full copy + # while CI reported success. Upstream replaced whatchanged with `git log` in v2025.08, and + # this action vendors exactly that version. - name: Restore file mtimes step + uses: chetan/git-restore-mtime-action@d186aca54f8760da4dec55313195e51ed3ebb0b3 # v2.3 + with: + args: static + + # The step above is the second tool to claim it restored these and not have done it, so + # the outcome is asserted rather than the tool trusted. This is the check whose absence + # let #65 ship broken: `git restore-mtime` printed "1,052 files to be processed" and then + # processed none of them, which reads exactly like success. + # + # The discriminator is that a restored file cannot be newer than the commit it was dated + # from, so no file under static/ may be newer than HEAD's own commit time. A checkout + # necessarily happens after the commit it checks out, so an unrestored tree is always + # newer than that bound and a restored one never is. It calibrates itself from the + # repository, so no measured constant goes stale as content moves. + # + # Counting distinct mtime days looks like the obvious check and is wrong: git restores + # the LAST COMMIT time, and static/ arrived in a bulk import, so a correctly restored + # tree here has exactly one distinct day. That version was written first and rejected + # only because it was run against a real restored clone before being trusted. + - name: Assert mtimes were restored step run: | set -Eeuo pipefail - git restore-mtime static + bound=$(git log -1 --format=%ct) + newest=$(find static -type f -printf '%T@\n' | sort -n | tail -1 | cut -d. -f1) + echo "static/ newest mtime $newest, HEAD committed $bound, margin $((bound - newest))s" + if [ "$newest" -gt "$bound" ]; then + echo "::error::mtime restore did nothing: static/ carries files newer than HEAD's commit, so they still hold their checkout time and --link-dest will link nothing." + exit 1 + fi # The pin lives in the action, so the deploy and validation cannot install different generators. - name: Install Hugo step From f78fa38c0c03c629159d8e4a1cda69635d1f13a8 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 05:55:26 -0700 Subject: [PATCH 4/6] Announce synthetic traffic with X-Blog-Check on every live-check request (#73) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Announce synthetic traffic with X-Blog-Check on every live-check request 97.7% of the traffic reaching production is ours, and the only thing separating it from real visitors was a coincidence: the CI runner's curl and the VPS host's curl are byte-identical at 8.5.0, so user agent cannot tell our deploy gate from their smoke probe, and the CI half of the client address rotates every run. One runner image bump merges the two and nothing would report it. The host side captures the field already and their ci/smoke.sh sends vps/smoke. This is the other half. The value carries provenance rather than a boolean, /, so "which run produced this 404" is one query rather than a correlation across timestamps. Derived rather than configured: github/- under Actions, proxmox/manual elsewhere, and CHECK_TAG overrides both to name a purpose for a hand run. The run attempt is in the id deliberately, which is a refinement on the design as proposed. A re-run of a failed workflow keeps its GITHUB_RUN_ID and takes a new GITHUB_RUN_ATTEMPT, so the id alone merges a retry into the run it was retrying, and that is exactly the case someone reads the log to understand. It is two curl config files rather than one, and that is the part not to collapse later. The tag is unconditional; the Pangolin token is sent only to the origin it belongs to, because a redirect that one day points off-site must not mail the credential there. Folding them together would either give the tag that restriction for no reason or take it away from the token. Both are now assembled from the tag first, with the token appended where it is allowed, so every request is attributable including the off-site hop that deliberately carries no credential. Verified rather than assumed: the header is on the wire under curl -v, all three derivations produce the expected value, the full run still reports PASS - 1253 URLs honored, and two tagged requests were sent to production for the host side to confirm capture. CHECK_TAG is added to the env-docs gate's KNOBS and described in ENVIRONMENT.md, and the gate was watched failing on it before the row was written. Design agreed with the VPS agent in their section 31. Co-Authored-By: Claude Opus 5 (1M context) * Validate what goes into the curl config, since a config is options not headers A curl config file is a list of directives rather than a list of headers, so a value interpolated into a quoted `header = "..."` line can leave it. A newline ends the line and starts a new directive; a double quote closes the string with the same effect. Either turns an override into "add a curl option nobody typed". CHECK_TAG takes a strict allowlist, because this repo defines its grammar: the `/` the design already states, so letters, digits, dot, underscore, hyphen and the separating slash. Anything else fails at entry. The Pangolin token values take a narrower rule, refusing only a quote or a newline, because the grammar of a credential belongs to its issuer and not to this script. Neither character is legal in an HTTP header value, so a token carrying one is a paste accident rather than a token. The failure names the variable and never echoes the value. Each guard was demonstrated failing rather than assumed: CHECK_TAG=$'proxmox/x\noutput = /tmp/pwned' exit 2 CHECK_TAG='proxmox/x" header = "X-Evil: 1' exit 2 CHECK_TAG='proxmox/media dev' exit 2 PANGOLIN_ACCESS_TOKEN_ID=$'a\nb' exit 2 And that exercise found one of my own: I had added a guard rejecting an empty CHECK_TAG, and it is unreachable. An empty value is already treated as unset and takes the derived default, which is the behaviour we want, so the guard could never fire and its message told the reader the opposite of what happens. Removed rather than left as a rule describing a case that cannot occur. Full run still reports PASS - 1253 URLs honored. shellcheck and shfmt clean. Found by Copilot review on #73. Co-Authored-By: Claude Opus 5 (1M context) * Enforce the tag shape the comment already claimed, rather than describing it The validation checked a character set and the comment above it said the grammar was /. Those are different rules, and the weaker one was the one running: `smoke`, `/smoke`, `proxmox/` and `a/b/c` all passed while reading as conforming. That matters because the shape is the whole point of provenance over a boolean. Grouping the log by source is only reliable if every tag has a source half, and a tag with no slash or three slashes breaks the query quietly rather than loudly. Now enforced: exactly one slash, both halves non-empty, from the same narrow character set. Each rule was demonstrated rejecting and the two legal shapes demonstrated passing: smoke must be / /smoke needs a non-empty half either side proxmox/ needs a non-empty half either side a/b/c takes exactly one / proxmox/media dev character set proxmox/media-dev accepted github/999-3 accepted Both derived defaults were re-checked against their own rule rather than assumed to satisfy it, which is the failure mode of adding a validator after the values it governs. ENVIRONMENT.md now states the shape is enforced instead of expected. Full run still PASS - 1253 URLs honored. Found by Copilot review on #73, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) * Rewrite the four failure messages so they read as prose A stray space before a comma, in `one / , as /` and in `/ , such as`, which reads as a typo in the one place a reader is already confused. It came from trying to keep a bare `/` from running into the punctuation after it. Fixed by naming the character instead of printing it: "takes exactly one slash" and "either side of the slash" have nothing to collide with. The example message loses the space and keeps the comma. The character-set message had the same collision in a different form, `. _ - / -- got`, where the slash ran into the separator. The set is quoted now rather than bare. All four read back as a user sees them: may contain only letters, digits, and the characters '. _ - /' -- got 'proxmox/media dev' takes exactly one slash, as / -- got 'a/b/c' needs a non-empty half either side of the slash -- got '/smoke' must be /, such as proxmox/media-dev -- got 'smoke' Found by Copilot review on #73, which also pointed out the second instance rather than only the first. Co-Authored-By: Claude Opus 5 (1M context) * Say which characters apply to which part of the tag The ENVIRONMENT.md row required "exactly one /" and then said the value may contain "only letters, digits, . _ -", a set with no slash in it. Read strictly the two clauses contradict; read charitably the reader has to guess that the character set governs the halves rather than the whole. The row now says the slash is the separator and the only one allowed, and that each half is drawn from the character set. That is what the script enforces: the charset check permits a slash anywhere, and the two rules after it leave exactly one, in the separator position, so each half can only contain the rest. Found by Copilot review on #73, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) * Set globasciiranges rather than inheriting it, since the allowlist rests on it `A-Za-z0-9` in a shell pattern is a collation range, not an ASCII range, so what the allowlist actually permits depends on a shell option this script never set. It passes today because `globasciiranges` is on by default in bash 5.2, which makes the guarantee an accident of the build. Demonstrated rather than argued, under en_US.UTF-8: shopt -u globasciiranges aé -> ACCEPT aÉ -> ACCEPT shopt -s globasciiranges aé -> REJECT aÉ -> REJECT So a value containing an accented letter would have been written into the curl config on a shell where the option is off, while the comment above it called the set narrow. The consequence was never a config injection: neither a quote nor a newline can arrive this way, and those are the two characters that break out of a quoted config line. What was wrong is the claim. A validator that describes itself as strict has to be strict on its own terms rather than on the terms of whatever shell runs it. Set explicitly, and re-verified against the hostile case: with the option forced off beforehand, in a UTF-8 locale, `proxmox/café` is rejected. Full run still PASS - 1253 URLs honored. shellcheck and shfmt clean. Found by Copilot review on #73, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) * Reject CR in a token, and fail when globasciiranges cannot be set Two findings, and the second is the more embarrassing. The token guard refused a quote and a newline and allowed a carriage return. Header injection is classically CRLF and a lone CR is enough on its own, so refusing LF while allowing CR leaves exactly the shape the guard exists for. Measured before fixing: a token containing \r was accepted. Both variables now reject all three, and the LF and quote cases were re-run to confirm nothing regressed. And the `shopt -s globasciiranges` added in the previous commit was itself unchecked. This script runs under `set -uo pipefail` and not `-e`, so on a shell without that option the command prints to stderr, returns 1, and execution steps straight over it — leaving the allowlist locale-dependent underneath a comment promising it is not. Which is the same defect the previous commit set out to fix, reintroduced one line below it: a guarantee resting on something nobody checked. `shopt` returns 1 on an unknown option name, verified, so the failure is now fatal and says why. Full run still PASS - 1253 URLs honored. shellcheck and shfmt clean. Both found by Copilot review on #73, as suppressed comments. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- ENVIRONMENT.md | 1 + checks/check-env-docs.py | 9 ++- checks/check-live-urls.sh | 117 +++++++++++++++++++++++++++++++++----- 3 files changed, 113 insertions(+), 14 deletions(-) diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md index c9bfdd9..700d172 100644 --- a/ENVIRONMENT.md +++ b/ENVIRONMENT.md @@ -89,6 +89,7 @@ Set on the command line for one run rather than stored anywhere. | `NO_LINK_DEST=1` | full copy instead of hard-linking from the previous release | | `KEEP_RELEASES` | how many releases `make-release.sh` leaves behind | | `EXPECT_RELEASE` | the release id `check-live-urls.sh` requires the live site to report, which is what makes a rollback verifiable rather than merely exiting zero | +| `CHECK_TAG` | the `X-Blog-Check` provenance this run announces on every request. **`/` is enforced, not merely expected**: exactly one `/`, which is the separator and the only one allowed, with both halves non-empty and each drawn from letters, digits, `.`, `_`, `-`. Rarely set by hand, since `check-live-urls.sh` derives `github/-` under Actions and `proxmox/manual` elsewhere. Set it to name a purpose for a hand run, as `proxmox/media-dev` | ## Two credentials to the VPS, and why they are separate diff --git a/checks/check-env-docs.py b/checks/check-env-docs.py index e2ed0f5..e22891d 100755 --- a/checks/check-env-docs.py +++ b/checks/check-env-docs.py @@ -40,7 +40,14 @@ # Set per invocation rather than stored, so they appear in no template and would otherwise # be invisible to this check. Listed here because the doc has a table for them, and a knob # nobody documented is the same failure as an undocumented file value. -KNOBS = {"ENV_FILE", "REQUIRE_BROTLI", "NO_LINK_DEST", "KEEP_RELEASES", "EXPECT_RELEASE"} +KNOBS = { + "ENV_FILE", + "REQUIRE_BROTLI", + "NO_LINK_DEST", + "KEEP_RELEASES", + "EXPECT_RELEASE", + "CHECK_TAG", +} # Names that look like configuration to the patterns above but are not. # ENVIRONMENT and RELEASE_ID are computed inside the workflow and passed down, and diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index 0bafda1..e37f12d 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -40,12 +40,99 @@ done FAILED="$(mktemp)" CURLERR="$(mktemp)" CURLRC="" -trap 'rm -f "$FAILED" "$CURLERR" ${CURLRC:+"$CURLRC"}' EXIT +CHECKRC="$(mktemp)" +trap 'rm -f "$FAILED" "$CURLERR" "$CHECKRC" ${CURLRC:+"$CURLRC"}' EXIT + +# Every request this script makes announces itself as synthetic, so the server's log can be +# filtered down to real visitors with one clause. Agreed with the host side, whose Traefik +# captures the field and whose own `ci/smoke.sh` already sends `vps/smoke`. +# +# The value carries provenance rather than a boolean, `/`, because "which run +# produced this 404" is then a one-line query against the log. +# +# The run attempt is part of the id deliberately. A re-run of a failed workflow keeps the +# same GITHUB_RUN_ID and gets a new GITHUB_RUN_ATTEMPT, so the id alone would merge a +# retried run into the run it was retrying, which is exactly the case someone reads the log +# to understand. +# +# It is forgeable and it gates nothing. Absence of the header is not proof of a human +# either: a scanner sends no header and neither does a forged request. It must never reach +# auth, rate limiting, robots handling, or caching. +if [ -z "${CHECK_TAG:-}" ]; then + if [ -n "${GITHUB_RUN_ID:-}" ]; then + CHECK_TAG="github/${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT:-1}" + else + CHECK_TAG="proxmox/manual" + fi +fi +# Validated before it is written, because this lands in a curl config file and a curl config +# file is a list of options rather than a list of headers. A value carrying a newline ends the +# header line and starts a new directive, so an override could add an option nobody typed; a +# value carrying a double quote ends the quoted string with the same result. Neither is a +# legal HTTP header value either, so refusing both loses nothing. +# +# The shape is enforced and not merely described, because the whole value of provenance over a +# boolean is that the log can be grouped by source, and `select(.tag | startswith("github/"))` +# is only reliable if every tag actually has a source half. A charset check alone would accept +# `smoke`, `/smoke` and `a/b/c`, each of which reads as conforming and breaks that query. +# Exactly one slash, both halves non-empty, from a deliberately narrow character set. +# +# The range `A-Za-z0-9` is collation-dependent, so the allowlist below is only ASCII-strict +# because `globasciiranges` happens to be on. Set explicitly rather than inherited, since a +# guarantee resting on a build default is not a guarantee. Demonstrated rather than assumed: +# with the option off, under en_US.UTF-8, `aé` and `aÉ` are both ACCEPTED by this pattern, +# and with it on they are rejected. +# Checked, because this script runs under `set -uo pipefail` and not `-e`, so an unsupported +# option would print to stderr, return 1, and be stepped straight over — leaving the +# validation locale-dependent underneath a comment promising it is not. `shopt` returns 1 on +# an unknown option name, which is what makes this testable rather than decorative. +shopt -s globasciiranges || { + echo "FAIL this shell does not support globasciiranges, so the character allowlist below would be locale-dependent" >&2 + exit 2 +} +case "$CHECK_TAG" in +*[!A-Za-z0-9._/-]*) + echo "FAIL CHECK_TAG may contain only letters, digits, and the characters '. _ - /' -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +*/*/*) + echo "FAIL CHECK_TAG takes exactly one slash, as / -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +/* | */) + echo "FAIL CHECK_TAG needs a non-empty half either side of the slash -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +*/*) ;; +*) + echo "FAIL CHECK_TAG must be /, such as proxmox/media-dev -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +esac +printf 'header = "X-Blog-Check: %s"\n' "$CHECK_TAG" >"$CHECKRC" +echo "==> tagging requests X-Blog-Check: $CHECK_TAG" # A resource access token opens the proxy's auth gate. # It goes into a curl config file because bash cannot export an array to the parallel checks. # A command line is also world-readable in ps output, and this runs 1,245 of them. if [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] && [ -n "${PANGOLIN_ACCESS_TOKEN:-}" ]; then + # Same hazard as CHECK_TAG above and the same reason, but a narrower rule, because the + # grammar of a credential is the issuer's to define and not this script's. Only the + # characters that break out of a quoted config line are refused, and none is legal in an + # HTTP header value, so a token containing one is a paste accident rather than a token. + # Reported without echoing the value, since it is a secret and the finding is its shape. + # + # Carriage return counts as a line ending here as much as newline does. Header injection + # is classically CRLF, and a lone CR is enough on its own, so refusing LF while allowing + # CR would leave the shape this guard exists for. + for name in PANGOLIN_ACCESS_TOKEN_ID PANGOLIN_ACCESS_TOKEN; do + case "${!name}" in + *'"'* | *$'\n'* | *$'\r'*) + echo "FAIL $name contains a quote, a newline, or a carriage return, none of which can appear in an HTTP header value" >&2 + exit 2 + ;; + esac + done CURLRC="$(mktemp)" chmod 600 "$CURLRC" printf 'header = "P-Access-Token-Id: %s"\nheader = "P-Access-Token: %s"\n' \ @@ -58,14 +145,18 @@ elif [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] || [ -n "${PANGOLIN_ACCESS_TOKEN:-}" fi # Assembled once here rather than per request, since it is the same for every call. -AUTH=() -[ -n "$CURLRC" ] && AUTH=(-K "$CURLRC") +# The check tag is unconditional and the token is not, which is why they are two files +# rather than one. Every request should be attributable; only a same-origin request may +# carry the credential, and folding them together would make the tag inherit that +# restriction for no reason, or the token lose it, depending on which way it was folded. +AUTH=(-K "$CHECKRC") +[ -n "$CURLRC" ] && AUTH+=(-K "$CURLRC") # Invoked indirectly, through `export -f` and the `xargs bash -c` calls below. # shellcheck disable=SC2329 check_render() { - local url="$1" code auth=() - [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + local url="$1" code auth=(-K "$CHECKRC") + [ -n "$CURLRC" ] && auth+=(-K "$CURLRC") code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$BASE$url") [ "$code" = "200" ] || echo "render $url expected 200, got $code" >>"$FAILED" } @@ -81,8 +172,8 @@ check_render() { # arrived truncated to nothing, which still answers 200. Content type is asserted because a # server misconfigured into serving an error page for a missing asset answers 200 as well. check_media() { - local url="$1" code len type target auth=() target_auth=() - [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + local url="$1" code len type target auth=(-K "$CHECKRC") target_auth=() + [ -n "$CURLRC" ] && auth+=(-K "$CURLRC") target="$BASE$url" target_auth=("${auth[@]}") # One hop is followed rather than passed to curl -L, because -L would carry the @@ -102,10 +193,10 @@ check_media() { # Same origin boundary as check_redirect, and for the same reason: a rule that one # day points off-site must not mail the token there. A bare prefix would also accept # a lookalike host registered as an attacker's subdomain. - target_auth=() + target_auth=(-K "$CHECKRC") if [ -n "$CURLRC" ]; then case "$target" in - "$BASE" | "$BASE"/*) target_auth=(-K "$CURLRC") ;; + "$BASE" | "$BASE"/*) target_auth+=(-K "$CURLRC") ;; esac fi ;; @@ -144,8 +235,8 @@ check_media() { # Invoked indirectly, the same way as check_render above. # shellcheck disable=SC2329 check_redirect() { - local url="$1" code dest dcode auth=() dest_auth=() - [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + local url="$1" code dest dcode auth=(-K "$CHECKRC") dest_auth=(-K "$CHECKRC") + [ -n "$CURLRC" ] && auth+=(-K "$CURLRC") code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$BASE$url") case "$code" in 301 | 308) ;; @@ -162,7 +253,7 @@ check_redirect() { # starts with this one, such as a lookalike registered as an attacker's subdomain. if [ -n "$CURLRC" ]; then case "$dest" in - "$BASE" | "$BASE"/*) dest_auth=(-K "$CURLRC") ;; + "$BASE" | "$BASE"/*) dest_auth+=(-K "$CURLRC") ;; esac fi dcode=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${dest_auth[@]}" "$dest") @@ -174,7 +265,7 @@ check_redirect() { } export -f check_render check_redirect check_media -export BASE FAILED CURLRC +export BASE FAILED CURLRC CHECKRC echo "==> $BASE" From b30d62c4d612bc61eff9521d724034aa55285814 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 06:35:40 -0700 Subject: [PATCH 5/6] Restore mtimes in the local release too, so both paths fail the same way (#76) * Restore mtimes in the local release too, so both paths fail the same way The deploy workflow restores mtimes and asserts the restore happened. make-release.sh did neither, and the gap was invisible on this host because a long-lived working tree already carries old mtimes, so local releases linked 1052 files while CI linked zero. A fresh clone here would have reproduced the CI defect exactly and said nothing. git-restore-mtime is required rather than optional. Absent, the release refuses to build and names the version to install, because skipping when a tool is missing is how the CI version shipped broken for four releases: it printed a reassuring line and restored nothing. The assertion is the same self-calibrating one the workflow uses, with one difference that CI does not need. A working tree can legitimately hold a static file newer than any commit, so locally modified and untracked paths are excluded rather than the check being skipped whenever the tree is dirty. A clean tree takes the same single find the workflow runs. Demonstrated failing before being trusted, all three states: not installed exits 1, names the version and the reason installed but a no-op exits 1 on the assertion, having printed the same "1,052 files to be processed" line the broken v2022.12 prints working v2025.08 1,052 files updated, assertion passes The second is the real bug reproduced with a stub, rather than a hypothetical. One thing this surfaced that is not a defect and needs saying. The first restored release CANNOT link, because it is compared against a predecessor built with unrestored mtimes, so the existing zero-shared-files guard fires and refuses it. That guard is correct and the changeover needs one NO_LINK_DEST=1 release to seed a restored generation, which is what that knob already exists for. Measured on the staging mirror: first restored release, against an unrestored predecessor 0 of 3269 seeded with NO_LINK_DEST=1 full copy the next ordinary release 1052 of 3269 1052 is the same number the two-clone measurement in #65 predicted and the same count Hugo reports as static files. The mirror still answers PASS - 1253 URLs honored afterwards. Co-Authored-By: Claude Opus 5 (1M context) * Accept either install shape, and refuse the broken version by name Requiring `git-restore-mtime` on PATH would have refused a correctly installed tool. The Debian and Ubuntu package puts it in git's exec-path at /usr/lib/git-core, where only the subcommand form resolves, which is what the deploy workflow's own comment says and what this script ignored. A manual install to /usr/local/bin gives the opposite: the bare name works and the subcommand does not. Both are now accepted, and the one that resolves is the one used. More usefully, the version is gated rather than left to the assertion. v2022.12 fails in the one way an outcome check catches late and a reader never catches at all: it calls `git whatchanged`, current git refuses to run that, so it prints files to be processed, processes none, and exits 0. Refusing it here names the cause, where the assertion can only report the symptom. 2025.08 is the floor because that is the release which replaced whatchanged with `git log`. Versions are YYYY.MM, so dropping the dot compares them as integers. Four states, each demonstrated rather than assumed: absent names both invocation forms and where to get it v2022.12 refused, with the whatchanged defect named bare name restores, 1052 of 3269 linked git subcommand restores, 1052 of 3269 linked The last was tested through GIT_EXEC_PATH against a directory carrying the real exec-path plus the script, so the bare name genuinely did not resolve and only the subcommand branch could have run. Mirror still answers PASS - 1253 URLs honored. Found by Copilot review on #76. Co-Authored-By: Claude Opus 5 (1M context) * Make the version diagnostic reachable, and parse a rename correctly Two defects in the code added one commit ago, both of which made a guard describe a case it could not reach. With `set -e` and `pipefail`, an unmatched grep inside a command substitution aborts the script at the assignment, so the "did not report a version" branch below it was unreachable and an unparseable version would have surfaced as a bare exit 1 with no message. Reproduced in isolation first: the line after the assignment never printed. Tolerating the failed match on that assignment makes the diagnostic run, verified with a stub reporting "version unknown". And `git status --porcelain -z` emits TWO NUL records for a rename or a copy, `XY ` then a bare ``. The loop read the second as another status record and stripped three characters off a bare path, recording `tic/a.txt` for `static/a.txt`. The real path then stayed out of the exclusion set, so the assertion could fail on a file that is legitimately uncommitted. Both halves are now excluded, since both are uncommitted. Measured against a real rename in static/ rather than a constructed one: R static/apple-touch-icon-renamed.png static/apple-touch-icon.png excluded: static/apple-touch-icon-renamed.png excluded: static/apple-touch-icon.png ==> 2 uncommitted path(s) under static/, excluded from the mtime check 1051 of 3269 files hard-linked One fewer than 1052, which is the renamed file correctly not matching. The rename was reverted afterwards and the mirror rebuilt from the clean tree, answering PASS - 1253 URLs honored. Both found by Copilot review on #76, as suppressed comments. Co-Authored-By: Claude Opus 5 (1M context) * Pick the first usable git-restore-mtime, not the first one that exists Both invocation forms were accepted and the bare name always won, so a stale manual install at /usr/local/bin vetoed a current packaged one behind it and the release refused to build with a perfectly good tool present. Accepting both forms and then letting the worse one decide is not really accepting both. Each candidate is now version-checked and the first ACCEPTABLE one wins. Only when none meets the floor does it refuse, and it names what it found rather than only what it wanted. Measured, with a 2022.12 stub on PATH and a real 2025.08 in git's exec-path: ==> restoring file mtimes with git restore-mtime 2025.08 ==> 1052 of 3269 files hard-linked and with only the stale one reachable: no usable git-restore-mtime: found git-restore-mtime 2022.12, git restore-mtime 2022.12, and 2025.08 or newer is required Both forms report the same tool there, correctly: git resolves a subcommand from PATH as well as from its exec-path, so one stale binary is genuinely both candidates. Found by Copilot review on #76, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- deploy/make-release.sh | 119 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) diff --git a/deploy/make-release.sh b/deploy/make-release.sh index 89c5eec..170471c 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -19,6 +19,10 @@ usage() { REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +# The release upstream replaced `git whatchanged` with `git log`. Anything older restores +# nothing and exits 0, so this is the floor rather than a preference. +MTIME_MIN=2025.08 + # The deploy root and the base URL are the only host-specific values, and they pair per environment. # ENV_FILE selects the environment, because `set -a` overwrites a value the caller exported. # The first argument overrides the root, being read after this. @@ -82,6 +86,121 @@ command -v hugo >/dev/null || { cd "$REPO" +# Git stores no mtimes, so a checkout stamps every file with the moment it was written, and +# a release built from a fresh clone then links nothing against the previous one. This host's +# long-lived working tree has old mtimes already and links fine, which is exactly what makes +# the gap easy to miss: it is invisible here and total in a clean checkout. +# +# The deploy workflow does the same thing with the same assertion after it, deliberately, so +# the local path and CI fail the same way for the same reason rather than one of them being +# the trusted one. +# +# Required rather than optional. Skipping when absent is how the CI version shipped broken +# for four releases: it printed a reassuring line and restored nothing. +# Both invocation forms are accepted, because how it installs decides which one resolves. The +# Debian and Ubuntu package puts it in git's exec-path at /usr/lib/git-core, where only the +# subcommand form works; a manual install to /usr/local/bin gives the bare name and no +# subcommand. Testing only one would refuse a correctly installed tool. +# +# Each candidate is version-checked and the first ACCEPTABLE one wins, rather than the first +# one that merely exists. A host can carry both, and an old manual install must not veto a +# current packaged one sitting behind it. +# +# The version is gated rather than left to the assertion below, because before MTIME_MIN the +# tool calls `git whatchanged`, which current git refuses to run, so it reports files to be +# processed, processes none, and exits 0. Refusing it here names the cause; the assertion can +# only report the symptom. Versions are YYYY.MM, so dropping the dot compares them as integers. +mtime_probe() { + # The failed match is tolerated because `set -e` with `pipefail` would otherwise abort the + # whole script at the assignment, making every diagnostic below unreachable. + "$@" --version 2>/dev/null | grep -oE '[0-9]{4}\.[0-9]{2}' | head -1 || true +} + +MTIME_CMD=() +mtime_version="" +mtime_found="" +for mtime_form in bare subcommand; do + mtime_try=() + case "$mtime_form" in + bare) command -v git-restore-mtime >/dev/null 2>&1 && mtime_try=(git-restore-mtime) ;; + subcommand) git restore-mtime --version >/dev/null 2>&1 && mtime_try=(git restore-mtime) ;; + esac + [ ${#mtime_try[@]} -gt 0 ] || continue + + mtime_try_version="$(mtime_probe "${mtime_try[@]}")" + if [ -z "$mtime_try_version" ]; then + mtime_found="${mtime_found}${mtime_found:+, }${mtime_try[*]} (no version reported)" + continue + fi + mtime_found="${mtime_found}${mtime_found:+, }${mtime_try[*]} $mtime_try_version" + if [ "${mtime_try_version//./}" -ge "${MTIME_MIN//./}" ]; then + MTIME_CMD=("${mtime_try[@]}") + mtime_version="$mtime_try_version" + break + fi +done + +if [ ${#MTIME_CMD[@]} -eq 0 ]; then + if [ -z "$mtime_found" ]; then + echo "git-restore-mtime not found, as either 'git-restore-mtime' or 'git restore-mtime'" >&2 + else + echo "no usable git-restore-mtime: found $mtime_found, and $MTIME_MIN or newer is required" >&2 + echo " before $MTIME_MIN it calls 'git whatchanged', which current git refuses to run, so it" >&2 + echo " restores nothing and still exits 0 -- every release would silently be a full copy" >&2 + fi + echo " it is what makes --link-dest able to link, and a release built without it is a full copy" >&2 + echo " install git-tools $MTIME_MIN or newer, from https://github.com/MestreLion/git-tools" >&2 + exit 1 +fi + +echo "==> restoring file mtimes with ${MTIME_CMD[*]} $mtime_version" +"${MTIME_CMD[@]}" static + +# Asserted rather than trusted, because the failure this exists for is a restore that reports +# success and does nothing. A restored file cannot be newer than the commit it was dated from, +# so nothing under static/ may be newer than HEAD's commit time. +# +# Locally modified files are excluded, which is the one way this differs from CI. A working +# tree can legitimately hold a static file newer than any commit; a fresh CI checkout cannot, +# so there the same check needs no exclusion. Comparing the clean files only keeps the +# assertion meaningful during an edit loop instead of being skipped whenever the tree is dirty. +mtime_bound="$(git log -1 --format=%ct)" + +# `git status --porcelain` covers modified, staged and untracked in one list, so an empty +# result means every file under static/ is tracked and unchanged. That is the CI case, and it +# takes the same one-pass `find` the workflow uses. +# A rename or a copy emits TWO NUL records, `XY ` then a bare ``, so the loop has to +# consume the second explicitly. Reading it as another status record would strip three +# characters off a bare path and record `tic/a.txt` for `static/a.txt`, leaving the real path +# unexcluded and the assertion able to fail on a file that is legitimately uncommitted. +# Both halves of a rename are excluded, since both are uncommitted. +declare -A mtime_dirty=() +while IFS= read -r -d '' entry; do + mtime_dirty["${entry:3}"]=1 + case "${entry:0:1}" in + R | C) IFS= read -r -d '' mtime_orig && mtime_dirty["$mtime_orig"]=1 ;; + esac +done < <(git status --porcelain -z -- static) + +if [ ${#mtime_dirty[@]} -eq 0 ]; then + mtime_newest=$(find static -type f -printf '%T@\n' | sort -n | tail -1 | cut -d. -f1) +else + echo "==> ${#mtime_dirty[@]} uncommitted path(s) under static/, excluded from the mtime check" + mtime_newest=0 + while IFS= read -r -d '' f; do + [ -n "${mtime_dirty[$f]:-}" ] && continue + t=$(stat -c %Y "$f") + [ "$t" -gt "$mtime_newest" ] && mtime_newest=$t + done < <(git ls-files -z -- static) +fi + +if [ "$mtime_newest" -gt "$mtime_bound" ]; then + echo "mtime restore did nothing: static/ holds unmodified files newer than HEAD's commit," >&2 + echo " so they still carry their checkout time and --link-dest will link nothing" >&2 + exit 1 +fi +echo "==> mtimes restored, newest $mtime_newest against HEAD $mtime_bound" + # Hugo maps HUGO_ onto config, so HUGO_BASEURL overrides hugo.yaml with no flag. # A mirror built without it serves canonical tags, feed links, and permalinks pointing at production. # The build gate passes either way, so the effective value is logged rather than left implicit. From 018228523937ea83f47636d6edc61a385633e816 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 08:03:53 -0700 Subject: [PATCH 6/6] Stop crediting #65 with a fix that never ran (#78) * Stop crediting #65 with a fix that never ran The item described the mechanism #75 replaced: installing the Ubuntu package and calling `git restore-mtime static`. CI uses a pinned action now, and make-release.sh requires v2025.08 or newer by either invocation form. The worse half was a claim of mine, added earlier the same day, that release 20260809-030521 was the first production deploy to exercise the restore. It exercised nothing. That release ran v2022.12, which refuses to run under current git and exits 0 regardless, and the host measured the result as 0 of 3,275 shared inodes. The entry asserted a working feature on the strength of a merge rather than a measurement, which is the exact distinction this repo keeps having to relearn. The item now credits #75 and #76, says plainly that #65 shipped broken, and carries what was actually measured: 1052 of 3269 linked on the local production mirror with the tool installed by hand, the release stamped 2026-08-01 rather than at build time, and 584 MB then 18 MB for two releases. It also records the part that will otherwise read as a regression: the first restored release cannot link, because its predecessor was built with unrestored mtimes, so the first deploy after the promotion is expected to link zero and the one after it is the real test. The retained #65 diagnosis keeps its wording and gains a lead-in saying so. It contains a "today" that means 2026-08-08 and names a fix that did not run, both of which read as current state without it. Found by Copilot review on #77. Co-Authored-By: Claude Opus 5 (1M context) * Use the file's US spelling The lead-in added one commit ago wrote "behaviour" two words away from the "behavior" it was introducing, in a file that uses the US form throughout. Nothing gates spelling here: cspell runs over README.md and HISTORY.md only, so this is convention rather than a rule, which is why it needed a reader to catch it. Found by Copilot review on #78. Co-Authored-By: Claude Opus 5 (1M context) * Describe the assertion's actual scope, not a stronger one The entry said both paths assert that nothing under static/ is newer than HEAD's commit. CI does. make-release.sh excludes modified and untracked paths, deliberately, because a working tree can legitimately hold a static file newer than any commit and skipping the check whenever the tree is dirty would make it useless during an edit loop. So the claim was stronger than the code, which is the same defect this file keeps producing: prose that describes the rule someone meant rather than the one that runs. It now states the shared invariant as tracked and unmodified files, and names why the two paths differ: a fresh checkout has nothing uncommitted in it, so CI needs no exclusion to reach the same guarantee. Found by Copilot review on #78, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) * Say static/ where static/ is what is walked "CI compares the whole tree" reads as the repository. Both assertions run `find static`, and nothing outside that directory is examined by either, which matters because the sentence is the thing a future edit to the guard would be read against. Verified against both implementations rather than from memory: .github/workflows/deploy-site-task.yml:133 find static -type f deploy/make-release.sh:186 find static -type f The difference between the two paths is the exclusion, not the scope, and the sentence now says so. Found by Copilot review on #78, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) * Match the file's number format, and say permissions rather than mode Counts in the new text alternated between 1052 and 1,052 next to 0 of 3,275 in the same block, which makes two measurements of the same quantity look like different quantities at a glance. The comma form is what this file mostly uses, ten instances to six, and 1,052 already appeared with one, so the new text follows it. The preserved 2026-08-08 diagnosis is deliberately left alone. Its lead-in says it is kept as written, and reformatting inside it would make that false for the sake of consistency it explicitly opts out of. And "a badly moded file" reads as fashion rather than permissions. It is now "a file that acquires the wrong permissions", which is what the sentence is about: a hard link carries its inode's mode, so a wrong one rides the chain into every later release. Found by Copilot review on #78, as suppressed comments. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- TODO.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index e556aaf..055fe2f 100644 --- a/TODO.md +++ b/TODO.md @@ -42,8 +42,13 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **~~Restore file mtimes in CI so `--link-dest` links.~~ Done, [#65][pr-65], and on `main` since the 2026-08-09 promotion.** `deploy-site-task.yml` installs `git-restore-mtime` and runs `git restore-mtime static` before the upload. Measured with two independent clones each way: 0 of 1791 files linked without the restore, 1052 with, and that 1052 corroborates from three directions, being what Hugo reports as static files, what links on a locally built release, and the file count in `static/`. **Release `20260809-030521` is the first production deploy to exercise it**, done 2026-08-09, so it is also the first where a badly moded file can ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. **The shared-inode count is the open half**: it is server-side, the host had been measuring 0 across every pipeline release, and roughly 1052 of 1791 is what the change predicts. Asked in the channel rather than inferred from this side. The reasoning is kept below because it is what the next change to the deploy's linking behavior will need. - - The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. +- **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome, since a checkout always happens after the commit it checks out: **no tracked, unmodified file under `static/` may be newer than HEAD's own commit.** CI compares every file under `static/`, because a fresh checkout has nothing uncommitted in it; the local path walks the same directory but excludes modified and untracked paths, which a working tree can legitimately hold newer than any commit, so the check stays usable during an edit loop rather than being skipped whenever the tree is dirty. + - **#65 never restored anything and exited 0 for four releases.** It installed the Debian and Ubuntu package, git-tools v2022.12, which shells out to `git whatchanged`; current git refuses that without `--i-still-use-this`, a flag there is no way to pass through. It printed `1,052 files to be processed`, processed none, and succeeded, twelve refusals inside one green step. **Release `20260809-030521` did not exercise it**, an earlier version of this entry said it did, and the host measured the truth as 0 of 3,275 shared inodes. Local git still permits `whatchanged`, which is why #65's two-clone measurement was honest and measured everything except the machine that runs it. + - **Verified end to end on the local production mirror, 2026-08-09**, with the tool installed by hand: `1,052 of 3,269 files hard-linked`, and the release carrying `site/media` and `site/external` stamped `2026-08-01`, the commit date rather than the build date, which is what proves the restore reached the release and not merely the working tree. Two releases cost 584 MB then 18 MB. + - **The first restored release cannot link, by construction**, because it is compared against a predecessor built with unrestored mtimes. `make-release.sh`'s zero-shared-files guard refuses that one, so the changeover takes a single `NO_LINK_DEST=1` release to seed a restored generation. The deploy has no such guard and will simply link nothing for one deploy. **So the first deploy after the promotion is expected to link 0**, and the one after it is the real test. + - **The shared-inode count on the VPS is still the open half** and is the host's to measure, since nothing here can read an inode there. Told to them in the channel as §S, which corrects §R for predicting the jump one deploy too early. Roughly 1,052 of 3,275 is what the change predicts, at the *second* restored deploy. + - Making `--link-dest` effective is also what makes a file that acquires the wrong permissions able to ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. + - **The original diagnosis, kept as written on 2026-08-08 because it is what the next change to the deploy's linking behavior will need.** Read it as the record of why the change was made, not as current state: the "today" in it is that date, and the fix it names is #65's, which turned out not to run. The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **~~Publish a release from `main`, once the pipeline has soaked.~~ Done 2026-08-09, with the maintainer lifting the soak condition explicitly.** `1.0.54` is cut from `main` and marked Latest, and `1.0.54-gcd9285a9fe` from `develop`, each tag pinned to its own branch tip. Before this, `1.0.11` had been the newest from `main` since 2026-08-01, predating every deploy change. **The soak reasoning is kept because it was not satisfied, it was overridden**, and the same argument returns at the next release: what a release from `main` cannot yet claim is a switchover that has been *trusted* rather than merely green, which the log review under **Recurring operations** establishes and no gate can. `1.0.54` therefore names a working pipeline and a rehearsal host, not a completed cutover. - **Gate the deploy ref by environment in both directions, so a deploy cannot do what a release structurally cannot.** [`deploy-site.yml`](./.github/workflows/deploy-site.yml)'s `assert-ref` job refuses production from any ref but `main`, and **staging accepts any ref at all**. Add the missing arm so staging accepts only `develop`: `develop` deploys staging, `main` deploys production, and neither can reach the other's environment. @@ -201,6 +206,8 @@ The deploy root is deliberately absent from this table. The rsync destination is [pr-68]: https://github.com/ptr727/Blog/pull/68 [pr-69]: https://github.com/ptr727/Blog/pull/69 [pr-70]: https://github.com/ptr727/Blog/pull/70 +[pr-75]: https://github.com/ptr727/Blog/pull/75 +[pr-76]: https://github.com/ptr727/Blog/pull/76