From 6689e562d2bc0868b7738ca8ad15c6b250370149 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 08:15:24 -0700 Subject: [PATCH 1/2] Record the log pull as installed, and the rename it does not carry (#58) * Record the log pull as installed, and the rename it does not carry The off-host access-log pull is no longer a design. It is installed as a daily timer, and its first copy exists. Both records here said the opposite, because both were written before it was installed, and an operational document understating what runs is as misleading as one overstating it. The distinction kept is that the timer has never fired: that first copy was started by hand, so one copy is a fact and "backed up daily" is still a unit file until the first scheduled run. The finding the copy produced is the more useful half. The pull passes no --delete for the logs, deliberately, so a file the VPS renames after it has been pulled keeps its old name here forever alongside the new one. That has already happened, to two archives whose names were a day ahead of their contents, and a count over the archive by filename now returns 110 lines where 58 exist. Verified a strict subset with comm rather than assumed. Read a date from a line's StartUTC rather than from the filename holding it. Also recorded: why restoring mtimes in CI comes after a live media check rather than before, since making --link-dest effective means a third of the tree starts arriving as hard links carrying their inode's mode instead of having it re-asserted per deploy; and the log evidence that reframes the robots.txt decision, where no crawler has ever fetched the sitemap and the only thing advertising it is the file the cutover deletes. Co-Authored-By: Claude Opus 5 (1M context) * Name the VPS as the renaming side, and say scheduled rather than cadence Both review findings are about a reader landing on the wrong side of a two-host sentence. The rename paragraph sat under one that mentions the backup host, so "the host renames" could be read as the machine doing the pulling rather than the one being pulled from, and the State row's "has run on its cadence" reads as "has run at all" one clause before the same row says it has copied once. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- OPERATIONS.md | 4 +++- TODO.md | 10 +++++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index 225a56e..9e87c85 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -237,7 +237,9 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access **Retention on the local mirrors is a different question and is unsolved.** Those containers use Docker's `json-file` driver with the built-in defaults, so a mirror's log grows without bound and is discarded when its container is recreated. That matters less than it did on the VPS, because the mirrors serve no readers, and it belongs to the host rather than to this repository, the same split "Retention" and "Who Owns What" describe for release pruning. -**The off-host copy of the access log is a design rather than a fact.** The VPS holds the only copy until the pull to the backup host runs, which makes the host's retention window the hard deadline rather than a comfortable margin. A review reads whatever survived and reports confidently on it either way, so settle this before the first review is expected to mean anything. +**The off-host copy of the access log exists, and the schedule that maintains it is younger than the copy.** The pull to the backup host is installed as a `systemd` timer running daily at 09:00 UTC, chosen to sit behind both producers on the VPS rather than beside them, and its first copy was made by hand rather than by the timer. Read the unit and its last run on the backup host rather than trusting this paragraph, for the same reason retention is read from the VPS: a claim about a schedule is only worth what the machine says. + +**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it**, and treat a rename on the VPS as something the channel has to carry, because no transfer will. ## Who Owns What diff --git a/TODO.md b/TODO.md index 75cfb40..a8cbb5b 100644 --- a/TODO.md +++ b/TODO.md @@ -19,7 +19,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | | VPS production | **M7a done 2026-08-08.** Serving release `20260808-041050` at `blog.insanegenius.net`, answering `200` unauthenticated, verified 9/9 from the host side with the built `baseURL` read from the deployed bytes. DNS for the public name is still on the old platform | -| Operations | not started. Neither the periodic log review nor the off-host log pull has run on its cadence | +| Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has not run at all | ## Blocked on the maintainer @@ -36,7 +36,10 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Decide what `robots.txt` says, now before the cutover rather than before the first deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - **The deploy did not fix the 404, which is the part that reads the wrong way round, and is now measured rather than predicted.** The VPS agent raised this in §22.10 and asked two questions of this side. Both are now measured rather than assumed: **this site emits no `robots.txt` at all**, because `hugo.yaml` sets no `enableRobotsTXT`, so the 404 survives the deploy and `X-Robots-Tag` stays the only control. **`sitemap.xml` is emitted** and becomes fetchable on the interim name at that same deploy, listing whatever `HUGO_BASEURL` was built with, which is the second reason that variable mattered. Read together: after the rehearsal deploy a crawler gets a full sitemap and no robots file. - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the `.com` sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **Deliberately undecided**, and the three options are to preserve today's behavior, to write what this site actually wants, or to keep emitting nothing and accept the loss. -- **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. + - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. + - **`/robots.txt/`, with a trailing slash, is an entry in `slugs.map` pointing at `/`**, noted under "Open decisions" below. If a real file lands, that entry points at it instead, and the change ships in the same release rather than after it. +- **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. +- **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **Publish a release from `main`, once the pipeline has soaked.** `1.0.11` is the newest release from `main` and was cut on 2026-08-01, ahead of every deploy change, so the next one is the first that would describe a site actually serving its public address. The mechanism is proven and is not what this waits on: it waits on the switchover being trusted rather than merely green, which is what the log review under **Recurring operations** establishes and no gate can. A release cut before that names a state that has not held yet. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. @@ -47,7 +50,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y **Everything above ends; this section does not.** The list above finishes when the site serves its public address, and these two start there and never stop. Both are written down as procedures and neither has run on its cadence yet, which is the shape a recurring duty has right before it silently becomes a one-off. An end-to-end exercise of this system is a deploy **plus** the operations that follow it, so a run that proves staging and production and stops has proven the half that has an obvious finish line. - **Review the logs for non-200s**, daily for the first week after cutover, then monthly. Real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor, and a staging probe for `/wp-login.php` answered by the auth gate rather than by the site is the shape of what Caddy never sees. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". -- **Pull the log off the VPS, on a schedule**, which is [#53][issue-53]. The access log is deliberately outside the nightly encrypted archives, because those are fourteen full copies with no dedupe and an append-only file would be multiplied by fourteen for no recovery benefit. So the VPS's 400-day window is the **only** copy until this runs. The two rsync lines are now written, parameterized, and tested against the live host, and they are not installed, because installing needs root. **Until they are, say plainly that "backed up" describes a design rather than a fact in anything built on top of that log.** +- **Pull the log off the VPS, on a schedule**, which is [#53][issue-53]. The access log is deliberately outside the nightly encrypted archives, because those are fourteen full copies with no dedupe and an append-only file would be multiplied by fourteen for no recovery benefit, so the VPS's 400-day window was the **only** copy until this ran. **It is installed**, as `vps-backup-pull.timer` at 09:00 UTC daily with `Persistent=true`, and a first copy exists: 42 archives and 4 log files, pulled 2026-08-08 12:59 UTC. **That run was started by hand, so the timer has never fired**, which is the distinction worth keeping until 2026-08-09 09:00 UTC proves the schedule rather than the script. One copy is a fact; "backed up daily" is still a unit file. + - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returns 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently, and the only propagation mechanism is a note in the channel. Removing that one file is the maintainer's call, since it is a deletion inside a backup tree. ## Owed to the hub From 884c50f8a07d8514133f45856bc62d81286d8921 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 08:23:22 -0700 Subject: [PATCH 2/2] Emit robots.txt, and gate the base URL it is derived from (#60) * Emit robots.txt, and gate the base URL it is derived from The old platform serves a robots.txt today and this site emitted none, so the cutover was not a return to a previous state, it was a move from having crawl directives to having none on a site that has had them for years. Hugo emits the file only when enableRobotsTXT is set, which is why /robots.txt answered 404 through the first production deploy. The Sitemap: line is the load-bearing part rather than any rule. Across the interim hostname's first full day no crawler fetched sitemap.xml or feed.xml once, every request to either coming from curl: a crawler is told where a sitemap is rather than guessing, and the only thing telling them is the file the cutover deletes. Every Disallow the old platform serves names a WordPress path this site does not have, so the content that preserves today's behavior is User-agent: *, no Disallow, and the sitemap. The theme's template already derives that URL from the built baseURL, so setting the flag is the whole change and there is no second edit to remember at the cutover. Verified across two base URLs: the mirror advertises its own sitemap and a production build advertises the site's. /robots.txt/ with a trailing slash is in the redirect contract and sent visitors to the home page. It now resolves to the real file, fixed in build-redirects.py rather than in the generated map, since the map is rewritten from the capture and a hand edit does not survive. Verified byte-identical regeneration otherwise: the map diff is that one line. /osd.xml/ stays pointed at the home page, being an OpenSearch description this site does not emit. The check is worth more than the file. Every contract list is path-only and check-live-urls.sh joins whatever base it is handed, which is what lets one contract cover four environments and also means a build baked with the wrong host passes all 1,245 URLs while every canonical tag and sitemap entry names another site. robots.txt is the one artifact whose absolute URL a gate can read without being told what to expect, so the parity check compares it against the origin read from the home page's canonical link. Four failures gated, all four demonstrated failing before the check was trusted: file absent, no Sitemap line, a line naming another origin, and a line advertising a sitemap not built. Released to the local production mirror and checked live, since this touches deploy/maps and hugo.yaml: 1245 URLs honored, /robots.txt/ 301s to /robots.txt, and that file answers 200 as text/plain. Co-Authored-By: Claude Opus 5 (1M context) * Bound both claims to what was measured Two overclaims in comments, both real. The hugo.yaml note said no crawler has ever fetched this site's sitemap unprompted, where the evidence is one day of traffic on one hostname, and the docstring said the site answered 404 for years, where the behaviour is a config flag rather than a duration. Each now says what was observed and over what window. Co-Authored-By: Claude Opus 5 (1M context) * Withdraw the wrong-baseURL claim, which the check cannot support Four places said this check makes a wrong HUGO_BASEURL visible. It does not, and the reason is structural rather than a bug: the advertised sitemap origin and the canonical origin it is compared against are both derived from baseURL, so they agree whenever the build is coherent, including when baseURL held the wrong value for the environment being deployed to. Nothing inside the artifact can see that, which is exactly why the VPS side reads the origin out of the deployed sitemap.xml, og:url and feed.xml and reports the counts either way. Raised in review, in all four places at once, and correct. Each now states what the comparison proves, internal consistency, and what it still catches: an origin written rather than derived, where a committed static/robots.txt shadowing the template is how that happens, and a sitemap advertised but never built. A gate described as catching more than it catches is worse than no gate, because the next person stops looking for the failure it does not see. Co-Authored-By: Claude Opus 5 (1M context) * Stop the robots check crashing on bytes it exists to inspect, and narrow two claims Three findings, all correct. read_text with strict UTF-8 raises out of the entire parity run on an invalid byte, so the one gate written to report a bad robots.txt would stack-trace on one instead of failing normally. Confirmed both ways: strict raises UnicodeDecodeError at position 67 on a file carrying 0xff, and errors=replace reports through the ordinary path. The file most likely to carry it is a committed static/robots.txt, which is the case this check is for. The docstring still said two failures where there are four, and both the docstring and the README said the Sitemap: line is the only absolute URL this gate can read. It is not, since site_origin reads the canonical link and collect_refs reads absolute asset references. The true and narrower point is that it is the only place an origin is compared rather than joined. Co-Authored-By: Claude Opus 5 (1M context) * Print the robots summary after its last assertion, not before Two findings, both correct. The advertising line printed before the sitemap-existence check ran, so the one failing case read as a pass on the line immediately above its own failure. Now every branch prints exactly once, after deciding: missing-sitemap reports "1 advertised sitemap(s) not built" and only a clean run prints what it advertises. The TODO paragraph describing the cutover loss still said "since this site emits none" in the present tense, one bullet below the entry saying the file is now built. It reads as history now, which is what it is. Co-Authored-By: Claude Opus 5 (1M context) * Describe the missing robots.txt rather than asserting why it is missing The message named enableRobotsTXT as the cause. A partial build, a deleted file, and the wrong output directory reach the identical state, so asserting one sends a reader to verify a setting that is already correct. It now names the path it looked for, offers the likely cause as likely, and says the alternatives are indistinguishable from here, which is the same shape the orphan messages use for the same reason. Co-Authored-By: Claude Opus 5 (1M context) * Pluralize the failure noun by count, which fixes the gallery too "FAIL robots: 1 problems" was the report. The noun was a bare string and a count is always printed beside it, so every label with a plural noun had the same defect: the pre-existing gallery one prints "1 stray nodes" on a single stray node, which is the shape that made the new noun wrong in the same way. Now a (singular, plural) pair chosen by count. The default stays "missing" and needs no pair, being count-neutral already. Verified across both labels at 1 and at many, and the clean run is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- TODO.md | 11 +++--- checks/README.md | 12 +++++++ checks/build-redirects.py | 13 ++++++- checks/check-url-parity.py | 73 ++++++++++++++++++++++++++++++++++++-- deploy/maps/slugs.map | 2 +- hugo.yaml | 9 +++++ 6 files changed, 110 insertions(+), 10 deletions(-) diff --git a/TODO.md b/TODO.md index a8cbb5b..6795791 100644 --- a/TODO.md +++ b/TODO.md @@ -33,11 +33,12 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. -- **Decide what `robots.txt` says, now before the cutover rather than before the first deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - - **The deploy did not fix the 404, which is the part that reads the wrong way round, and is now measured rather than predicted.** The VPS agent raised this in §22.10 and asked two questions of this side. Both are now measured rather than assumed: **this site emits no `robots.txt` at all**, because `hugo.yaml` sets no `enableRobotsTXT`, so the 404 survives the deploy and `X-Robots-Tag` stays the only control. **`sitemap.xml` is emitted** and becomes fetchable on the interim name at that same deploy, listing whatever `HUGO_BASEURL` was built with, which is the second reason that variable mattered. Read together: after the rehearsal deploy a crawler gets a full sitemap and no robots file. - - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the `.com` sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **Deliberately undecided**, and the three options are to preserve today's behavior, to write what this site actually wants, or to keep emitting nothing and accept the loss. +- **`robots.txt` is decided and built, 2026-08-08, and what remains is that production has not been redeployed since.** The site emits one now, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. The record below is kept because the reasoning is what the next decision about crawl directives will need, and because production still answers 404 until a deploy carries this. + - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set, so this describes the release production is still serving rather than the current build. + - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Because this site emitted none, M7b would not have been a return to a previous state, it would have been a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer would have gone with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - - **`/robots.txt/`, with a trailing slash, is an entry in `slugs.map` pointing at `/`**, noted under "Open decisions" below. If a real file lands, that entry points at it instead, and the change ships in the same release rather than after it. + - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. + - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. - **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. @@ -72,7 +73,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. - **What `robots.txt` says, which is undecided and is the last non-mechanical item before M7b.** Recorded under "Next" above, where it blocks the cutover. -- `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`. +- `/osd.xml/` sits in `slugs.map` pointing at `/`, and stays there unless this site ever emits an OpenSearch description. `/robots.txt/` was the other half and is resolved, above. - Content is capped at a fixed 720px on every screen, because PaperMod's width is four CSS variables with no responsive term and no Hugo parameter. The prose measure is right and should stay; images and galleries inheriting the same cap is the part that costs something on a wide display. The knobs, the override location, and the `--gap` trap are documented under "Customization points" in [`themes/README.md`](./themes/README.md). ## Deliberate deviations from the fleet baseline diff --git a/checks/README.md b/checks/README.md index 83bc682..89d866a 100644 --- a/checks/README.md +++ b/checks/README.md @@ -84,6 +84,18 @@ A count is all the check can observe, and two causes reach each direction: it ri **Both directions read absolute references as well as relative ones.** Hugo writes an absolute URL wherever a template resolves one against the base, which the entry-cover image on every list page does. Reading only rooted paths made those files look linked from nowhere while they were being displayed, and left a broken one unchecked in the other direction. The origin is read from the home page's canonical link rather than assumed, since staging and production build with different base URLs and a hardcoded host would check one environment's output against another's. No canonical link is a hard failure, because a guessed origin inflates the orphan count by exactly the pages that use one. +## The robots check, and the one thing no gate here can do + +Hugo emits no `robots.txt` unless `enableRobotsTXT` is set, so this site served none until the flag was turned on, and the old platform serves one. The `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first full day of traffic, every request for `sitemap.xml` came from `curl` and none from a crawler, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted. + +**The `Sitemap:` line is the one place this gate compares an origin rather than joining one.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments. Other absolute URLs are read here, the home page's canonical link and the absolute asset references described above among them, but they are read to resolve a reference rather than to check one host against another. + +**What that comparison proves is internal consistency, and it is worth being exact about the limit.** The advertised origin must match the one read from the home page's canonical link, and both are derived from the same `baseURL`, so they agree whenever the build is coherent — including when `baseURL` held the wrong value for the environment being deployed to. **Nothing inside the artifact can detect that**, which is why the check belongs on the side that knows which host it is serving: the VPS side reads the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml` and reports the counts either way. A build baked with the wrong host still passes all 1,245 URLs here, and it passes this too. + +What the comparison does catch is an origin that was **written rather than derived** — a committed `static/robots.txt` shadowing the template is the way that happens, and pasting the old platform's `.com` sitemap line into one is the specific mistake it would catch — along with a sitemap advertised but never built. + +`/robots.txt/`, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolves through `slugs.map` like any other one-segment legacy URL, and the generator special-cases it to the real file rather than to the home page. `/osd.xml/` stays pointed at the home page deliberately: it was the old platform's OpenSearch description and this site emits no such file. + ## The gallery check, which no direction above can reach Every check above reasons about a URL: whether it renders, whether it resolves, whether anything points at it. Content misplaced **inside** a gallery satisfies all of that. The file exists, the reference resolves, and something links it, so the media surface is green in both directions while the page is laid out wrong. The defect is one of structure, which is why three variants of it survived the conversion and every gate since. diff --git a/checks/build-redirects.py b/checks/build-redirects.py index 414ef9f..3d94171 100755 --- a/checks/build-redirects.py +++ b/checks/build-redirects.py @@ -19,6 +19,15 @@ # The WordPress importer registers the full slug, and both answer, so both are mapped. BLOGGER_SLUG_LIMIT = 40 +# A one-segment URL naming a file the site serves at the root, rather than a page slug. +# The resolver below reads these as unresolvable attachment slugs and sends them to the home +# page, which is the right answer for a slug nothing claims and the wrong one for a file that +# exists: /robots.txt/ should reach /robots.txt. Named here rather than hand-edited into the +# generated map, because the map is rewritten from the capture and a hand edit does not survive. +# /osd.xml/ stays out deliberately. It was the old platform's OpenSearch description and this +# site emits no such file, so the home page remains the honest destination for it. +WELL_KNOWN = {"/robots.txt/": "/robots.txt"} + def text(el, path): node = el.find(path, NS) @@ -179,7 +188,9 @@ def main(argv): resolved, via_parent, via_media, orphan = [], 0, 0, [] for u in needed: slug = u.strip("/") - if slug in by_slug: + if u in WELL_KNOWN: + resolved.append((u, WELL_KNOWN[u])) + elif slug in by_slug: resolved.append((u, by_slug[slug])) via_parent += 1 elif slug.lower() in file_to_post: diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 2d017fa..86a24e5 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -41,8 +41,15 @@ # Every check above returns a list, and the shared summary called all of them "missing". That is # what a URL that did not build is, and it is not what a stray node inside a gallery is: those are # present, which is the whole complaint. The default stays "missing" so a check added later reads -# the way the older ones do unless it says otherwise. -FAILURE_NOUN = {"gallery": "stray nodes"} +# the way the older ones do unless it says otherwise, and it needs no pair because it is already +# count-neutral. A count is always printed beside the noun, so the pair is (singular, plural) and +# "1 stray nodes" was the reason for making it a pair rather than a string. +FAILURE_NOUN = {"gallery": ("stray node", "stray nodes"), "robots": ("problem", "problems")} + + +def failure_noun(label, count): + forms = FAILURE_NOUN.get(label, ("missing", "missing")) + return forms[0] if count == 1 else forms[1] def load(name): @@ -92,6 +99,65 @@ def check_media(public): return missing +def check_robots(public): + """Check that robots.txt exists and advertises this build's own sitemap. + + Four failures, one gate: the file absent, no Sitemap: line, a line naming another origin, and a + line advertising a sitemap that was not built. Hugo emits no robots.txt unless enableRobotsTXT + is set, which is why this site served none until that flag was turned on, and the file's only + load-bearing line points at the sitemap a crawler is otherwise unlikely to find, being told + rather than guessing. + + Every contract list is path-only and check-live-urls.sh joins whatever base URL it is handed, so + this is the only place the gate *compares* an origin rather than joining one. Other absolute + URLs are read here - the home page's canonical link, and absolute asset references - but they + are read to resolve a reference rather than to check a host against another. + + What the comparison proves is internal consistency: the advertised origin matches the one the + canonical link carries. Both come from baseURL, so this cannot tell that baseURL was the wrong + value for the environment being deployed to - nothing in the artifact can, which is why that + check belongs on the side that knows which host it is serving. It does catch an origin that was + written rather than derived, a static robots.txt shadowing the template being the way that + happens, and it catches a sitemap advertised but not built. + """ + robots = public / "robots.txt" + if not robots.is_file(): + # Naming one cause as the cause sends a reader to check a setting that is already correct. + # enableRobotsTXT is the likely one and a partial build or the wrong output directory reach + # the same state, which is the same reason the orphan messages name both of their causes. + print("robots : missing") + return [ + f"{robots} does not exist - likely enableRobotsTXT is unset in hugo.yaml, " + "though a partial build or the wrong output directory look identical here" + ] + + origin = site_origin(public) + # errors="replace" rather than strict, or invalid UTF-8 raises out of the whole parity run and a + # gate that exists to report a bad robots.txt stack-traces on one instead. A committed + # static/robots.txt is the file most likely to carry it, and it is the case this check is for. + text = robots.read_text(encoding="utf-8", errors="replace") + advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", text) + if not advertised: + print("robots : built, no Sitemap line") + return ["robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided"] + + wrong = [u for u in advertised if not u.startswith(origin + "/")] + if wrong: + print(f"robots : built, {len(wrong)} Sitemap line(s) naming another origin") + return [f"{u} (this build's origin is {origin})" for u in wrong] + + # The summary is printed after the last assertion rather than before it, or the missing-sitemap + # case reads as a pass on the line above its own failure. Every branch here prints exactly once. + unbuilt = [u for u in advertised if not (public / u[len(origin) + 1 :]).is_file()] + if unbuilt: + print(f"robots : built, {len(unbuilt)} advertised sitemap(s) not built") + return [f"{u} (advertised, but {u[len(origin) + 1:]} was not built)" for u in unbuilt] + + # Named rather than counted, since there is one today and the line is the thing being checked. + print(f"robots : built, advertising {advertised[0]}") + return [] + + def site_origin(public): """The site's own scheme and host, read from the artifact rather than assumed. @@ -307,6 +373,7 @@ def main(argv): ("assets", check_assets(public, refs)), ("orphans", check_orphans(public, refs)), ("gallery", check_galleries(public)), + ("robots", check_robots(public)), ): if found: failures.append((label, found)) @@ -317,7 +384,7 @@ def main(argv): print() for label, found in failures: - print(f"FAIL {label}: {len(found)} {FAILURE_NOUN.get(label, 'missing')}") + print(f"FAIL {label}: {len(found)} {failure_noun(label, len(found))}") for item in found[:20]: print(f" {item}") if len(found) > 20: diff --git a/deploy/maps/slugs.map b/deploy/maps/slugs.map index d27247c..2eb63c0 100644 --- a/deploy/maps/slugs.map +++ b/deploy/maps/slugs.map @@ -100,7 +100,7 @@ /jellyfish-lighting-ports-500x214-1/ / /osd.xml/ / /packet/ /2020/09/05/abandoning-greeneye-for-sense/ -/robots.txt/ / +/robots.txt/ /robots.txt /sce_smartmeter/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-dashboard/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-graphing/ /2020/09/05/abandoning-greeneye-for-sense/ diff --git a/hugo.yaml b/hugo.yaml index 2ec8d0f..ded53f7 100644 --- a/hugo.yaml +++ b/hugo.yaml @@ -27,6 +27,15 @@ refLinksErrorLevel: ERROR # The deploy rebuilds the site rather than receiving it as an artifact, so the build stays reproducible. enableGitInfo: false +# Off by default in Hugo, which is why this site answered 404 for a file the old platform served. +# The Sitemap: line is the load-bearing part: across the interim hostname's first full day of +# traffic, every request for sitemap.xml came from curl and none from a crawler, because a crawler +# is told where a sitemap is rather than guessing, and the only thing telling them today is the +# robots.txt the cutover deletes. +# The theme's template derives that URL from the built baseURL, so it names .net during the +# rehearsal and .com afterwards with no second edit to remember at the cutover. +enableRobotsTXT: true + params: description: My discoveries, frustrations, and delights defaultTheme: auto