From cece2b0c78c163eea0d4648b70b36a6a1e333fa1 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Fri, 7 Aug 2026 21:02:21 -0700 Subject: [PATCH 1/2] Record operations as recurring work, not as a finished milestone The backlog listed the log review and the off-host log pull among the one-off migration tasks, so a reader finishing that list would read the migration as done while neither duty had ever run on its cadence. Both move to a Recurring operations section that says outright that everything above it ends and this does not, and the State table gains an Operations row so the summary stops implying the pipeline is the whole system. The open decision behind them is that the tooling has no home in git. vps-backup-pull, its systemd units, and the variables naming both ends of the copy exist only on the Proxmox host, which is the host the backup runs from, so losing it loses both the copies and the means of making them. Two candidate homes are recorded without choosing between them. The same question covers the fourteen migration scripts in the capture directory, and one of them already shows the cost of leaving it open: build-redirects.py exists in three places, the two capture copies identical to each other at 115 lines and this repo's maintained copy at 225. The capture is not a git repository, so nothing detects that. Two open questions are closed with measurements rather than assumptions. The production environment's HUGO_BASEURL held the live WordPress address, which the deploy workflow both builds with and points the live check at, and it is now the interim name. And the first production deploy does not fix the robots.txt 404 the VPS agent flagged in its 22.10: hugo.yaml sets no enableRobotsTXT so the site emits none at all, while sitemap.xml is emitted and becomes fetchable at that same deploy. The VPS production row said the resource was deliberately disabled, which its own file contradicted twenty lines later. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/TODO.md b/TODO.md index e419543..a03f41a 100644 --- a/TODO.md +++ b/TODO.md @@ -17,7 +17,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical | | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | -| VPS production | environment configured, resource deliberately disabled, DNS still on the old platform | +| VPS production | reachable at `blog.insanegenius.net`, answering `200` unauthenticated, still serving `0000-placeholder`. DNS for the public name is still on the old platform | +| Operations | not started. Neither the periodic log review nor the off-host log pull has run on its cadence | ## Blocked on the maintainer @@ -30,16 +31,22 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Declare what the VPS keeps.** `hugo.deploy.retention` asks for a retention count declared at the destination, and this repo's deploy credential is write-only by design so the prune belongs to the host. The ownership is recorded in `OPERATIONS.md`, the count is not, and the "ten releases" beside it describes `deploy/make-release.sh` on the local mirrors rather than the containers on the VPS. Confirm the host's timer and its count, then write it next to the ownership line. - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. - **Deploy production once, to a name that is not the live one.** The host side is done and the interim name is live: `blog.insanegenius.net` answers `200` unauthenticated, on a Let's Encrypt certificate issued 2026-08-07, serving `0000-placeholder`. The VPS agent calls this M7a, and M7b is the `.com` cutover. Three items belong to this repo before the first production deploy, all detailed in the VPS agent's §19 and §20. That file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": - - **Set `HUGO_BASEURL` to `https://blog.insanegenius.net` on the `production` environment**, and back to `.com` at M7b. Hugo bakes `baseURL` into the canonical tags, the feed, and `sitemap.xml`, so a deploy without this edit serves a site whose every absolute URL points at the live WordPress site. It works mechanically and it is wrong. + - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - - **Answer the two questions in §19.3**: what `HUGO_BASEURL` holds on `production`, and whether anything else in the build or the checks hardcodes `blog.insanegenius.com`. + - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. - **Decide what `robots.txt` says before the first production deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. + - **The first production deploy does not fix the 404, which is the part that reads the wrong way round.** The VPS agent raised this in §22.10 and asked two questions of this side. Both are now measured rather than assumed: **this site emits no `robots.txt` at all**, because `hugo.yaml` sets no `enableRobotsTXT`, so the 404 survives the deploy and `X-Robots-Tag` stays the only control. **`sitemap.xml` is emitted** and becomes fetchable on the interim name at that same deploy, listing whatever `HUGO_BASEURL` was built with, which is the second reason that variable mattered. Read together: after the rehearsal deploy a crawler gets a full sitemap and no robots file. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. -- **"Backed up" describes a design rather than a fact, for the log.** The VPS holds the only copy of the access log until the pull to the backup host runs, and that log rotates and ages out on a schedule the host owns, so the window is the deadline. The pull is two rsync lines the maintainer adds on the backup host, and the log stays deliberately outside the nightly encrypted archives because those are full copies with no dedupe. State that plainly in anything built on top of the log until the pull runs. -- Watch server logs for non-200s daily for the first week, then monthly, because real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site. +## Recurring operations + +**Everything above ends; this section does not.** The list above finishes when the site serves its public address, and these two start there and never stop. Both are written down as procedures and neither has run on its cadence yet, which is the shape a recurring duty has right before it silently becomes a one-off. An end-to-end exercise of this system is a deploy **plus** the operations that follow it, so a run that proves staging and production and stops has proven the half that has an obvious finish line. + +- **Review the logs for non-200s**, daily for the first week after cutover, then monthly. Real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor, and a staging probe for `/wp-login.php` answered by the auth gate rather than by the site is the shape of what Caddy never sees. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". +- **Pull the log off the VPS, on a schedule**, which is [#53][issue-53]. The access log is deliberately outside the nightly encrypted archives, because those are fourteen full copies with no dedupe and an append-only file would be multiplied by fourteen for no recovery benefit. So the VPS's 400-day window is the **only** copy until this runs. The two rsync lines are now written, parameterized, and tested against the live host, and they are not installed, because installing needs root. **Until they are, say plainly that "backed up" describes a design rather than a fact in anything built on top of that log.** + ## Owed to the hub Nothing. The spec update this repo owed the hub has landed: [ProjectTemplate#560][hub-type-pr] authored the `hugo` type, the `self-hosted` target, the `deploy-ssh` mechanism, guarantees D4.6 and D5.6, and a reference leaf pair, all measured from what this repo actually runs rather than from the prediction the intake carried. [#456][hub-issue] and [#558][hub-spec-issue] are closed with it. @@ -55,6 +62,10 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions +- **Where the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy are an operational asset built from another agent's instructions, and they exist only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. + - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. + - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. +- **Gallery captions render as a stray text node inside the grid** rather than as a caption, which is [#52][issue-52]. Twelve occurrences across two pages, from one source file, and it is a conversion artifact rather than anything the recent content work introduced. A fix has to decide whether the caption belongs to the last image or to the set, and the provenance capture is the only source for what the old platform actually displayed. - `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`. - Content is capped at a fixed 720px on every screen, because PaperMod's width is four CSS variables with no responsive term and no Hugo parameter. The prose measure is right and should stay; images and galleries inheriting the same cap is the part that costs something on a wide display. The knobs, the override location, and the `--gap` trap are documented under "Customization points" in [`themes/README.md`](./themes/README.md). @@ -148,6 +159,8 @@ The deploy root is deliberately absent from this table. The rsync destination is [issue-33]: https://github.com/ptr727/Blog/issues/33 +[issue-52]: https://github.com/ptr727/Blog/issues/52 +[issue-53]: https://github.com/ptr727/Blog/issues/53 [migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md [pr-30]: https://github.com/ptr727/Blog/pull/30 [pr-40]: https://github.com/ptr727/Blog/pull/40 From 4d64eedad38adc1515165270829c281284ac8e3d Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Fri, 7 Aug 2026 22:02:34 -0700 Subject: [PATCH 2/2] Record the robots.txt cutover loss and the media gate gap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gallery caption entry under Open decisions is gone: #55 answered the question it posed, against the captured live site, and closed #52 with it. Two findings from the VPS agent's §23.3 and §24.3 replace it. The robots.txt entry described a gap on the interim name and missed that the cutover makes it a loss. The live .com blog serves a robots.txt today carrying a Sitemap: line, and this site emits none, so M7b moves a site that has had crawl directives for years to having none and takes the sitemap pointer with them. That side found it from the outside, will not put a file in this bundle, and has made it block step 1 of the M7b checklist rather than be discovered after it. Recorded as deliberately undecided with the three options, and Open decisions points at it rather than restating it, so the two cannot drift. Nothing checks that media survived the trip to the server. A 3,095-request gate run fetched no image at all, which prompted the question of whether the legacy media list is wired in. It is, at build time, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost between the build and the server is caught by neither, the build having passed before the loss and the live gate never asking. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index a03f41a..e65030c 100644 --- a/TODO.md +++ b/TODO.md @@ -36,6 +36,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. - **Decide what `robots.txt` says before the first production deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - **The first production deploy does not fix the 404, which is the part that reads the wrong way round.** The VPS agent raised this in §22.10 and asked two questions of this side. Both are now measured rather than assumed: **this site emits no `robots.txt` at all**, because `hugo.yaml` sets no `enableRobotsTXT`, so the 404 survives the deploy and `X-Robots-Tag` stays the only control. **`sitemap.xml` is emitted** and becomes fetchable on the interim name at that same deploy, listing whatever `HUGO_BASEURL` was built with, which is the second reason that variable mattered. Read together: after the rehearsal deploy a crawler gets a full sitemap and no robots file. + - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the `.com` sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **Deliberately undecided**, and the three options are to preserve today's behavior, to write what this site actually wants, or to keep emitting nothing and accept the loss. +- **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site. @@ -65,7 +67,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no - **Where the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy are an operational asset built from another agent's instructions, and they exist only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. -- **Gallery captions render as a stray text node inside the grid** rather than as a caption, which is [#52][issue-52]. Twelve occurrences across two pages, from one source file, and it is a conversion artifact rather than anything the recent content work introduced. A fix has to decide whether the caption belongs to the last image or to the set, and the provenance capture is the only source for what the old platform actually displayed. +- **What `robots.txt` says, which is undecided and is the last non-mechanical item before M7b.** Recorded under "Next" above, where it blocks the cutover. - `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`. - Content is capped at a fixed 720px on every screen, because PaperMod's width is four CSS variables with no responsive term and no Hugo parameter. The prose measure is right and should stay; images and galleries inheriting the same cap is the part that costs something on a wide display. The knobs, the override location, and the `--gap` trap are documented under "Customization points" in [`themes/README.md`](./themes/README.md). @@ -159,7 +161,6 @@ The deploy root is deliberately absent from this table. The rsync destination is [issue-33]: https://github.com/ptr727/Blog/issues/33 -[issue-52]: https://github.com/ptr727/Blog/issues/52 [issue-53]: https://github.com/ptr727/Blog/issues/53 [migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md [pr-30]: https://github.com/ptr727/Blog/pull/30