diff --git a/OPERATIONS.md b/OPERATIONS.md index 1c35786..5ac3142 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -46,9 +46,32 @@ This site has served the same domain across earlier platforms, so its whole oper **The contract is ground truth.** [`checks/golden-urls.txt`](./checks/golden-urls.txt) and [`checks/redirect-urls.txt`](./checks/redirect-urls.txt) record URLs verified with a live request, not predicted from the content tree. The lists are **append-only**: nothing legitimately removes a URL the site has served, so a change that would drop one is a change to reject rather than a list to shorten. A list-driven check also carries a length floor, or a truncated list passes while checking almost nothing. +## The Migration Record + +**The migration is documented once, as a post on the site, and that post is the artifact to reference.** [`content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md`](./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md) holds how the URL surface was captured, why the contract splits into a render half and a redirect half, why the Blogger permalink map needs more entries than the posts it covers, why the media had to come from the export tar and be hash-verified, and which Hugo taxonomy default moves every archive to a new address without reporting anything. + +**Read it before changing anything under [`checks/`](./checks/) or [`deploy/maps/`](./deploy/maps/).** Both hold values that no code derives and no test explains, and the reasoning behind them is in the post rather than beside them. Cite the post rather than restating it. This file is the procedure and the post is the account of how the procedure came to be, so where the two disagree this file governs what to do while the post explains why the check exists. + +**The post is content, so it sits under the URL contract.** Editing it moves nothing. Renaming it or taking it down breaks an address the site serves. A fact in it that proves wrong is corrected in the post rather than footnoted here. + +### Rebuilding from the Exports + +Everything derived is in this repository. Everything it was derived *from* is in a capture directory outside it, which is where a rebuild starts. **The capture path is `CAPTURE_ROOT` in `secrets/.env`**, recorded alongside the other values that name a machine rather than the project, so it is read from there rather than searched for. The capture is not a git repository, so it has no history to revert to, and it is read-only in normal use. + +| Under the capture | Holds | Recoverable | +| --- | --- | --- | +| `export/raw/` | the WordPress content export, WXR XML | yes, from the WordPress account while it exists | +| `export/media-tar/` | the media export, the only trustworthy copy of the images | yes, from the same place | +| `mirror/` | a crawl of the old platform as it served, including the media it linked from other hosts | no, once the old hosting ends | +| `inventory/` | the URL and media inventories derived from that crawl | no, for the same reason | + +The two exports are the only inputs a person has to fetch, and `EXPORT-INSTRUCTIONS.md` at the root of the capture records which two menu items produce them and the counts each has to reconcile against. The counts are the point, because a partial export is the common way a migration loses posts without reporting anything. + +[`checks/build-redirects.py`](./checks/build-redirects.py) takes the capture directory as its one argument and rebuilds everything under `deploy/maps/` from it. It selects the export **by content** rather than by filename and fails unless exactly one candidate holds published posts, because the capture also holds a media-only export whose zero posts produce empty maps that are indistinguishable from working ones until the redirects are live. + ## Local Verification Before a Pull Request -**CI cannot prove a redirect.** The validation workflow builds the site and checks the render half of the contract, which is every URL that must return a page. The other 917 URLs are the web server's job, and nothing in a build exercises them. A change to the Caddy config or to a generated map is therefore invisible to CI: the workflow goes green while the redirect it broke stays broken until someone follows a sixteen-year-old link. +**CI cannot prove a redirect.** The validation workflow builds the site and checks the render half of the contract, which is every URL that must return a page. Most of the contract is not pages, and those URLs are the web server's job, which nothing in a build exercises. A change to the Caddy config or to a generated map is therefore invisible to CI: the workflow goes green while the redirect it broke stays broken until someone follows a sixteen-year-old link. So release to the local mirror and run the live check **before** opening a pull request that touches any of these: @@ -93,9 +116,9 @@ Sourcing the environment file first puts the deploy root and the base URL in the `ENV_FILE` is set as well as sourced, and the redundancy is deliberate. The script sources its own file regardless, so leaving `ENV_FILE` off would build and install against `secrets/.env` while the shell's `$HUGO_BASEURL` still named staging, and the run would check the staging site after publishing to the production root. The script prints the file it read, on every build, for that reason. -It refuses to install a release that fails the build gate. `check-live-urls.sh` does take a base URL, which is where the sourced `$HUGO_BASEURL` goes. It follows all 1,245 URLs against the running mirror, checking each redirect's destination rather than trusting its status code. +It refuses to install a release that fails the build gate. `check-live-urls.sh` does take a base URL, which is where the sourced `$HUGO_BASEURL` goes. It follows every URL in the contract against the running mirror, checking each redirect's destination rather than trusting its status code. -Expect `PASS - 1245 URLs honored`. Anything less is a finding, and the output names each URL that failed and what it answered. +Expect a `PASS` naming the number of URLs honored, which is the two lists' combined length and grows as they do. Anything less is a finding, and the output names each URL that failed and what it answered. A documentation-only or workflow-only change does not need this. A change to the four paths above does, because for those CI's green is not evidence. @@ -117,9 +140,9 @@ Set both or neither. Half a pair is a typo rather than a choice, and it is rejec Three properties of how the credential is handled, each there for a reason worth keeping: -- **It travels in a mode-`600` curl config file, not in `-H` arguments.** A command line is readable in `ps` for the life of the process, and this runs 1,245 of them. The config file is also the only form that survives the `export -f` the parallel checks run under, because bash cannot export an array. +- **It travels in a mode-`600` curl config file, not in `-H` arguments.** A command line is readable in `ps` for the life of the process, and this runs one per URL in the contract. The config file is also the only form that survives the `export -f` the parallel checks run under, because bash cannot export an array. - **It is sent to the base URL's own origin and nowhere else.** The check follows every redirect's destination, and every destination in the contract is same-origin today. A rule that one day points off-site must not mail the credential to whoever is on the other end. -- **A preflight request runs before the 1,245.** Behind an auth gate a wrong token fails *every* URL, and the output then reads as a site that has vanished rather than as a bad credential. The two are indistinguishable from the far end of a CI log, so the run stops on the first request with a message naming which of the two it was. +- **A preflight request runs before the rest.** Behind an auth gate a wrong token fails *every* URL, and the output then reads as a site that has vanished rather than as a bad credential. The two are indistinguishable from the far end of a CI log, so the run stops on the first request with a message naming which of the two it was. ## Deploying @@ -173,7 +196,7 @@ The script asserts both halves of that rather than assuming them. It fails when ## Log Review -**Real traffic is the only source that finds what every check here is blind to.** The URL contract proves the 328 URLs someone thought to list and the 917 redirects derived from the export. It cannot know about a URL nobody recorded, because the crawl that produced the lists is the same crawl the lists are checked against. A visitor following a sixteen-year-old link is the one reader who tests that. +**Real traffic is the only source that finds what every check here is blind to.** The URL contract proves the URLs someone thought to list and the redirects derived from the export. It cannot know about a URL nobody recorded, because the lists are their own standard: the gates check the built site and the running server against those lists, never against the old platform that served the addresses. An address the crawl missed is therefore missing from every gate that reads them, and a visitor following a sixteen-year-old link is the one reader who tests for it. Review runs in both directions, which are the same two the media checks read and have the same blind spots for the same reason. @@ -184,7 +207,7 @@ Review runs in both directions, which are the same two the media checks read and **The outward pass is the one with an action.** A 404 on a path shaped like real content means the golden list missed a URL: add it to [`checks/golden-urls.txt`](./checks/golden-urls.txt) and add a redirect, per that file's own maintenance rules. Expect the raw counts to be dominated by scanners probing for `wp-login.php`, `.env`, and `.git/config`, which is noise from a site that used to run WordPress and should be filtered by shape rather than investigated. -**The inward pass answers a question nothing else can.** Subtracting every URL that has ever returned 200 from the set the site builds names the content no reader has reached. It is slow evidence and deliberately so, since a post can go a year without a visit and still be worth keeping. Its first concrete use is the 97 carried media files that no page links and that the old platform never published, recorded in [`checks/README.md`](./checks/README.md): if nothing requests them across a year, that settles whether carrying them is preservation or clutter, and no reasoning from the repository alone can settle it. +**The inward pass answers a question nothing else can.** Subtracting every URL that has ever returned 200 from the set the site builds names the content no reader has reached. It is slow evidence and deliberately so, since a post can go a year without a visit and still be worth keeping. Its first concrete use is the carried media that no page links and that the old platform never published, counted exactly by the parity gate and broken down in [`checks/README.md`](./checks/README.md): if nothing requests those files across a year, that settles whether carrying them is preservation or clutter, and no reasoning from the repository alone can settle it. ### The log is three tiers, and each is blind to something @@ -192,19 +215,27 @@ A request crosses the proxy before it reaches the site, so no single log answers | Tier | Sees | Cannot see | | --- | --- | --- | -| Traefik, or Pangolin's Traefik on the VPS | every request reaching the host, including unknown hostnames, TLS failures, and traffic aimed at names this site does not serve | which release answered, or anything about the site's own routing | +| Traefik, or Pangolin's Traefik on the VPS | every request reaching the host, including unknown hostnames, TLS failures, and traffic aimed at names this site does not serve | which release answered, since Traefik logs request headers and not response headers | | Pangolin, on the VPS only | requests the auth gate rejected | anything on the local mirrors, which have no gate | | Caddy, per environment | path, status, and the `X-Blog-Release` that answered | anything the tiers above rejected, which never arrives | **A 404 count taken from Caddy alone is therefore a floor, not a total.** A request the edge refused is a reader who found nothing just as surely, and it appears in no Caddy log. Read the edge for what never arrived and Caddy for what arrived and failed, and treat the two as one answer. -Two properties of the Caddy side worth knowing before parsing it. Its access log is `format console`, so each line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON itself, and a parser that assumes one object per line reads nothing. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, which is the same setting "Serving" describes as a security boundary — without it every request in the log appears to come from one internal address and the inward pass cannot distinguish a reader from a health check. +Two properties of the Caddy side are worth knowing before parsing it. Its access log is `format console`, so each line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON itself, and a parser that assumes one object per line reads nothing. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, which is the same setting "Serving" describes as a security boundary. Without it every request in the log appears to come from one internal address, and the inward pass cannot distinguish a reader from a health check. + +### Retention Is the Prerequisite, and It Belongs to the Host + +**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=` traffic is visible as itself. Request headers are dropped except `Referer` and `User-Agent`, which is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. + +**That log rotates and is eventually deleted, on a schedule the host sets and can change.** The window is long, and it is finite, so anything the inward pass depends on has to be copied off the host before the archive ages out. Read the current retention from the host rather than from this file, because a number written here is a number nothing checks. + +**Caddy's container log is the runtime log rather than the access log**, bounded by the container's own log rotation. It is where a failed config load and a dead `--watch` surface. It is not durable across a container recreate, since the Docker `json-file` log lives under the container id, and an operator editing the compose file is the event that discards it. A release deploy is not: an rsync and a symlink flip run no Docker operation at all. -### Retention is the prerequisite, and it is not configured +**Release attribution comes from a join rather than from a header.** Traefik cannot log a response header, so no access-log line names the release that answered. Join `StartUTC` against the release flip instead, which is exact outside a deploy window and ambiguous only inside one. -**The containers log to Docker's `json-file` driver with no rotation and no size limit.** `/etc/docker/daemon.json` sets only a storage driver, so the built-in defaults apply: the file grows without bound, and it is discarded entirely when the container is recreated. Both halves defeat a periodic review — the first is a disk risk on a VPS, and the second silently resets the window that the inward pass depends on being long. +**Retention on the local mirrors is a different question and is unsolved.** Those containers use Docker's `json-file` driver with the built-in defaults, so a mirror's log grows without bound and is discarded when its container is recreated. That matters less than it did on the VPS, because the mirrors serve no readers, and it belongs to the host rather than to this repository, the same split "Retention" and "Who Owns What" describe for release pruning. -Nothing here fixes that, because the log driver belongs to the host rather than to this repository, the same split "Retention" and "Who Owns What" describe for release pruning. **Settle it before the first review is expected to mean anything**, since a review reads whatever survived and reports confidently on it either way. +**The off-host copy of the access log is a design rather than a fact.** The VPS holds the only copy until the pull to the backup host runs, which makes the host's retention window the hard deadline rather than a comfortable margin. A review reads whatever survived and reports confidently on it either way, so settle this before the first review is expected to mean anything. ## Who Owns What @@ -295,7 +326,7 @@ Everything inside the bundle, `deploy/Caddyfile` and anything under `deploy/maps ## Redirects -The site answers 917 addresses it does not render, satisfied by 13 `redir` directives reading 5 map files, all inside the bundle. [`deploy/README.md`](./deploy/README.md) carries the per-class breakdown and the counts. This section covers the operational shape only, so the two do not restate each other. +The site answers far more addresses than it renders, satisfied by a small set of `redir` directives and the map files they read, all inside the bundle. [`deploy/README.md`](./deploy/README.md) carries the per-class breakdown and the counts. This section covers the operational shape only, so the two do not restate each other. Ordering is load-bearing, so every redirect lives in a single `route` block. Outside one, Caddy sorts directives by its own precedence rather than by file order, and the broad attachment rule claims the per-post comment feeds that the narrower rule must match first. diff --git a/README.md b/README.md index b25ad3b..a81aa2e 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,8 @@ See [Release History][history] for complete release notes and older versions. ## Table of Contents - [Use Cases](#use-cases) +- [Migration from WordPress](#migration-from-wordpress) +- [How a Change Reaches the Site](#how-a-change-reaches-the-site) - [Configuration](#configuration) - [Questions or Issues](#questions-or-issues) - [Development Environment Setup](#development-environment-setup) @@ -52,6 +54,97 @@ The site answers far more addresses than it renders pages, because it has served Deployment is a release directory plus a symlink. A build is installed alongside its predecessors, verified, and made live by swapping one link, so a rollback is the same swap in reverse. See [OPERATIONS.md][operations]. +## Migration from WordPress + +The site has served the same domain since 2008, across three platforms: Blogger until 2012, WordPress until 2026, and Hugo from then on. Converting the posts took an afternoon. Preserving sixteen years of inbound links was the work, and it is why this repository carries a URL contract and gates it rather than trusting the build. + +The account of that migration is a post on the site, [Moving This Blog From WordPress to Hugo][migration-post]. It covers what a WordPress export holds and what it leaves out, why the sitemap named barely a tenth of the addresses the site was actually serving, how the Blogger-era permalinks resolve through a lookup table rather than a pattern, why media fetched over HTTP is not the same bytes as the media in the export and only a content hash tells them apart, and which Hugo default moves every taxonomy archive to a new address without reporting anything. + +## How a Change Reaches the Site + +A change starts on a branch in this repository and ends as bytes on a VPS. Every stage carries a gate, and three of them add a human read, because the failure this repository exists to catch is an address that stops answering, and no check that skips a running server can see one. + +Both deploys are manual workflow dispatches, so a merge publishes nothing and each environment goes live by a deliberate run. + +[`WORKFLOW.md`][workflow] states the same CI machinery as a contract for tooling, and [`OPERATIONS.md`][operations] holds the procedure to run at a keyboard. + +```mermaid +flowchart LR + subgraph local["1. On a branch"] + direction TB + w["New branch, write the post"] --> b["Build, warnings are fatal"] + b --> p["Gate: every page that must render, renders"] + p --> m["Release onto the local mirror"] + m --> l["Gate: every URL in the contract, against a running server"] + l --> e1["Human: read the page"] + end + + subgraph gh["2. On GitHub"] + direction TB + pr["Pull request into develop"] --> ci["CI: lint, spelling, build, URL parity"] + ci --> rv["Automated review, then a human squash merge"] + end + + subgraph stg["3. Staging, behind the auth gate"] + direction TB + ds["Dispatch: deploy staging"] --> ls["Gate: every URL in the contract, run by CI"] + ls --> e2["Human: read staging"] + end + + subgraph prd["4. Production, public"] + direction TB + pm["Pull request: develop into main"] --> dp["Dispatch: deploy production"] + dp --> lp["Gate: every URL in the contract, run by CI"] + lp --> e3["Human: read production"] + end + + e1 --> pr + rv --> ds + e2 --> pm +``` + +**1. Write on a branch, and prove the artifact locally.** A post is a markdown file under `content/posts/`, written on a feature branch with whatever editor the author prefers. The build treats a warning as fatal, so a deprecated theme API fails it rather than accumulating. The build gate then checks the render half of the contract, which is every address that must return a page. + +Most of the contract is not pages, though. It is redirects, and a redirect is the web server's job, so no build reaches them. The release installs onto a mirror on the maintainer's own network, which runs the same Caddy container and the same bundle as the server behind the same Traefik front end. The live gate follows every URL in the contract against that mirror and checks each redirect's destination rather than its status code. A human then reads the page, because no gate has an opinion about the writing. + +**2. Open a pull request into `develop`.** CI runs lint, spelling, workflow and config validation, the build, and the render gate on a clean checkout. It does not prove the redirects, which is why the local run is a prerequisite rather than a convenience: a change to the Caddy config or to a redirect map goes green in CI while the redirect it broke stays broken. An automated review runs against the branch, and a human squash-merges it. + +**3. Deploy to staging.** A dispatched workflow builds the commit, uploads the release, flips the symlink, and runs the same live gate from CI against the running site. The local mirror proves the artifact, and staging proves the infrastructure that exists only on the server: routing, TLS, the deploy key, and its confined transport. Staging keeps its authentication gate on and the check presents a token, so a byte-identical copy of the public site is never exposed to a crawler. A human then reads it. + +**4. Promote, and deploy production.** `develop` merges into `main` through a pull request, and production is a second dispatch that accepts `main` alone. The same gate runs a third time, against the public address and with no token, and a human reads the result. + +Each branch feeds one environment: + +```mermaid +flowchart LR + feat["feature branch"] -->|squash| dev["develop"] + dev -->|merge commit| main["main"] + dev -.->|manual dispatch| stg["staging site"] + main -.->|manual dispatch| prod["production site"] +``` + +### What Happens While the Site Runs + +Publishing is half of it. The other half runs on its own cadence, because the contract proves only the addresses someone wrote down. + +```mermaid +flowchart LR + site["Production, serving"] --> log["Edge access log, one line per request"] + log --> outward["Outward: asked for, and not here"] + log --> inward["Inward: here, and never asked for"] + outward --> add["Add the address to the contract, add a redirect"] + add --> next["Ships with the next change, top of the pipeline"] + inward --> judge["Decide whether unread content is preservation or clutter"] + site --> cfg["Host configuration archived off the VPS"] + cfg --> rebuild["A rebuild restores the config and redeploys"] +``` + +**Real traffic is the only source that finds what the contract misses.** Every request to the host is recorded at the edge, one line per request, and the review runs in two directions. The outward pass reads non-200 responses. A 404 on a path shaped like real content means an old link nobody recorded, and the fix is to add the address to the contract and add a redirect, which then ships through the pipeline above like any other change. Scanners probing for `wp-login.php` dominate the raw count and are filtered by shape rather than investigated. The inward pass subtracts every address that has ever answered 200 from the set the site builds, which names content no reader has reached. That evidence accumulates slowly, and it is the only thing that settles whether media the old platform never published is worth carrying. + +**Backups protect the server rather than the site.** The site is reproducible from this repository by running a deploy, so what is worth keeping is the host's configuration: the container definitions, the proxy configuration, and the deploy account with its restricted key. A bare-metal restore rebuilds the host, restores that configuration, and deploys again. A procedure that backs up the deploy root protects a copy of something git already holds. + +[OPERATIONS.md][operations] holds the detail: the commands, the environments, the rollback, and the three log tiers with what each cannot see. + ## Configuration | Path | Holds | @@ -126,8 +219,10 @@ Licensed under the [MIT License][license]\ [history]: ./HISTORY.md [hugo-config]: ./hugo.yaml [license]: ./LICENSE +[migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md [operations]: ./OPERATIONS.md [releases-link]: https://github.com/ptr727/Blog/releases +[workflow]: ./WORKFLOW.md diff --git a/TODO.md b/TODO.md index 7810bf5..5959b55 100644 --- a/TODO.md +++ b/TODO.md @@ -8,7 +8,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Piece | State | | --- | --- | -| Content and media | done. 514 pages, 778 media files hash-verified against the export tar | +| Content and media | done. Carries every migrated post and page, with the media hash-verified against the export tar | | URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated | | Deploy shape | done. Proven on two local mirrors and on the VPS, by hand and by pipeline | | CI workflows | green. Validation runs on every pull request and feeds the required check | @@ -29,9 +29,13 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Retest the deploy transport against the real host**, which is [#33][issue-33] and is joint work with whoever holds the server. The transport now pins `StrictHostKeyChecking`, `UserKnownHostsFile`, and `BatchMode`, so it fails closed where it previously failed open, and a stale `DEPLOY_SSH_KNOWN_HOSTS` stops a deploy rather than being tolerated. Staging first, since a broken transport blocks the rollback path as well as the deploy. - **Declare what the VPS keeps.** `hugo.deploy.retention` asks for a retention count declared at the destination, and this repo's deploy credential is write-only by design so the prune belongs to the host. The ownership is recorded in `OPERATIONS.md`, the count is not, and the "ten releases" beside it describes `deploy/make-release.sh` on the local mirrors rather than the containers on the VPS. Confirm the host's timer and its count, then write it next to the ownership line. - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. -- **Deploy production once, to a name that is not the live one.** The production environment is configured and its Pangolin resource is deliberately disabled, so nothing has ever run against it. Validate there before the record moves. +- **Deploy production once, to a name that is not the live one.** The host side is done and the interim name is live: `blog.insanegenius.net` answers `200` unauthenticated, on a Let's Encrypt certificate issued 2026-08-07, serving `0000-placeholder`. The VPS agent calls this M7a, and M7b is the `.com` cutover. Three items belong to this repo before the first production deploy, all detailed in the VPS agent's §19 and §20. That file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": + - **Set `HUGO_BASEURL` to `https://blog.insanegenius.net` on the `production` environment**, and back to `.com` at M7b. Hugo bakes `baseURL` into the canonical tags, the feed, and `sitemap.xml`, so a deploy without this edit serves a site whose every absolute URL points at the live WordPress site. It works mechanically and it is wrong. + - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. + - **Answer the two questions in §19.3**: what `HUGO_BASEURL` holds on `production`, and whether anything else in the build or the checks hardcodes `blog.insanegenius.com`. +- **Decide what `robots.txt` says before the first production deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. -- **Configure log retention before the cutover, or the review below reads whatever happened to survive.** The containers use Docker's `json-file` driver with no rotation and no size limit, so the log grows without bound and is discarded when the container is recreated. It belongs to the host rather than to this repo, like release pruning. +- **"Backed up" describes a design rather than a fact, for the log.** The VPS holds the only copy of the access log until the pull to the backup host runs, and that log rotates and ages out on a schedule the host owns, so the window is the deadline. The pull is two rsync lines the maintainer adds on the backup host, and the log stays deliberately outside the nightly encrypted archives because those are full copies with no dedupe. State that plainly in anything built on top of the log until the pull runs. - Watch server logs for non-200s daily for the first week, then monthly, because real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site. diff --git a/checks/README.md b/checks/README.md index 38a9b56..38da17f 100644 --- a/checks/README.md +++ b/checks/README.md @@ -17,6 +17,8 @@ The contract is enforced by two gates, because one cannot cover both halves: **`golden-urls.txt`, 328 URLs Hugo must render.** Missing any one is a hard CI failure. +**Every count on this page describes the two lists, not the site.** The lists are the legacy contract, closed by the migration, so a new post adds a URL the parity gate reports as `additional URLs built (not a failure)` and changes nothing here. A count moves only when a log review finds a legacy address the crawl missed, which is a deliberate append. + | Shape | Count | Note | | --- | --- | --- | | Tag archives | 180 | The site serves `/tag/`, and **Hugo defaults to `/tags/`** | @@ -28,7 +30,7 @@ The contract is enforced by two gates, because one cannot cover both halves: | Pages | 2 | `/about/`, `/viljoen-family/` | | Home | 1 | | -The 180 tag archives are exactly the tags carried by a published post. Three further terms answer with an empty page and are redirects rather than renders: `brultech` and `phyn`, which no published post uses, and `review`, an empty tag that is also a 12-post category. Hugo generates a term page only where posts exist, which is why the render list is 180 tags rather than 183. +The 180 tag archives are exactly the tags the migrated posts carry. Three further terms answer with an empty page and are redirects rather than renders: `brultech` and `phyn`, which no published post uses, and `review`, an empty tag that is also a 12-post category. Hugo generates a term page only where posts exist, which is why the render list is 180 tags rather than 183. **`redirect-urls.txt`, 917 URLs that must resolve but need not render.** These have no Hugo equivalent. Reproducing them would be absurd, and 404ing them discards real inbound links. @@ -39,7 +41,7 @@ The 180 tag archives are exactly the tags carried by a published post. Three fur | `?p=` shortlinks | 110 | Redirect to the permalink, via `p-ids.map` | | Attachment pages, root level | 107 | Redirect to the parent post, via `slugs.map` | | Per-post comment feeds | 107 | Redirect to the parent post | -| Date archives | 83 | Redirect to `/all/`, since **Hugo has no built-in year or month archive** | +| Date archives | 83 | Redirect to `/all/`, since **Hugo has no built-in year or month archive**. The matcher accepts any date, including dates absent from the list | | Blogger permalinks | 59 | Redirect to the current post, via `blogger.map` | | Blogger monthly archives | 21 | Redirect to `/all/` | | Author archive and pagination | 12 | Redirect to `/`, a single-author blog duplicating home | @@ -62,7 +64,7 @@ Two properties of the maps are non-obvious and easy to break when regenerating t **Adding a URL.** Real traffic finds what the lists missed. When a server log shows a 404 for an address that should work, append it to the appropriate list and add a redirect rule or map entry to cover it. The lists are append-only, per Directionality below. -**Regenerating the maps.** `build-redirects.py` rebuilds everything under `deploy/maps/` from the source export, which lives in a capture directory outside this repo and is passed as an argument. It is a provenance tool rather than a CI step, and it selects the export **by content**, failing unless exactly one candidate contains published posts. The capture holds both a full export and a media-only one with zero posts, and taking the wrong one yields empty maps that are indistinguishable from working ones until the redirects are live. +**Regenerating the maps.** `build-redirects.py` rebuilds everything under `deploy/maps/` from the source export, which lives in a capture directory outside this repo and is passed as an argument. **That directory's path is `CAPTURE_ROOT` in `secrets/.env`**, and `OPERATIONS.md` "Rebuilding from the Exports" records what it holds and which parts of it a person can fetch again. It is a provenance tool rather than a CI step, and it selects the export **by content**, failing unless exactly one candidate contains published posts. The capture holds both a full export and a media-only one with zero posts, and taking the wrong one yields empty maps that are indistinguishable from working ones until the redirects are live. **Checking a count.** Every count above is derivable from the files, so check rather than trust: @@ -88,7 +90,7 @@ The count is not a backlog. It opened at 120 and was adjudicated against the cap | | | | --- | --- | -| 17 | conversion losses, restored — five `gallery` shortcodes emitted empty | +| 17 | conversion losses, restored. The conversion emitted five `gallery` shortcodes empty | | 5 | never orphans, referenced only by an absolute URL the check could not read | | 97 | uploaded to the old platform's media library and never placed on a published page | | 1 | that platform's site icon, superseded by the favicon set at the static root | diff --git a/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md b/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md index cec4a88..e915a59 100644 --- a/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md +++ b/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md @@ -57,6 +57,26 @@ I ended up splitting the contract in two: **328 URLs that Hugo must render**, an The order matters more than the tools. Everything up to step 4 is reversible and costs nothing. Do all of it before you convert anything. +### 0. Decide where the inputs live, before you fetch any of them + +Every step below produces a file you will still be reading months later. The crawl is the input to every check. The export is the input to every redirect map. The media tar is the only correct copy of your images. Put all of it in one directory, outside the site repository, and keep it there. + +Outside the repository, because none of it belongs in a published history: it carries commenter email addresses, IP addresses, and a complete copy of a site you are about to take down. In one directory, because the tooling that derives your URL contract has to be pointed at it, and a derivation you cannot re-run is a result nobody can check. + +The shape that worked here: + +```text +capture/ + export/raw/ the content export, as downloaded + export/media-tar/ the media export, unpacked + mirror/ the crawl, as the old site served it + inventory/ the URL and media lists derived from that crawl +``` + +Then record where that directory is, in the repository, alongside whatever other machine-specific values you keep out of git. That sounds too obvious to write down, which is exactly why it does not get written down: the one thing you cannot re-derive from the repository is the location of the thing everything was derived from. + +Two of those four are re-fetchable from WordPress for as long as the account exists, and those two are the only things a person has to go and download. The crawl and the inventories are not re-fetchable at all, and they stop being possible the day the old hosting ends. + ### 1. Capture the URL surface while the old site is still up Do this **first**. Once you cancel the old hosting you can never reconstruct this, and it is the input to every check later. diff --git a/deploy/README.md b/deploy/README.md index 4d7759f..0c34f69 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -109,7 +109,7 @@ open the auth gate: Set both or neither; half a pair is rejected as the typo it is. They go to curl through a mode-`600` config file rather than as `-H` arguments, which keeps the credential out of the -`ps` output of 1,245 requests, and is also the only form that survives the `export -f` the +`ps` output of one request per URL in the contract, and is also the only form that survives the `export -f` the parallel checks run under. The token is sent to the base URL's own origin and to nothing else, so a redirect that one day points off-site cannot carry it away. @@ -203,7 +203,7 @@ So the default is the value that is harmless on production, and `noindex` is rea asking for it explicitly. `checks/check-live-urls.sh` asserts this when `EXPECT_SITE_ENV` is set, before it checks the -1,245 URLs, since checking the contract against the wrong environment proves nothing. +contract, since checking the contract against the wrong environment proves nothing. ## Trusting the proxy @@ -259,6 +259,8 @@ Directives and URL classes are not one to one, in both directions. `@mapped` is Each row below is a **URL class**, named by the matcher that serves it, so the table can be checked against [`Caddyfile`](./Caddyfile) by grep rather than by trust. +Every class below is a legacy shape, closed by the migration, so no count here moves when content is added. New content is served by the render half of the contract, never by these. + | Matcher | Class size | Shape | | --- | --- | --- | | `@post_child` | 216 | `/YYYY/MM/DD/post//` -> the post, attachment pages | @@ -266,7 +268,7 @@ Each row below is a **URL class**, named by the matcher that serves it, so the t | `@post_id` | 110 | `/?p=` -> the permalink, via `p-ids.map` | | `@post_child_feed` | 107 | `/YYYY/MM/DD/post//feed/` -> the post, ordered **before** `@post_child` | | `@mapped` via `slugs.map` | 107 | bare `//` -> best destination | -| `@date_archive` | 83 | `/YYYY/`, `/YYYY/MM/`, and their pagination -> `/all/` | +| `@date_archive` | 83 | `/YYYY/`, `/YYYY/MM/`, and their pagination -> `/all/`. The matcher accepts any date, including dates absent from the list | | `@mapped` via `blogger.map` | 59 | `/YYYY/MM/slug.html` -> the current post | | `@blogger_archive` | 21 | `/YYYY_MM_01_archive.html` -> `/all/`, any date, including ones never covered | | `@author` | 12 | `/author//`, its pagination and feed -> `/` | @@ -275,9 +277,9 @@ Each row below is a **URL class**, named by the matcher that serves it, so the t | `@blogger_feed` | 2 | `/feeds/posts/default` -> `/feed.xml`, Blogger's Atom feed | | `@blogger_page` | 2 | `/p/.html` -> `//`, Blogger's static-page shape | -**Those thirteen classes sum to 917**, which is the line count of [`checks/redirect-urls.txt`](../checks/redirect-urls.txt) and the whole redirect contract. +**Those thirteen classes account for every line in [`checks/redirect-urls.txt`](../checks/redirect-urls.txt), with nothing in the contract outside the table.** Completeness is the property worth holding, and the list's line count is how to check it. -`@uploads` is deliberately absent from that table and from the 917. It rewrites `/wp-content/uploads/(.*)` to `/media/$1`, preserving all 778 legacy image URLs, which are gated by `golden-media-legacy.txt` on their own. Counting them here would double-count a set that has its own list. +`@uploads` is deliberately absent from that table and from the redirect contract. It rewrites `/wp-content/uploads/(.*)` to `/media/$1`, preserving all 778 legacy image URLs, which are gated by `golden-media-legacy.txt` on their own. Counting them here would double-count a set that has its own list. `@label` is the fourteenth class and is deliberately **not** in the contract. `/search/label/