Skip to content

Document the publishing workflow and the migration record#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ptr727 merged 5 commits into from
Aug 8, 2026
Merged

Document the publishing workflow and the migration record #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
def8c70
Document the publishing workflow and the migration record
ptr727 Aug 8, 2026
8c6a676
Clarify three phrasings flagged in review
ptr727 Aug 8, 2026
e5f8f27
Point the VPS agent references at how to obtain the file
ptr727 Aug 8, 2026
9780a3c
Reword three table cells flagged in review
ptr727 Aug 8, 2026
cd9d176
Correct what the gates check the contract against
ptr727 Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 44 additions & 13 deletions OPERATIONS.md
View file
Open in desktop

Large diffs are not rendered by default.

95 changes: 95 additions & 0 deletions README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,8 @@ See [Release History][history] for complete release notes and older versions.
## Table of Contents <!-- omit from toc -->

- [Use Cases](#use-cases)
- [Migration from WordPress](#migration-from-wordpress)
- [How a Change Reaches the Site](#how-a-change-reaches-the-site)
- [Configuration](#configuration)
- [Questions or Issues](#questions-or-issues)
- [Development Environment Setup](#development-environment-setup)
Expand All@@ -52,6 +54,97 @@ The site answers far more addresses than it renders pages, because it has served

Deployment is a release directory plus a symlink. A build is installed alongside its predecessors, verified, and made live by swapping one link, so a rollback is the same swap in reverse. See [OPERATIONS.md][operations].

## Migration from WordPress

The site has served the same domain since 2008, across three platforms: Blogger until 2012, WordPress until 2026, and Hugo from then on. Converting the posts took an afternoon. Preserving sixteen years of inbound links was the work, and it is why this repository carries a URL contract and gates it rather than trusting the build.

The account of that migration is a post on the site, [Moving This Blog From WordPress to Hugo][migration-post]. It covers what a WordPress export holds and what it leaves out, why the sitemap named barely a tenth of the addresses the site was actually serving, how the Blogger-era permalinks resolve through a lookup table rather than a pattern, why media fetched over HTTP is not the same bytes as the media in the export and only a content hash tells them apart, and which Hugo default moves every taxonomy archive to a new address without reporting anything.

## How a Change Reaches the Site

A change starts on a branch in this repository and ends as bytes on a VPS. Every stage carries a gate, and three of them add a human read, because the failure this repository exists to catch is an address that stops answering, and no check that skips a running server can see one.

Both deploys are manual workflow dispatches, so a merge publishes nothing and each environment goes live by a deliberate run.

[`WORKFLOW.md`][workflow] states the same CI machinery as a contract for tooling, and [`OPERATIONS.md`][operations] holds the procedure to run at a keyboard.

```mermaid
flowchart LR
subgraph local["1. On a branch"]
direction TB
w["New branch, write the post"] --> b["Build, warnings are fatal"]
b --> p["Gate: every page that must render, renders"]
p --> m["Release onto the local mirror"]
m --> l["Gate: every URL in the contract, against a running server"]
l --> e1["Human: read the page"]
end

subgraph gh["2. On GitHub"]
direction TB
pr["Pull request into develop"] --> ci["CI: lint, spelling, build, URL parity"]
ci --> rv["Automated review, then a human squash merge"]
end

subgraph stg["3. Staging, behind the auth gate"]
direction TB
ds["Dispatch: deploy staging"] --> ls["Gate: every URL in the contract, run by CI"]
ls --> e2["Human: read staging"]
end

subgraph prd["4. Production, public"]
direction TB
pm["Pull request: develop into main"] --> dp["Dispatch: deploy production"]
dp --> lp["Gate: every URL in the contract, run by CI"]
lp --> e3["Human: read production"]
end

e1 --> pr
rv --> ds
e2 --> pm
```

**1. Write on a branch, and prove the artifact locally.** A post is a markdown file under `content/posts/`, written on a feature branch with whatever editor the author prefers. The build treats a warning as fatal, so a deprecated theme API fails it rather than accumulating. The build gate then checks the render half of the contract, which is every address that must return a page.

Most of the contract is not pages, though. It is redirects, and a redirect is the web server's job, so no build reaches them. The release installs onto a mirror on the maintainer's own network, which runs the same Caddy container and the same bundle as the server behind the same Traefik front end. The live gate follows every URL in the contract against that mirror and checks each redirect's destination rather than its status code. A human then reads the page, because no gate has an opinion about the writing.

**2. Open a pull request into `develop`.** CI runs lint, spelling, workflow and config validation, the build, and the render gate on a clean checkout. It does not prove the redirects, which is why the local run is a prerequisite rather than a convenience: a change to the Caddy config or to a redirect map goes green in CI while the redirect it broke stays broken. An automated review runs against the branch, and a human squash-merges it.

**3. Deploy to staging.** A dispatched workflow builds the commit, uploads the release, flips the symlink, and runs the same live gate from CI against the running site. The local mirror proves the artifact, and staging proves the infrastructure that exists only on the server: routing, TLS, the deploy key, and its confined transport. Staging keeps its authentication gate on and the check presents a token, so a byte-identical copy of the public site is never exposed to a crawler. A human then reads it.

**4. Promote, and deploy production.** `develop` merges into `main` through a pull request, and production is a second dispatch that accepts `main` alone. The same gate runs a third time, against the public address and with no token, and a human reads the result.

Each branch feeds one environment:

```mermaid
flowchart LR
feat["feature branch"] -->|squash| dev["develop"]
dev -->|merge commit| main["main"]
dev -.->|manual dispatch| stg["staging site"]
main -.->|manual dispatch| prod["production site"]
```

### What Happens While the Site Runs <!-- omit from toc -->

Publishing is half of it. The other half runs on its own cadence, because the contract proves only the addresses someone wrote down.

```mermaid
flowchart LR
site["Production, serving"] --> log["Edge access log, one line per request"]
log --> outward["Outward: asked for, and not here"]
log --> inward["Inward: here, and never asked for"]
outward --> add["Add the address to the contract, add a redirect"]
add --> next["Ships with the next change, top of the pipeline"]
inward --> judge["Decide whether unread content is preservation or clutter"]
site --> cfg["Host configuration archived off the VPS"]
cfg --> rebuild["A rebuild restores the config and redeploys"]
```

**Real traffic is the only source that finds what the contract misses.** Every request to the host is recorded at the edge, one line per request, and the review runs in two directions. The outward pass reads non-200 responses. A 404 on a path shaped like real content means an old link nobody recorded, and the fix is to add the address to the contract and add a redirect, which then ships through the pipeline above like any other change. Scanners probing for `wp-login.php` dominate the raw count and are filtered by shape rather than investigated. The inward pass subtracts every address that has ever answered 200 from the set the site builds, which names content no reader has reached. That evidence accumulates slowly, and it is the only thing that settles whether media the old platform never published is worth carrying.

**Backups protect the server rather than the site.** The site is reproducible from this repository by running a deploy, so what is worth keeping is the host's configuration: the container definitions, the proxy configuration, and the deploy account with its restricted key. A bare-metal restore rebuilds the host, restores that configuration, and deploys again. A procedure that backs up the deploy root protects a copy of something git already holds.

[OPERATIONS.md][operations] holds the detail: the commands, the environments, the rollback, and the three log tiers with what each cannot see.

## Configuration

| Path | Holds |
Expand DownExpand Up@@ -126,8 +219,10 @@ Licensed under the [MIT License][license]\
[history]: ./HISTORY.md
[hugo-config]: ./hugo.yaml
[license]: ./LICENSE
[migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md
[operations]: ./OPERATIONS.md
[releases-link]: https://github.com/ptr727/Blog/releases
[workflow]: ./WORKFLOW.md

<!-- External -->

Expand Down
10 changes: 7 additions & 3 deletions TODO.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y

| Piece | State |
| --- | --- |
| Content and media | done. 514 pages, 778 media files hash-verified against the export tar |
| Content and media | done. Carries every migrated post and page, with the media hash-verified against the export tar |
| URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated |
| Deploy shape | done. Proven on two local mirrors and on the VPS, by hand and by pipeline |
| CI workflows | green. Validation runs on every pull request and feeds the required check |
Expand All@@ -29,9 +29,13 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y
- **Retest the deploy transport against the real host**, which is [#33][issue-33] and is joint work with whoever holds the server. The transport now pins `StrictHostKeyChecking`, `UserKnownHostsFile`, and `BatchMode`, so it fails closed where it previously failed open, and a stale `DEPLOY_SSH_KNOWN_HOSTS` stops a deploy rather than being tolerated. Staging first, since a broken transport blocks the rollback path as well as the deploy.
- **Declare what the VPS keeps.** `hugo.deploy.retention` asks for a retention count declared at the destination, and this repo's deploy credential is write-only by design so the prune belongs to the host. The ownership is recorded in `OPERATIONS.md`, the count is not, and the "ten releases" beside it describes `deploy/make-release.sh` on the local mirrors rather than the containers on the VPS. Confirm the host's timer and its count, then write it next to the ownership line.
- **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way.
- **Deploy production once, to a name that is not the live one.** The production environment is configured and its Pangolin resource is deliberately disabled, so nothing has ever run against it. Validate there before the record moves.
- **Deploy production once, to a name that is not the live one.** The host side is done and the interim name is live: `blog.insanegenius.net` answers `200` unauthenticated, on a Let's Encrypt certificate issued 2026-08-07, serving `0000-placeholder`. The VPS agent calls this M7a, and M7b is the `.com` cutover. Three items belong to this repo before the first production deploy, all detailed in the VPS agent's §19 and §20. That file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides":
- **Set `HUGO_BASEURL` to `https://blog.insanegenius.net` on the `production` environment**, and back to `.com` at M7b. Hugo bakes `baseURL` into the canonical tags, the feed, and `sitemap.xml`, so a deploy without this edit serves a site whose every absolute URL points at the live WordPress site. It works mechanically and it is wrong.
- **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment.
- **Answer the two questions in §19.3**: what `HUGO_BASEURL` holds on `production`, and whether anything else in the build or the checks hardcodes `blog.insanegenius.com`.
- **Decide what `robots.txt` says before the first production deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links.
- Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied.
- **Configure log retention before the cutover, or the review below reads whatever happened to survive.** The containers use Docker's `json-file` driver with no rotation and no size limit, so the log grows without bound and is discarded when the container is recreated. It belongs to the host rather than to this repo, like release pruning.
- **"Backed up" describes a design rather than a fact, for the log.** The VPS holds the only copy of the access log until the pull to the backup host runs, and that log rotates and ages out on a schedule the host owns, so the window is the deadline. The pull is two rsync lines the maintainer adds on the backup host, and the log stays deliberately outside the nightly encrypted archives because those are full copies with no dedupe. State that plainly in anything built on top of the log until the pull runs.
- Watch server logs for non-200s daily for the first week, then monthly, because real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review".
- Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages.
- Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site.
Expand Down
10 changes: 6 additions & 4 deletions checks/README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -17,6 +17,8 @@ The contract is enforced by two gates, because one cannot cover both halves:

**`golden-urls.txt`, 328 URLs Hugo must render.** Missing any one is a hard CI failure.

**Every count on this page describes the two lists, not the site.** The lists are the legacy contract, closed by the migration, so a new post adds a URL the parity gate reports as `additional URLs built (not a failure)` and changes nothing here. A count moves only when a log review finds a legacy address the crawl missed, which is a deliberate append.

| Shape | Count | Note |
| --- | --- | --- |
| Tag archives | 180 | The site serves `/tag/`, and **Hugo defaults to `/tags/`** |
Expand All@@ -28,7 +30,7 @@ The contract is enforced by two gates, because one cannot cover both halves:
| Pages | 2 | `/about/`, `/viljoen-family/` |
| Home | 1 | |

The 180 tag archives are exactly the tags carried by a published post. Three further terms answer with an empty page and are redirects rather than renders: `brultech` and `phyn`, which no published post uses, and `review`, an empty tag that is also a 12-post category. Hugo generates a term page only where posts exist, which is why the render list is 180 tags rather than 183.
The 180 tag archives are exactly the tags the migrated posts carry. Three further terms answer with an empty page and are redirects rather than renders: `brultech` and `phyn`, which no published post uses, and `review`, an empty tag that is also a 12-post category. Hugo generates a term page only where posts exist, which is why the render list is 180 tags rather than 183.

**`redirect-urls.txt`, 917 URLs that must resolve but need not render.** These have no Hugo equivalent. Reproducing them would be absurd, and 404ing them discards real inbound links.

Expand All@@ -39,7 +41,7 @@ The 180 tag archives are exactly the tags carried by a published post. Three fur
| `?p=<id>` shortlinks | 110 | Redirect to the permalink, via `p-ids.map` |
| Attachment pages, root level | 107 | Redirect to the parent post, via `slugs.map` |
| Per-post comment feeds | 107 | Redirect to the parent post |
| Date archives | 83 | Redirect to `/all/`, since **Hugo has no built-in year or month archive** |
| Date archives | 83 | Redirect to `/all/`, since **Hugo has no built-in year or month archive**. The matcher accepts any date, including dates absent from the list |
| Blogger permalinks | 59 | Redirect to the current post, via `blogger.map` |
| Blogger monthly archives | 21 | Redirect to `/all/` |
| Author archive and pagination | 12 | Redirect to `/`, a single-author blog duplicating home |
Expand All@@ -62,7 +64,7 @@ Two properties of the maps are non-obvious and easy to break when regenerating t

**Adding a URL.** Real traffic finds what the lists missed. When a server log shows a 404 for an address that should work, append it to the appropriate list and add a redirect rule or map entry to cover it. The lists are append-only, per Directionality below.

**Regenerating the maps.** `build-redirects.py` rebuilds everything under `deploy/maps/` from the source export, which lives in a capture directory outside this repo and is passed as an argument. It is a provenance tool rather than a CI step, and it selects the export **by content**, failing unless exactly one candidate contains published posts. The capture holds both a full export and a media-only one with zero posts, and taking the wrong one yields empty maps that are indistinguishable from working ones until the redirects are live.
**Regenerating the maps.** `build-redirects.py` rebuilds everything under `deploy/maps/` from the source export, which lives in a capture directory outside this repo and is passed as an argument. **That directory's path is `CAPTURE_ROOT` in `secrets/.env`**, and `OPERATIONS.md` "Rebuilding from the Exports" records what it holds and which parts of it a person can fetch again. It is a provenance tool rather than a CI step, and it selects the export **by content**, failing unless exactly one candidate contains published posts. The capture holds both a full export and a media-only one with zero posts, and taking the wrong one yields empty maps that are indistinguishable from working ones until the redirects are live.

**Checking a count.** Every count above is derivable from the files, so check rather than trust:

Expand All@@ -88,7 +90,7 @@ The count is not a backlog. It opened at 120 and was adjudicated against the cap

| | |
| --- | --- |
| 17 | conversion losses, restored — five `gallery` shortcodes emitted empty |
| 17 | conversion losses, restored. The conversion emitted five `gallery` shortcodes empty |
| 5 | never orphans, referenced only by an absolute URL the check could not read |
| 97 | uploaded to the old platform's media library and never placed on a published page |
| 1 | that platform's site icon, superseded by the favicon set at the static root |
Expand Down
Loading