From 24ad5688df185e66a94665f3cc649efa67c90569 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 12:10:11 -0700 Subject: [PATCH 1/5] Fetch media in the live check, so a lost image is caught somewhere The build gate proves the media set against files on disk. Nothing proved those files reached the server or that the server can read them: the live check requested pages and redirects and never one image, so a media tree lost between a passing build and the server was caught by neither gate. On a site whose value is eighteen years of posts with images in them, that was the gap worth closing. checks/golden-media-live.txt is a handful rather than exhaustive, because the set is already proven and this is a delivery check. Its entries cover both trees, since they arrive by different routes and a partial transfer is unlikely to land evenly, plus the legacy /wp-content/uploads/ form, which nothing else exercised against a running server. Three assertions, each for a different loss. A missing file answers 404, a file whose mode went wrong answers 403, a file truncated to nothing still answers 200 so the byte count is checked, and a server answering an error page for a missing asset answers 200 as text/html so the content type is checked too. The 403 case is why this is not theoretical: a hard-linked file carries its inode's mode, so one that acquires a bad mode rides the chain into every later release, present and correctly named and unreadable, which a build-time is_file() cannot see. check_media follows one hop by hand rather than passing -L to curl, because -L would carry the auth-gate credential to wherever the rule points. Same origin boundary as check_redirect, for the same reason. Verified against production: PASS at 1253, and each failure shape reproduced rather than assumed. A missing image, a legacy URL redirecting to a missing image, a 200 that is not an image, and a zero-byte 200 all fail with the list entry named. The legacy entries deliberately went in this list rather than redirect-urls.txt. Adding them there would move a count stated in six documents and in the published migration post, which is disproportionate to three test URLs. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 5 +-- checks/README.md | 16 +++++++++ checks/check-live-urls.sh | 65 +++++++++++++++++++++++++++++++++--- checks/golden-media-live.txt | 8 +++++ 4 files changed, 87 insertions(+), 7 deletions(-) create mode 100644 checks/golden-media-live.txt diff --git a/TODO.md b/TODO.md index 29ba0f6..ef0762e 100644 --- a/TODO.md +++ b/TODO.md @@ -39,8 +39,9 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. -- **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. +- **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. +- **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. +- **Restore file mtimes in CI so `--link-dest` links. The media check it waited on now exists, so this is unblocked.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **Publish a release from `main`, once the pipeline has soaked.** `1.0.11` is the newest release from `main` and was cut on 2026-08-01, ahead of every deploy change, so the next one is the first that would describe a site actually serving its public address. The mechanism is proven and is not what this waits on: it waits on the switchover being trusted rather than merely green, which is what the log review under **Recurring operations** establishes and no gate can. A release cut before that names a state that has not held yet. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. diff --git a/checks/README.md b/checks/README.md index 89d866a..a8ecaf0 100644 --- a/checks/README.md +++ b/checks/README.md @@ -72,6 +72,22 @@ Two properties of the maps are non-obvious and easy to break when regenerating t wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy.txt ``` +## `golden-media-live.txt`, and why the media set needs a second list + +`golden-media-legacy.txt` proves the **set**, at build time, against files on disk. That cannot prove the files reached the server or that the server can read them, and the live check requested pages and redirects and never one image, so a media tree lost between a passing build and the server was caught by neither gate. `golden-media-live.txt` closes that, and it is fetched by `check-live-urls.sh` against a running server. + +**It is a handful rather than exhaustive, deliberately.** The set is already proven, so this is a delivery check, and its entries are chosen to cover both trees and a spread of years because the trees arrive by different routes and a partial transfer is unlikely to land evenly. + +| Entry shape | Proves | +| --- | --- | +| `/media/` paths | the imported uploads tree arrives and is served | +| `/external/` paths | the tree of media localized from other hosts arrives too | +| `/wp-content/uploads/` paths | the `@uploads` rule still lands on the image, which nothing else exercises against a running server | + +**Three assertions rather than one, because each catches a different loss.** A file missing from the transfer answers 404. A file whose mode went wrong answers 403, which is the case this exists for. A file truncated to nothing still answers 200, so the byte count is asserted. And a server that answers an error page for a missing asset answers 200 with `text/html`, so the content type is asserted as well. The check follows one redirect by hand rather than passing `-L` to curl, for the reason `check_redirect` does: `-L` would carry the auth-gate credential to wherever the rule points. + +**The mode case is why this is not theoretical.** A hard-linked file carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server. A build-time `is_file()` on the runner cannot see it, and neither can a check that never requests an image. + ## Directionality The parity check fails on a **missing** URL and only notes an **extra** one. New posts, new tags, and deeper pagination legitimately add URLs, and nothing legitimately removes a URL the site has served. That asymmetry is what makes the lists append-only, which in turn is what makes the length-floor assertion in `check-live-urls.sh` sound: without it a truncated list would make every assertion below it pass vacuously. diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index a5de29a..a4de282 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -15,8 +15,8 @@ CHECKS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PARALLEL="${PARALLEL:-16}" # A truncated list otherwise turns this into a gate that passes while checking almost nothing. -declare -A FLOOR=(["golden-urls.txt"]=320 ["redirect-urls.txt"]=900) -for list in golden-urls.txt redirect-urls.txt; do +declare -A FLOOR=(["golden-urls.txt"]=320 ["redirect-urls.txt"]=900 ["golden-media-live.txt"]=8) +for list in golden-urls.txt redirect-urls.txt golden-media-live.txt; do n=$(grep -c . "$CHECKS/$list") if [ "$n" -lt "${FLOOR[$list]}" ]; then echo "FAIL $list: $n URLs, expected at least ${FLOOR[$list]} - the list has been truncated" >&2 @@ -57,6 +57,57 @@ check_render() { [ "$code" = "200" ] || echo "render $url expected 200, got $code" >>"$FAILED" } +# Invoked indirectly, the same way as check_render above. +# shellcheck disable=SC2329 +# The build gate proves the media SET against files on disk. It cannot prove the files +# reached the server or that the server can read them, and until this ran the live check +# requested pages and redirects and never an image. +# +# Status alone is most of the value: a file lost in transfer answers 404, and one whose +# mode went wrong answers 403. The byte count catches the remaining case, a file that +# arrived truncated to nothing, which still answers 200. Content type is asserted because a +# server misconfigured into serving an error page for a missing asset answers 200 as well. +check_media() { + local url="$1" code len type target auth=() target_auth=() + [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + target="$BASE$url" + target_auth=("${auth[@]}") + # One hop is followed rather than passed to curl -L, because -L would carry the + # credential to wherever the rule points. The legacy /wp-content/uploads/ entries reach + # the image through the @uploads rule, and what this proves is that the image arrives, + # not that the hop happened. + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$target") + case "$code" in + 301 | 308) + target=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "${auth[@]}" "$target") + # Same origin boundary as check_redirect, and for the same reason: a rule that one + # day points off-site must not mail the token there. A bare prefix would also accept + # a lookalike host registered as an attacker's subdomain. + target_auth=() + if [ -n "$CURLRC" ]; then + case "$target" in + "$BASE" | "$BASE"/*) target_auth=(-K "$CURLRC") ;; + esac + fi + ;; + esac + read -r code len type < <(curl -s -o /dev/null \ + -w '%{http_code} %{size_download} %{content_type}\n' \ + --max-time 30 "${target_auth[@]}" "$target") + if [ "$code" != "200" ]; then + echo "media $url expected 200, got $code" >>"$FAILED" + return + fi + if [ "${len:-0}" -eq 0 ]; then + echo "media $url answered 200 with an empty body" >>"$FAILED" + return + fi + case "$type" in + image/*) ;; + *) echo "media $url answered 200 as $type, expected an image" >>"$FAILED" ;; + esac +} + # Invoked indirectly, the same way as check_render above. # shellcheck disable=SC2329 check_redirect() { @@ -89,7 +140,7 @@ check_redirect() { esac } -export -f check_render check_redirect +export -f check_render check_redirect check_media export BASE FAILED CURLRC echo "==> $BASE" @@ -192,16 +243,20 @@ n_redirect=$(grep -c . "$CHECKS/redirect-urls.txt") echo "==> checking $n_redirect URLs that must redirect" grep . "$CHECKS/redirect-urls.txt" | xargs -P "$PARALLEL" -I{} bash -c 'check_redirect "$@"' _ {} +n_media=$(grep -c . "$CHECKS/golden-media-live.txt") +echo "==> checking $n_media media URLs that must be served as images" +grep . "$CHECKS/golden-media-live.txt" | xargs -P "$PARALLEL" -I{} bash -c 'check_media "$@"' _ {} + # A count of zero exits non-zero, so a fallback that echoes would append a second zero. # Swallowing only the exit status keeps the printed count usable. failures=$(grep -c . "$FAILED" 2>/dev/null || true) if [ "$failures" -eq 0 ]; then - echo "PASS - $((n_render + n_redirect)) URLs honored" + echo "PASS - $((n_render + n_redirect + n_media)) URLs honored" exit 0 fi echo -echo "FAIL - $failures of $((n_render + n_redirect)) URLs" +echo "FAIL - $failures of $((n_render + n_redirect + n_media)) URLs" sort "$FAILED" | head -40 [ "$failures" -gt 40 ] && echo "... and $((failures - 40)) more" exit 1 diff --git a/checks/golden-media-live.txt b/checks/golden-media-live.txt new file mode 100644 index 0000000..dca31f0 --- /dev/null +++ b/checks/golden-media-live.txt @@ -0,0 +1,8 @@ +/media/2010/05/2010-05-1818-13-351.jpg +/media/2016/01/e3-log.jpg +/media/2023/03/image-12.png +/external/7c7d2c8ca28c936b.png +/external/68d24de592ffcefb.png +/wp-content/uploads/2010/05/2010-05-1815-32-351.jpg +/wp-content/uploads/2016/01/e3-log.jpg +/wp-content/uploads/2023/03/image-12.png From 1b9fde70e0097471847e427ae3c7bc16b742fb3a Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 12:17:28 -0700 Subject: [PATCH 2/5] Capture curl's exit status in check_media, and name a transport error `read ... < <(curl ...)` discards curl's exit status. Measured before acting on it: the check still fails closed, because curl writes 000 for http_code on a refused connection, a DNS failure and a timeout alike, so none of them can pass. What was lost was the diagnosis, since a bare "expected 200, got 000" leaves a reader to infer that no request completed at all. Command substitution preserves the status, so both are reported now: "no HTTP response: curl exit 7, transport error or timeout". The two older functions in this file use the same process-substitution shape and are left alone here, since they fail closed for the same reason and changing them is a separate change with its own testing. Raised by Copilot on PR #64. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-live-urls.sh | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index a4de282..8df64d9 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -91,9 +91,20 @@ check_media() { fi ;; esac - read -r code len type < <(curl -s -o /dev/null \ + # Command substitution rather than `read < <(...)`, because process substitution discards + # curl's exit status. It still fails closed either way, since curl writes 000 for + # http_code on a transport error, measured against a refused connection, a DNS failure + # and a timeout. What the status buys is a message that says which of the two happened, + # rather than leaving a reader to infer it from a bare 000. + local out rc=0 + out=$(curl -s -o /dev/null \ -w '%{http_code} %{size_download} %{content_type}\n' \ - --max-time 30 "${target_auth[@]}" "$target") + --max-time 30 "${target_auth[@]}" "$target") || rc=$? + read -r code len type <<<"$out" + if [ "$rc" -ne 0 ] || [ "${code:-000}" = "000" ]; then + echo "media $url no HTTP response: curl exit $rc, transport error or timeout" >>"$FAILED" + return + fi if [ "$code" != "200" ]; then echo "media $url expected 200, got $code" >>"$FAILED" return From 66c49b94fc28c73e2083cce389190529a8bb2ec7 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 12:23:22 -0700 Subject: [PATCH 3/5] Note that content_type must stay last in the curl format Declining the parsing change the review proposed, since `read` assigns the whole remainder of the line to its final variable and the type arrives intact, verified against both a literal and a live response. What the finding does surface is a latent edit hazard rather than a current defect: the value survives only because it is last, so a fourth field appended after it would be swallowed into the type. The constraint now sits on the format string, where someone would break it. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-live-urls.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index 8df64d9..b31e227 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -96,6 +96,9 @@ check_media() { # http_code on a transport error, measured against a refused connection, a DNS failure # and a timeout. What the status buys is a message that says which of the two happened, # rather than leaving a reader to infer it from a bare 000. + # content_type stays LAST in this format. `read` assigns the whole remainder of the line + # to its final variable, which is what lets a value containing spaces survive intact; a + # field added after it would be swallowed into the type instead. local out rc=0 out=$(curl -s -o /dev/null \ -w '%{http_code} %{size_download} %{content_type}\n' \ From 3d6eb1475407237dc316bda222f0a8938c738199 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 12:27:31 -0700 Subject: [PATCH 4/5] Include the new list in the count command it documents The README's check-the-counts command names three lists and there are four, so following it silently omits the one this branch adds. A command that is meant to save you from trusting a written number should not itself be a written number that drifted. Verified the command now covers every list the live check reads. Raised by Copilot on PR #64. Co-Authored-By: Claude Opus 5 (1M context) --- checks/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/checks/README.md b/checks/README.md index a8ecaf0..c86ebb0 100644 --- a/checks/README.md +++ b/checks/README.md @@ -69,7 +69,7 @@ Two properties of the maps are non-obvious and easy to break when regenerating t **Checking a count.** Every count above is derivable from the files, so check rather than trust: ```sh -wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy.txt +wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy.txt checks/golden-media-live.txt ``` ## `golden-media-live.txt`, and why the media set needs a second list From 5b92b467e34f9a6bcb1be811b55f9f37bb19042f Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 12:34:44 -0700 Subject: [PATCH 5/5] Stop the truncation guard from being skipped by its own failure Two suppressed findings, both real, and the first is the failure family this repo already documents: a query that matched nothing reading as a clean result. An unreadable list makes `grep -c` yield nothing, and `[ "" -lt N ]` is a syntax error that evaluates false, so the guard against a truncated list was itself skipped. Reproduced: the guard does not fire, the run continues with no URLs, and it exits 0. A guard that fails open is worse than no guard, and this one protected every assertion below it. Readability is asserted and the count is validated as a number before it is compared. Verified that a missing list and a truncated list now both exit 1 with distinct messages, and that a healthy run still passes. Second: a 301 carrying no usable Location left the follow-up target empty, and fetching an empty URL was reported as a transport error, which names the wrong problem. It now says what actually happened. Raised by Copilot on PR #64. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-live-urls.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index b31e227..0bafda1 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -17,7 +17,20 @@ PARALLEL="${PARALLEL:-16}" # A truncated list otherwise turns this into a gate that passes while checking almost nothing. declare -A FLOOR=(["golden-urls.txt"]=320 ["redirect-urls.txt"]=900 ["golden-media-live.txt"]=8) for list in golden-urls.txt redirect-urls.txt golden-media-live.txt; do + # The count is validated before it is compared. An unreadable list makes `grep -c` yield + # nothing, and `[ "" -lt N ]` is a syntax error that evaluates false, so the guard against + # a truncated list would itself be skipped and the run would pass having checked nothing. + [ -r "$CHECKS/$list" ] || { + echo "FAIL $list: not readable at $CHECKS/$list" >&2 + exit 1 + } n=$(grep -c . "$CHECKS/$list") + case "$n" in + '' | *[!0-9]*) + echo "FAIL $list: could not count URLs, got '$n'" >&2 + exit 1 + ;; + esac if [ "$n" -lt "${FLOOR[$list]}" ]; then echo "FAIL $list: $n URLs, expected at least ${FLOOR[$list]} - the list has been truncated" >&2 exit 1 @@ -80,6 +93,12 @@ check_media() { case "$code" in 301 | 308) target=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "${auth[@]}" "$target") + # A 301 carrying no usable Location leaves this empty, and fetching an empty URL would + # be reported below as a transport error, which names the wrong problem. + if [ -z "$target" ]; then + echo "media $url answered $code with no usable Location" >>"$FAILED" + return + fi # Same origin boundary as check_redirect, and for the same reason: a rule that one # day points off-site must not mail the token there. A bare prefix would also accept # a lookalike host registered as an attacker's subdomain.