diff --git a/TODO.md b/TODO.md index 29ba0f6..ef0762e 100644 --- a/TODO.md +++ b/TODO.md @@ -39,8 +39,9 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. -- **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. +- **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. +- **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. +- **Restore file mtimes in CI so `--link-dest` links. The media check it waited on now exists, so this is unblocked.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **Publish a release from `main`, once the pipeline has soaked.** `1.0.11` is the newest release from `main` and was cut on 2026-08-01, ahead of every deploy change, so the next one is the first that would describe a site actually serving its public address. The mechanism is proven and is not what this waits on: it waits on the switchover being trusted rather than merely green, which is what the log review under **Recurring operations** establishes and no gate can. A release cut before that names a state that has not held yet. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. diff --git a/checks/README.md b/checks/README.md index 89d866a..c86ebb0 100644 --- a/checks/README.md +++ b/checks/README.md @@ -69,9 +69,25 @@ Two properties of the maps are non-obvious and easy to break when regenerating t **Checking a count.** Every count above is derivable from the files, so check rather than trust: ```sh -wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy.txt +wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy.txt checks/golden-media-live.txt ``` +## `golden-media-live.txt`, and why the media set needs a second list + +`golden-media-legacy.txt` proves the **set**, at build time, against files on disk. That cannot prove the files reached the server or that the server can read them, and the live check requested pages and redirects and never one image, so a media tree lost between a passing build and the server was caught by neither gate. `golden-media-live.txt` closes that, and it is fetched by `check-live-urls.sh` against a running server. + +**It is a handful rather than exhaustive, deliberately.** The set is already proven, so this is a delivery check, and its entries are chosen to cover both trees and a spread of years because the trees arrive by different routes and a partial transfer is unlikely to land evenly. + +| Entry shape | Proves | +| --- | --- | +| `/media/` paths | the imported uploads tree arrives and is served | +| `/external/` paths | the tree of media localized from other hosts arrives too | +| `/wp-content/uploads/` paths | the `@uploads` rule still lands on the image, which nothing else exercises against a running server | + +**Three assertions rather than one, because each catches a different loss.** A file missing from the transfer answers 404. A file whose mode went wrong answers 403, which is the case this exists for. A file truncated to nothing still answers 200, so the byte count is asserted. And a server that answers an error page for a missing asset answers 200 with `text/html`, so the content type is asserted as well. The check follows one redirect by hand rather than passing `-L` to curl, for the reason `check_redirect` does: `-L` would carry the auth-gate credential to wherever the rule points. + +**The mode case is why this is not theoretical.** A hard-linked file carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server. A build-time `is_file()` on the runner cannot see it, and neither can a check that never requests an image. + ## Directionality The parity check fails on a **missing** URL and only notes an **extra** one. New posts, new tags, and deeper pagination legitimately add URLs, and nothing legitimately removes a URL the site has served. That asymmetry is what makes the lists append-only, which in turn is what makes the length-floor assertion in `check-live-urls.sh` sound: without it a truncated list would make every assertion below it pass vacuously. diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index a5de29a..0bafda1 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -15,9 +15,22 @@ CHECKS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PARALLEL="${PARALLEL:-16}" # A truncated list otherwise turns this into a gate that passes while checking almost nothing. -declare -A FLOOR=(["golden-urls.txt"]=320 ["redirect-urls.txt"]=900) -for list in golden-urls.txt redirect-urls.txt; do +declare -A FLOOR=(["golden-urls.txt"]=320 ["redirect-urls.txt"]=900 ["golden-media-live.txt"]=8) +for list in golden-urls.txt redirect-urls.txt golden-media-live.txt; do + # The count is validated before it is compared. An unreadable list makes `grep -c` yield + # nothing, and `[ "" -lt N ]` is a syntax error that evaluates false, so the guard against + # a truncated list would itself be skipped and the run would pass having checked nothing. + [ -r "$CHECKS/$list" ] || { + echo "FAIL $list: not readable at $CHECKS/$list" >&2 + exit 1 + } n=$(grep -c . "$CHECKS/$list") + case "$n" in + '' | *[!0-9]*) + echo "FAIL $list: could not count URLs, got '$n'" >&2 + exit 1 + ;; + esac if [ "$n" -lt "${FLOOR[$list]}" ]; then echo "FAIL $list: $n URLs, expected at least ${FLOOR[$list]} - the list has been truncated" >&2 exit 1 @@ -57,6 +70,77 @@ check_render() { [ "$code" = "200" ] || echo "render $url expected 200, got $code" >>"$FAILED" } +# Invoked indirectly, the same way as check_render above. +# shellcheck disable=SC2329 +# The build gate proves the media SET against files on disk. It cannot prove the files +# reached the server or that the server can read them, and until this ran the live check +# requested pages and redirects and never an image. +# +# Status alone is most of the value: a file lost in transfer answers 404, and one whose +# mode went wrong answers 403. The byte count catches the remaining case, a file that +# arrived truncated to nothing, which still answers 200. Content type is asserted because a +# server misconfigured into serving an error page for a missing asset answers 200 as well. +check_media() { + local url="$1" code len type target auth=() target_auth=() + [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + target="$BASE$url" + target_auth=("${auth[@]}") + # One hop is followed rather than passed to curl -L, because -L would carry the + # credential to wherever the rule points. The legacy /wp-content/uploads/ entries reach + # the image through the @uploads rule, and what this proves is that the image arrives, + # not that the hop happened. + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$target") + case "$code" in + 301 | 308) + target=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "${auth[@]}" "$target") + # A 301 carrying no usable Location leaves this empty, and fetching an empty URL would + # be reported below as a transport error, which names the wrong problem. + if [ -z "$target" ]; then + echo "media $url answered $code with no usable Location" >>"$FAILED" + return + fi + # Same origin boundary as check_redirect, and for the same reason: a rule that one + # day points off-site must not mail the token there. A bare prefix would also accept + # a lookalike host registered as an attacker's subdomain. + target_auth=() + if [ -n "$CURLRC" ]; then + case "$target" in + "$BASE" | "$BASE"/*) target_auth=(-K "$CURLRC") ;; + esac + fi + ;; + esac + # Command substitution rather than `read < <(...)`, because process substitution discards + # curl's exit status. It still fails closed either way, since curl writes 000 for + # http_code on a transport error, measured against a refused connection, a DNS failure + # and a timeout. What the status buys is a message that says which of the two happened, + # rather than leaving a reader to infer it from a bare 000. + # content_type stays LAST in this format. `read` assigns the whole remainder of the line + # to its final variable, which is what lets a value containing spaces survive intact; a + # field added after it would be swallowed into the type instead. + local out rc=0 + out=$(curl -s -o /dev/null \ + -w '%{http_code} %{size_download} %{content_type}\n' \ + --max-time 30 "${target_auth[@]}" "$target") || rc=$? + read -r code len type <<<"$out" + if [ "$rc" -ne 0 ] || [ "${code:-000}" = "000" ]; then + echo "media $url no HTTP response: curl exit $rc, transport error or timeout" >>"$FAILED" + return + fi + if [ "$code" != "200" ]; then + echo "media $url expected 200, got $code" >>"$FAILED" + return + fi + if [ "${len:-0}" -eq 0 ]; then + echo "media $url answered 200 with an empty body" >>"$FAILED" + return + fi + case "$type" in + image/*) ;; + *) echo "media $url answered 200 as $type, expected an image" >>"$FAILED" ;; + esac +} + # Invoked indirectly, the same way as check_render above. # shellcheck disable=SC2329 check_redirect() { @@ -89,7 +173,7 @@ check_redirect() { esac } -export -f check_render check_redirect +export -f check_render check_redirect check_media export BASE FAILED CURLRC echo "==> $BASE" @@ -192,16 +276,20 @@ n_redirect=$(grep -c . "$CHECKS/redirect-urls.txt") echo "==> checking $n_redirect URLs that must redirect" grep . "$CHECKS/redirect-urls.txt" | xargs -P "$PARALLEL" -I{} bash -c 'check_redirect "$@"' _ {} +n_media=$(grep -c . "$CHECKS/golden-media-live.txt") +echo "==> checking $n_media media URLs that must be served as images" +grep . "$CHECKS/golden-media-live.txt" | xargs -P "$PARALLEL" -I{} bash -c 'check_media "$@"' _ {} + # A count of zero exits non-zero, so a fallback that echoes would append a second zero. # Swallowing only the exit status keeps the printed count usable. failures=$(grep -c . "$FAILED" 2>/dev/null || true) if [ "$failures" -eq 0 ]; then - echo "PASS - $((n_render + n_redirect)) URLs honored" + echo "PASS - $((n_render + n_redirect + n_media)) URLs honored" exit 0 fi echo -echo "FAIL - $failures of $((n_render + n_redirect)) URLs" +echo "FAIL - $failures of $((n_render + n_redirect + n_media)) URLs" sort "$FAILED" | head -40 [ "$failures" -gt 40 ] && echo "... and $((failures - 40)) more" exit 1 diff --git a/checks/golden-media-live.txt b/checks/golden-media-live.txt new file mode 100644 index 0000000..dca31f0 --- /dev/null +++ b/checks/golden-media-live.txt @@ -0,0 +1,8 @@ +/media/2010/05/2010-05-1818-13-351.jpg +/media/2016/01/e3-log.jpg +/media/2023/03/image-12.png +/external/7c7d2c8ca28c936b.png +/external/68d24de592ffcefb.png +/wp-content/uploads/2010/05/2010-05-1815-32-351.jpg +/wp-content/uploads/2016/01/e3-log.jpg +/wp-content/uploads/2023/03/image-12.png