diff --git a/OPERATIONS.md b/OPERATIONS.md index 225a56e..9e87c85 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -237,7 +237,9 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access **Retention on the local mirrors is a different question and is unsolved.** Those containers use Docker's `json-file` driver with the built-in defaults, so a mirror's log grows without bound and is discarded when its container is recreated. That matters less than it did on the VPS, because the mirrors serve no readers, and it belongs to the host rather than to this repository, the same split "Retention" and "Who Owns What" describe for release pruning. -**The off-host copy of the access log is a design rather than a fact.** The VPS holds the only copy until the pull to the backup host runs, which makes the host's retention window the hard deadline rather than a comfortable margin. A review reads whatever survived and reports confidently on it either way, so settle this before the first review is expected to mean anything. +**The off-host copy of the access log exists, and the schedule that maintains it is younger than the copy.** The pull to the backup host is installed as a `systemd` timer running daily at 09:00 UTC, chosen to sit behind both producers on the VPS rather than beside them, and its first copy was made by hand rather than by the timer. Read the unit and its last run on the backup host rather than trusting this paragraph, for the same reason retention is read from the VPS: a claim about a schedule is only worth what the machine says. + +**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it**, and treat a rename on the VPS as something the channel has to carry, because no transfer will. ## Who Owns What diff --git a/TODO.md b/TODO.md index 75cfb40..6795791 100644 --- a/TODO.md +++ b/TODO.md @@ -19,7 +19,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | | VPS production | **M7a done 2026-08-08.** Serving release `20260808-041050` at `blog.insanegenius.net`, answering `200` unauthenticated, verified 9/9 from the host side with the built `baseURL` read from the deployed bytes. DNS for the public name is still on the old platform | -| Operations | not started. Neither the periodic log review nor the off-host log pull has run on its cadence | +| Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has not run at all | ## Blocked on the maintainer @@ -33,10 +33,14 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. -- **Decide what `robots.txt` says, now before the cutover rather than before the first deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - - **The deploy did not fix the 404, which is the part that reads the wrong way round, and is now measured rather than predicted.** The VPS agent raised this in §22.10 and asked two questions of this side. Both are now measured rather than assumed: **this site emits no `robots.txt` at all**, because `hugo.yaml` sets no `enableRobotsTXT`, so the 404 survives the deploy and `X-Robots-Tag` stays the only control. **`sitemap.xml` is emitted** and becomes fetchable on the interim name at that same deploy, listing whatever `HUGO_BASEURL` was built with, which is the second reason that variable mattered. Read together: after the rehearsal deploy a crawler gets a full sitemap and no robots file. - - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the `.com` sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **Deliberately undecided**, and the three options are to preserve today's behavior, to write what this site actually wants, or to keep emitting nothing and accept the loss. -- **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. +- **`robots.txt` is decided and built, 2026-08-08, and what remains is that production has not been redeployed since.** The site emits one now, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. The record below is kept because the reasoning is what the next decision about crawl directives will need, and because production still answers 404 until a deploy carries this. + - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set, so this describes the release production is still serving rather than the current build. + - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Because this site emitted none, M7b would not have been a return to a previous state, it would have been a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer would have gone with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. + - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. + - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. + - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. +- **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. +- **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **Publish a release from `main`, once the pipeline has soaked.** `1.0.11` is the newest release from `main` and was cut on 2026-08-01, ahead of every deploy change, so the next one is the first that would describe a site actually serving its public address. The mechanism is proven and is not what this waits on: it waits on the switchover being trusted rather than merely green, which is what the log review under **Recurring operations** establishes and no gate can. A release cut before that names a state that has not held yet. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. @@ -47,7 +51,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y **Everything above ends; this section does not.** The list above finishes when the site serves its public address, and these two start there and never stop. Both are written down as procedures and neither has run on its cadence yet, which is the shape a recurring duty has right before it silently becomes a one-off. An end-to-end exercise of this system is a deploy **plus** the operations that follow it, so a run that proves staging and production and stops has proven the half that has an obvious finish line. - **Review the logs for non-200s**, daily for the first week after cutover, then monthly. Real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor, and a staging probe for `/wp-login.php` answered by the auth gate rather than by the site is the shape of what Caddy never sees. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". -- **Pull the log off the VPS, on a schedule**, which is [#53][issue-53]. The access log is deliberately outside the nightly encrypted archives, because those are fourteen full copies with no dedupe and an append-only file would be multiplied by fourteen for no recovery benefit. So the VPS's 400-day window is the **only** copy until this runs. The two rsync lines are now written, parameterized, and tested against the live host, and they are not installed, because installing needs root. **Until they are, say plainly that "backed up" describes a design rather than a fact in anything built on top of that log.** +- **Pull the log off the VPS, on a schedule**, which is [#53][issue-53]. The access log is deliberately outside the nightly encrypted archives, because those are fourteen full copies with no dedupe and an append-only file would be multiplied by fourteen for no recovery benefit, so the VPS's 400-day window was the **only** copy until this ran. **It is installed**, as `vps-backup-pull.timer` at 09:00 UTC daily with `Persistent=true`, and a first copy exists: 42 archives and 4 log files, pulled 2026-08-08 12:59 UTC. **That run was started by hand, so the timer has never fired**, which is the distinction worth keeping until 2026-08-09 09:00 UTC proves the schedule rather than the script. One copy is a fact; "backed up daily" is still a unit file. + - **A rename on the VPS does not reach this copy, and nothing detects that it did not.** The pull deliberately passes no `--delete` for the logs, since that flag exists to mirror the VPS's fourteen-archive window and must never touch an append-only file. So when the host side renamed and merged its two mis-dated archives, the pre-fix name survived here: `access.log-2026-08-08`, 52 lines, every one of them 2026-08-07 traffic and every one already inside the merged `access.log-2026-08-07.gz`, which holds 58. Verified a strict subset with `comm -23` rather than assumed. **A line count over the off-host archive therefore returns 110 lines where 58 exist, half of them filed under a date whose traffic they are not** — which is exactly the defect the host side fixed, surviving on the copy the log review will read once the VPS's window rolls past what it needs. The general form is that any rename, merge, or re-compression of an already-pulled log leaves the old name here permanently, and the only propagation mechanism is a note in the channel. Removing that one file is the maintainer's call, since it is a deletion inside a backup tree. ## Owed to the hub @@ -68,7 +73,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. - **What `robots.txt` says, which is undecided and is the last non-mechanical item before M7b.** Recorded under "Next" above, where it blocks the cutover. -- `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`. +- `/osd.xml/` sits in `slugs.map` pointing at `/`, and stays there unless this site ever emits an OpenSearch description. `/robots.txt/` was the other half and is resolved, above. - Content is capped at a fixed 720px on every screen, because PaperMod's width is four CSS variables with no responsive term and no Hugo parameter. The prose measure is right and should stay; images and galleries inheriting the same cap is the part that costs something on a wide display. The knobs, the override location, and the `--gap` trap are documented under "Customization points" in [`themes/README.md`](./themes/README.md). ## Deliberate deviations from the fleet baseline diff --git a/checks/README.md b/checks/README.md index 83bc682..89d866a 100644 --- a/checks/README.md +++ b/checks/README.md @@ -84,6 +84,18 @@ A count is all the check can observe, and two causes reach each direction: it ri **Both directions read absolute references as well as relative ones.** Hugo writes an absolute URL wherever a template resolves one against the base, which the entry-cover image on every list page does. Reading only rooted paths made those files look linked from nowhere while they were being displayed, and left a broken one unchecked in the other direction. The origin is read from the home page's canonical link rather than assumed, since staging and production build with different base URLs and a hardcoded host would check one environment's output against another's. No canonical link is a hard failure, because a guessed origin inflates the orphan count by exactly the pages that use one. +## The robots check, and the one thing no gate here can do + +Hugo emits no `robots.txt` unless `enableRobotsTXT` is set, so this site served none until the flag was turned on, and the old platform serves one. The `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first full day of traffic, every request for `sitemap.xml` came from `curl` and none from a crawler, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted. + +**The `Sitemap:` line is the one place this gate compares an origin rather than joining one.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments. Other absolute URLs are read here, the home page's canonical link and the absolute asset references described above among them, but they are read to resolve a reference rather than to check one host against another. + +**What that comparison proves is internal consistency, and it is worth being exact about the limit.** The advertised origin must match the one read from the home page's canonical link, and both are derived from the same `baseURL`, so they agree whenever the build is coherent — including when `baseURL` held the wrong value for the environment being deployed to. **Nothing inside the artifact can detect that**, which is why the check belongs on the side that knows which host it is serving: the VPS side reads the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml` and reports the counts either way. A build baked with the wrong host still passes all 1,245 URLs here, and it passes this too. + +What the comparison does catch is an origin that was **written rather than derived** — a committed `static/robots.txt` shadowing the template is the way that happens, and pasting the old platform's `.com` sitemap line into one is the specific mistake it would catch — along with a sitemap advertised but never built. + +`/robots.txt/`, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolves through `slugs.map` like any other one-segment legacy URL, and the generator special-cases it to the real file rather than to the home page. `/osd.xml/` stays pointed at the home page deliberately: it was the old platform's OpenSearch description and this site emits no such file. + ## The gallery check, which no direction above can reach Every check above reasons about a URL: whether it renders, whether it resolves, whether anything points at it. Content misplaced **inside** a gallery satisfies all of that. The file exists, the reference resolves, and something links it, so the media surface is green in both directions while the page is laid out wrong. The defect is one of structure, which is why three variants of it survived the conversion and every gate since. diff --git a/checks/build-redirects.py b/checks/build-redirects.py index 414ef9f..3d94171 100755 --- a/checks/build-redirects.py +++ b/checks/build-redirects.py @@ -19,6 +19,15 @@ # The WordPress importer registers the full slug, and both answer, so both are mapped. BLOGGER_SLUG_LIMIT = 40 +# A one-segment URL naming a file the site serves at the root, rather than a page slug. +# The resolver below reads these as unresolvable attachment slugs and sends them to the home +# page, which is the right answer for a slug nothing claims and the wrong one for a file that +# exists: /robots.txt/ should reach /robots.txt. Named here rather than hand-edited into the +# generated map, because the map is rewritten from the capture and a hand edit does not survive. +# /osd.xml/ stays out deliberately. It was the old platform's OpenSearch description and this +# site emits no such file, so the home page remains the honest destination for it. +WELL_KNOWN = {"/robots.txt/": "/robots.txt"} + def text(el, path): node = el.find(path, NS) @@ -179,7 +188,9 @@ def main(argv): resolved, via_parent, via_media, orphan = [], 0, 0, [] for u in needed: slug = u.strip("/") - if slug in by_slug: + if u in WELL_KNOWN: + resolved.append((u, WELL_KNOWN[u])) + elif slug in by_slug: resolved.append((u, by_slug[slug])) via_parent += 1 elif slug.lower() in file_to_post: diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 2d017fa..86a24e5 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -41,8 +41,15 @@ # Every check above returns a list, and the shared summary called all of them "missing". That is # what a URL that did not build is, and it is not what a stray node inside a gallery is: those are # present, which is the whole complaint. The default stays "missing" so a check added later reads -# the way the older ones do unless it says otherwise. -FAILURE_NOUN = {"gallery": "stray nodes"} +# the way the older ones do unless it says otherwise, and it needs no pair because it is already +# count-neutral. A count is always printed beside the noun, so the pair is (singular, plural) and +# "1 stray nodes" was the reason for making it a pair rather than a string. +FAILURE_NOUN = {"gallery": ("stray node", "stray nodes"), "robots": ("problem", "problems")} + + +def failure_noun(label, count): + forms = FAILURE_NOUN.get(label, ("missing", "missing")) + return forms[0] if count == 1 else forms[1] def load(name): @@ -92,6 +99,65 @@ def check_media(public): return missing +def check_robots(public): + """Check that robots.txt exists and advertises this build's own sitemap. + + Four failures, one gate: the file absent, no Sitemap: line, a line naming another origin, and a + line advertising a sitemap that was not built. Hugo emits no robots.txt unless enableRobotsTXT + is set, which is why this site served none until that flag was turned on, and the file's only + load-bearing line points at the sitemap a crawler is otherwise unlikely to find, being told + rather than guessing. + + Every contract list is path-only and check-live-urls.sh joins whatever base URL it is handed, so + this is the only place the gate *compares* an origin rather than joining one. Other absolute + URLs are read here - the home page's canonical link, and absolute asset references - but they + are read to resolve a reference rather than to check a host against another. + + What the comparison proves is internal consistency: the advertised origin matches the one the + canonical link carries. Both come from baseURL, so this cannot tell that baseURL was the wrong + value for the environment being deployed to - nothing in the artifact can, which is why that + check belongs on the side that knows which host it is serving. It does catch an origin that was + written rather than derived, a static robots.txt shadowing the template being the way that + happens, and it catches a sitemap advertised but not built. + """ + robots = public / "robots.txt" + if not robots.is_file(): + # Naming one cause as the cause sends a reader to check a setting that is already correct. + # enableRobotsTXT is the likely one and a partial build or the wrong output directory reach + # the same state, which is the same reason the orphan messages name both of their causes. + print("robots : missing") + return [ + f"{robots} does not exist - likely enableRobotsTXT is unset in hugo.yaml, " + "though a partial build or the wrong output directory look identical here" + ] + + origin = site_origin(public) + # errors="replace" rather than strict, or invalid UTF-8 raises out of the whole parity run and a + # gate that exists to report a bad robots.txt stack-traces on one instead. A committed + # static/robots.txt is the file most likely to carry it, and it is the case this check is for. + text = robots.read_text(encoding="utf-8", errors="replace") + advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", text) + if not advertised: + print("robots : built, no Sitemap line") + return ["robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided"] + + wrong = [u for u in advertised if not u.startswith(origin + "/")] + if wrong: + print(f"robots : built, {len(wrong)} Sitemap line(s) naming another origin") + return [f"{u} (this build's origin is {origin})" for u in wrong] + + # The summary is printed after the last assertion rather than before it, or the missing-sitemap + # case reads as a pass on the line above its own failure. Every branch here prints exactly once. + unbuilt = [u for u in advertised if not (public / u[len(origin) + 1 :]).is_file()] + if unbuilt: + print(f"robots : built, {len(unbuilt)} advertised sitemap(s) not built") + return [f"{u} (advertised, but {u[len(origin) + 1:]} was not built)" for u in unbuilt] + + # Named rather than counted, since there is one today and the line is the thing being checked. + print(f"robots : built, advertising {advertised[0]}") + return [] + + def site_origin(public): """The site's own scheme and host, read from the artifact rather than assumed. @@ -307,6 +373,7 @@ def main(argv): ("assets", check_assets(public, refs)), ("orphans", check_orphans(public, refs)), ("gallery", check_galleries(public)), + ("robots", check_robots(public)), ): if found: failures.append((label, found)) @@ -317,7 +384,7 @@ def main(argv): print() for label, found in failures: - print(f"FAIL {label}: {len(found)} {FAILURE_NOUN.get(label, 'missing')}") + print(f"FAIL {label}: {len(found)} {failure_noun(label, len(found))}") for item in found[:20]: print(f" {item}") if len(found) > 20: diff --git a/deploy/maps/slugs.map b/deploy/maps/slugs.map index d27247c..2eb63c0 100644 --- a/deploy/maps/slugs.map +++ b/deploy/maps/slugs.map @@ -100,7 +100,7 @@ /jellyfish-lighting-ports-500x214-1/ / /osd.xml/ / /packet/ /2020/09/05/abandoning-greeneye-for-sense/ -/robots.txt/ / +/robots.txt/ /robots.txt /sce_smartmeter/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-dashboard/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-graphing/ /2020/09/05/abandoning-greeneye-for-sense/ diff --git a/hugo.yaml b/hugo.yaml index 2ec8d0f..ded53f7 100644 --- a/hugo.yaml +++ b/hugo.yaml @@ -27,6 +27,15 @@ refLinksErrorLevel: ERROR # The deploy rebuilds the site rather than receiving it as an artifact, so the build stays reproducible. enableGitInfo: false +# Off by default in Hugo, which is why this site answered 404 for a file the old platform served. +# The Sitemap: line is the load-bearing part: across the interim hostname's first full day of +# traffic, every request for sitemap.xml came from curl and none from a crawler, because a crawler +# is told where a sitemap is rather than guessing, and the only thing telling them today is the +# robots.txt the cutover deletes. +# The theme's template derives that URL from the built baseURL, so it names .net during the +# rehearsal and .com afterwards with no second edit to remember at the cutover. +enableRobotsTXT: true + params: description: My discoveries, frustrations, and delights defaultTheme: auto