Conversation
* Record the log pull as installed, and the rename it does not carry The off-host access-log pull is no longer a design. It is installed as a daily timer, and its first copy exists. Both records here said the opposite, because both were written before it was installed, and an operational document understating what runs is as misleading as one overstating it. The distinction kept is that the timer has never fired: that first copy was started by hand, so one copy is a fact and "backed up daily" is still a unit file until the first scheduled run. The finding the copy produced is the more useful half. The pull passes no --delete for the logs, deliberately, so a file the VPS renames after it has been pulled keeps its old name here forever alongside the new one. That has already happened, to two archives whose names were a day ahead of their contents, and a count over the archive by filename now returns 110 lines where 58 exist. Verified a strict subset with comm rather than assumed. Read a date from a line's StartUTC rather than from the filename holding it. Also recorded: why restoring mtimes in CI comes after a live media check rather than before, since making --link-dest effective means a third of the tree starts arriving as hard links carrying their inode's mode instead of having it re-asserted per deploy; and the log evidence that reframes the robots.txt decision, where no crawler has ever fetched the sitemap and the only thing advertising it is the file the cutover deletes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Name the VPS as the renaming side, and say scheduled rather than cadence Both review findings are about a reader landing on the wrong side of a two-host sentence. The rename paragraph sat under one that mentions the backup host, so "the host renames" could be read as the machine doing the pulling rather than the one being pulled from, and the State row's "has run on its cadence" reads as "has run at all" one clause before the same row says it has copied once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Emit robots.txt, and gate the base URL it is derived from The old platform serves a robots.txt today and this site emitted none, so the cutover was not a return to a previous state, it was a move from having crawl directives to having none on a site that has had them for years. Hugo emits the file only when enableRobotsTXT is set, which is why /robots.txt answered 404 through the first production deploy. The Sitemap: line is the load-bearing part rather than any rule. Across the interim hostname's first full day no crawler fetched sitemap.xml or feed.xml once, every request to either coming from curl: a crawler is told where a sitemap is rather than guessing, and the only thing telling them is the file the cutover deletes. Every Disallow the old platform serves names a WordPress path this site does not have, so the content that preserves today's behavior is User-agent: *, no Disallow, and the sitemap. The theme's template already derives that URL from the built baseURL, so setting the flag is the whole change and there is no second edit to remember at the cutover. Verified across two base URLs: the mirror advertises its own sitemap and a production build advertises the site's. /robots.txt/ with a trailing slash is in the redirect contract and sent visitors to the home page. It now resolves to the real file, fixed in build-redirects.py rather than in the generated map, since the map is rewritten from the capture and a hand edit does not survive. Verified byte-identical regeneration otherwise: the map diff is that one line. /osd.xml/ stays pointed at the home page, being an OpenSearch description this site does not emit. The check is worth more than the file. Every contract list is path-only and check-live-urls.sh joins whatever base it is handed, which is what lets one contract cover four environments and also means a build baked with the wrong host passes all 1,245 URLs while every canonical tag and sitemap entry names another site. robots.txt is the one artifact whose absolute URL a gate can read without being told what to expect, so the parity check compares it against the origin read from the home page's canonical link. Four failures gated, all four demonstrated failing before the check was trusted: file absent, no Sitemap line, a line naming another origin, and a line advertising a sitemap not built. Released to the local production mirror and checked live, since this touches deploy/maps and hugo.yaml: 1245 URLs honored, /robots.txt/ 301s to /robots.txt, and that file answers 200 as text/plain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Bound both claims to what was measured Two overclaims in comments, both real. The hugo.yaml note said no crawler has ever fetched this site's sitemap unprompted, where the evidence is one day of traffic on one hostname, and the docstring said the site answered 404 for years, where the behaviour is a config flag rather than a duration. Each now says what was observed and over what window. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Withdraw the wrong-baseURL claim, which the check cannot support Four places said this check makes a wrong HUGO_BASEURL visible. It does not, and the reason is structural rather than a bug: the advertised sitemap origin and the canonical origin it is compared against are both derived from baseURL, so they agree whenever the build is coherent, including when baseURL held the wrong value for the environment being deployed to. Nothing inside the artifact can see that, which is exactly why the VPS side reads the origin out of the deployed sitemap.xml, og:url and feed.xml and reports the counts either way. Raised in review, in all four places at once, and correct. Each now states what the comparison proves, internal consistency, and what it still catches: an origin written rather than derived, where a committed static/robots.txt shadowing the template is how that happens, and a sitemap advertised but never built. A gate described as catching more than it catches is worse than no gate, because the next person stops looking for the failure it does not see. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Stop the robots check crashing on bytes it exists to inspect, and narrow two claims Three findings, all correct. read_text with strict UTF-8 raises out of the entire parity run on an invalid byte, so the one gate written to report a bad robots.txt would stack-trace on one instead of failing normally. Confirmed both ways: strict raises UnicodeDecodeError at position 67 on a file carrying 0xff, and errors=replace reports through the ordinary path. The file most likely to carry it is a committed static/robots.txt, which is the case this check is for. The docstring still said two failures where there are four, and both the docstring and the README said the Sitemap: line is the only absolute URL this gate can read. It is not, since site_origin reads the canonical link and collect_refs reads absolute asset references. The true and narrower point is that it is the only place an origin is compared rather than joined. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Print the robots summary after its last assertion, not before Two findings, both correct. The advertising line printed before the sitemap-existence check ran, so the one failing case read as a pass on the line immediately above its own failure. Now every branch prints exactly once, after deciding: missing-sitemap reports "1 advertised sitemap(s) not built" and only a clean run prints what it advertises. The TODO paragraph describing the cutover loss still said "since this site emits none" in the present tense, one bullet below the entry saying the file is now built. It reads as history now, which is what it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Describe the missing robots.txt rather than asserting why it is missing The message named enableRobotsTXT as the cause. A partial build, a deleted file, and the wrong output directory reach the identical state, so asserting one sends a reader to verify a setting that is already correct. It now names the path it looked for, offers the likely cause as likely, and says the alternatives are indistinguishable from here, which is the same shape the orphan messages use for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Pluralize the failure noun by count, which fixes the gallery too "FAIL robots: 1 problems" was the report. The noun was a bare string and a count is always printed beside it, so every label with a plural noun had the same defect: the pre-existing gallery one prints "1 stray nodes" on a single stray node, which is the shape that made the new noun wrong in the same way. Now a (singular, plural) pair chosen by count. The default stays "missing" and needs no pair, being count-neutral already. Verified across both labels at 1 and at many, and the clean run is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR promotes two develop changes to main: (1) correct operational records to reflect that the off-host access-log pull is installed and has produced an initial copy, and (2) introduce/verify robots.txt behavior (including redirect parity for /robots.txt/) and add CI gating to ensure the built site emits a robots.txt that advertises the build’s own sitemap.xml.
Changes:
- Update operational documentation to reflect the installed
systemdtimer for off-host access-log pulls and the known rename/divergence caveat. - Enable Hugo’s
enableRobotsTXTand document the reasoning/constraints around theSitemap:line being derived frombaseURL. - Extend the URL parity gate to validate
robots.txtexistence and correctness, and ensure/robots.txt/maps to/robots.txtvia redirect map generation.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| TODO.md | Updates project status/next-steps records to reflect the log-pull installation state and the robots.txt decision being implemented (but not yet deployed to production). |
| OPERATIONS.md | Documents that the off-host log copy exists, how to verify the timer/schedule, and the non-propagating rename caveat for pulled logs. |
| hugo.yaml | Enables enableRobotsTXT and documents why robots.txt (specifically Sitemap:) matters operationally for this site. |
| deploy/maps/slugs.map | Updates the redirect contract so /robots.txt/ maps to /robots.txt instead of /. |
| checks/README.md | Documents what the robots check gates, and clearly states what it cannot prove (wrong baseURL for an environment). |
| checks/check-url-parity.py | Adds check_robots() and improves failure summary nouns to avoid incorrect singular/plural output. |
| checks/build-redirects.py | Special-cases /robots.txt/ as a well-known root file path during redirect map generation so the fix survives regeneration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes,
6689e56and884c50f. One is a record correction, the other is the first behaviour change to the built site since the gallery fix.robots.txt,/robots.txt/reaches it, and the parity gate covers bothWhat changes on the site
robots.txtis emitted, and itsSitemap:line is derived from the builtbaseURL. The old platform serves one today; this site served none, so the cutover would have moved a site that has had crawl directives for years to having none, taking the sitemap pointer with it. Deriving the URL rather than typing it means the file is correct on.netduring the rehearsal and correct on.comafterwards, with nothing to remember at the cutover.The evidence that settled the content came from the VPS side's access log: across the interim hostname's first full day, no crawler fetched
sitemap.xmlorfeed.xmlonce — every request to either came fromcurl— while/robots.txtwas asked for nine times and 404'd every time. A crawler is told where a sitemap is rather than guessing, so theSitemap:line is the load-bearing part and everyDisallowthe old platform serves names a WordPress path this site does not have./robots.txt/, with a trailing slash, now reaches the file instead of the home page, fixed in the map generator rather than the generated map so it survives regeneration.What this does not do
Production does not serve any of it yet.
blog.insanegenius.netis still on release20260808-041050, which predates both this and the gallery fix, so a deploy is what carries them and that deploy is a separate decision.Nothing here detects a wrong
HUGO_BASEURL. An earlier draft of #60 claimed the new check did; it cannot, because both sides of its comparison derive frombaseURL. That gap stays on the side that can read deployed bytes, which is where it already lives.Verification
Both pull requests passed the validation gate on
develop. #60 additionally went through the local mirror, since it touchesdeploy/maps/andhugo.yamland CI cannot prove a redirect:1245 URLs honored,/robots.txt/301s to/robots.txt, and that file answers 200 astext/plain.check_robotsgates four failures and each was demonstrated failing before the check was trusted. One of them was found by review rather than by design: strict UTF-8 decoding would have raised out of the entire parity run on an invalid byte, in the one gate written to report a badrobots.txt.Review history worth knowing about.#60 replaces #59, which GitHub closed when its base branch was deleted. #59 never ran the validation gate, because it was opened against a feature branch and
test-pull-request.ymltriggers only onmainanddevelop— that is why #60 exists as a separate pull request rather than as a rebase. Six review rounds on #59 found six defects, all accepted and none declined, and every fix is in884c50f. #60's own round reportedcoverage=PARTIALat 5 of 6 files while naming all six in its own overview table, which is filed at the hub as ProjectTemplate#623.🤖 Generated with Claude Code