Emit robots.txt, and gate the base URL it is derived from

[ptr727]

Stacked on #58 — its commits appear here until that one merges, at which point GitHub retargets this to develop. Review the top commits only.

The old platform serves a robots.txt today and this site emitted none, so the cutover was not a return to a previous state. It was a move from having crawl directives to having none, on a site that has had them for years, and the sitemap pointer went with them. Hugo emits the file only when enableRobotsTXT is set, which is why /robots.txt answered 404 through the first production deploy.

The Sitemap: line is the load-bearing part, not the rules

Across the interim hostname's first full day, every request for sitemap.xml and feed.xml came from curl — the deploy gate's or the host side's — and none from a crawler, while /robots.txt was requested nine times and 404'd every time, five of those from real agents on a name with no inbound links. A crawler is told where a sitemap is rather than guessing it, and the only thing telling them is the file the cutover deletes.

Every Disallow the old platform serves names a WordPress path this site does not have, so the content that preserves today's behaviour is User-agent: *, no Disallow, and the sitemap:

User-agent: *
Disallow:
Sitemap: https://blog.insanegenius.com/sitemap.xml

The theme's template already derives that URL from the built baseURL, so setting the flag is the whole change and there is no second edit to remember at M7b. Verified across two base URLs rather than reasoned about: a mirror build advertises https://blog.home.insanegenius.net/sitemap.xml and a production build advertises the .com one, from the same bytes.

What check_robots gates, and what it does not

Four failures, each demonstrated failing before the check was trusted:

Case	Reported
robots.txt absent	robots.txt was not built - enableRobotsTXT is off in hugo.yaml
no Sitemap: line	robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided
line naming another origin	https://blog.insanegenius.net/sitemap.xml (this build's origin is https://blog.insanegenius.com)
line advertising a sitemap not built	https://blog.insanegenius.com/news-sitemap.xml (advertised, but news-sitemap.xml was not built)

Each exits 1.

It does not detect a wrong HUGO_BASEURL, and an earlier draft of this description claimed it did. Raised in review, in four places at once, and correct: the advertised origin and the canonical origin it is compared against are both derived from baseURL, so they agree whenever the build is coherent, including when baseURL was wrong for the target environment. Nothing inside the artifact can see that, so the gap stays on the side that knows which host it is serving — the VPS agent reads the origin out of the deployed sitemap.xml, og:url and feed.xml, which is how the first production deploy was cleared.

What the origin comparison is genuinely for is an origin written rather than derived: a committed static/robots.txt shadowing the template, with the old platform's .com sitemap line pasted into it, is the mistake it catches.

/robots.txt/ reaches the file rather than the home page

/robots.txt/, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolved through slugs.map as an unresolvable attachment slug, which sends a visitor to the home page — right for a slug nothing claims, wrong for a file that now exists.

Fixed in build-redirects.py rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. Confirmed by regenerating first and diffing: the committed maps were byte-identical to a fresh run, so the map diff here is exactly the one line. /osd.xml/ stays pointed at the home page deliberately, being an OpenSearch description this site does not emit.

Verification

This touches deploy/maps/ and hugo.yaml, so the local mirror gate applies rather than CI's green being sufficient:

PASS - the built site honors the URL contract
robots : built, advertising https://blog.home.insanegenius.net/sitemap.xml
==> installing release 20260808-142254
==> served by mirror-production
PASS - 1245 URLs honored
/robots.txt/ 301 -> https://blog.home.insanegenius.net/robots.txt
/robots.txt 200 text/plain; charset=utf-8
/osd.xml/ 301 -> https://blog.home.insanegenius.net/ (unchanged)

Production still answers 404 for the file until a deploy carries this, which is recorded in TODO.md rather than assumed.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR ensures the site emits a robots.txt (including a sitemap pointer derived from baseURL) and adds a parity gate that verifies the advertised sitemap origin and existence, while also fixing the legacy /robots.txt/ redirect behavior.

Changes:

• Enable Hugo robots.txt generation via enableRobotsTXT: true.
• Add a robots check to check-url-parity.py to verify robots.txt exists, contains Sitemap: lines, matches the build’s own origin, and doesn’t advertise missing sitemap files.
• Redirect /robots.txt/ to /robots.txt by special-casing the map generator and updating deploy/maps/slugs.map.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
TODO.md	Updates operational notes to reflect the robots/sitemap decision and the new gating behavior.
hugo.yaml	Enables Hugo robots.txt emission and documents why it matters for sitemap discovery.
deploy/maps/slugs.map	Redirects /robots.txt/ to /robots.txt instead of /.
checks/README.md	Documents the new robots/baseURL gate and why it catches wrong-baseURL builds.
checks/check-url-parity.py	Implements check_robots() and integrates it into the parity gate output.
checks/build-redirects.py	Ensures regenerated redirect maps keep /robots.txt/ -> /robots.txt without manual edits.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (4)

checks/README.md:93

• This paragraph says a "HUGO_BASEURL naming the wrong environment fails here". The described mechanism compares the robots.txt Sitemap origin to the canonical origin extracted from the same built output, so it verifies consistency and sitemap existence but won't necessarily fail just because baseURL points at the wrong environment (unless some template hardcodes a different origin).

Every list here is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate: it is what lets one contract be run against four environments without editing a line. The cost is that **a build baked with the wrong host passes the entire contract.** Point the check at the site it just built and all 1,245 URLs answer correctly while every canonical tag, feed entry, and sitemap entry names a different site. The host side found the same blind spot in its own smoke test and reported it, having checked the deployed bytes by hand instead.
`robots.txt` is the one artifact that carries an absolute URL the gate can read without being told what to expect, because its `Sitemap:` line is derived from the same `baseURL` everything else is, and the origin it must match is read from the home page's canonical link rather than configured. So the check compares the build against itself, and a `HUGO_BASEURL` naming the wrong environment fails here rather than at a reader.
The file is worth asserting for its own sake as well. Hugo emits none unless `enableRobotsTXT` is set, which is why this site answered 404 for a file the old platform served, and the `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first day no crawler fetched the sitemap once, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted.

checks/README.md:87

• The section heading states this is "the only place a wrong base URL is visible", but the robots check (and the rest of check-url-parity) currently validates the build against itself rather than against an expected environment host. The heading should avoid implying it will detect an incorrect HUGO_BASEURL by itself.

This issue also appears on line 89 of the same file.

## The robots check, which is the only place a wrong base URL is visible

TODO.md:41

• This item claims a wrong HUGO_BASEURL is now visible to a gate. The new robots check (and the explanation here) compare the robots Sitemap origin to the canonical origin extracted from the same built output, which asserts internal consistency and sitemap existence but doesn't necessarily detect that baseURL is "wrong" for the target environment.

 - **A wrong `HUGO_BASEURL` is now visible to a gate here, which it was not before.** Every contract list is path-only and the live check joins whatever base it is given, so a build baked with the wrong host passes all 1,245 URLs; the host side found the same blind spot in its own smoke test and had to read the deployed bytes by hand. `robots.txt` carries an absolute URL derived from the same `baseURL`, and the parity check compares it against the origin read from the home page's canonical link, so the two must agree. That is a side effect of this change rather than its purpose, and it is the more durable half.

checks/check-url-parity.py:105

• The check_robots docstring claims this check can "see a wrong baseURL". As implemented it compares the robots Sitemap URL to the origin extracted from the built home page canonical link, so it only verifies internal consistency (and that the advertised sitemap file exists), not that baseURL matches an externally expected host/environment.

 The sitemap URL is absolute and derived from baseURL, which makes this the one build-time
assertion that can see a wrong baseURL at all. Every contract list is path-only and the live
check joins whatever base it is given, so a build baked with the wrong host passes all of them
while every canonical tag, feed entry and sitemap entry names the wrong site.

[ptr727]

The four suppressed findings are one finding, and it is right

They carry no thread, so they are answered here. All four name the same claim in four places — checks/README.md at the heading and at line 93, TODO.md, and the check_robots docstring — and the claim was wrong:

The described mechanism compares the robots.txt Sitemap origin to the canonical origin extracted from the same built output, so it verifies consistency and sitemap existence but won't necessarily fail just because baseURL points at the wrong environment.

Withdrawn in 8ae0a4b. The reason it cannot work is structural rather than a bug in the check: both sides of the comparison are derived from baseURL, so they agree whenever the build is coherent, including when baseURL held the wrong value for the environment being deployed to. Adding an expected origin from this side would not fix it either — the only candidate value is the same variable, and a per-environment host written into the repo is a literal that has to be edited at the cutover, which is the shape this repo already treats as a trap.

So the gap stays where it was, on the side that knows which host it is serving: the VPS agent reads the origin out of the deployed sitemap.xml, og:url and feed.xml and reports the counts either way, which is how the first production deploy was cleared.

Each of the four places now states what the comparison actually proves, internal consistency, and what it still catches:

• an origin written rather than derived — a committed static/robots.txt shadowing the template is how that happens, and pasting the old platform's .com sitemap line into one is the specific mistake
• a sitemap advertised but never built
• the file absent, and no Sitemap: line at all

A gate described as catching more than it catches is worse than no gate, because the next person stops looking for the failure it does not see. Good catch, and it was worth four comments rather than one.

Also noted, not actionable here

The first review round reported coverage=PARTIAL, 5 of 6 changed files read. Per the runbook a re-request has never cleared a partial on record, so it is reported rather than retried into, and the merge decision is the maintainer's with that known.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (4)

checks/check-url-parity.py:103

• This docstring claims the Sitemap: line is the only absolute URL the gate can read, but this script already reads absolute URLs elsewhere (e.g., the home page canonical link in site_origin and absolute asset references in collect_refs). Consider rephrasing to the narrower point: this is the only place the check compares an origin rather than joining one.

 The Sitemap: line is the only absolute URL in the build that a gate here can read at all, every
contract list being path-only, so it is the only place an origin is checked rather than joined.

checks/check-url-parity.py:98

• The docstring says "Two failures, one gate", but this function actually gates multiple distinct failure modes (missing file, missing Sitemap line, wrong origin, and advertised-but-not-built sitemap). Updating the wording keeps the docstring consistent with the behavior and the checks/README description.

This issue also appears on line 102 of the same file.

 Two failures, one gate. Hugo emits no robots.txt unless enableRobotsTXT is set, which is why

checks/check-url-parity.py:117

• robots.read_text(encoding="utf-8") will raise UnicodeDecodeError and crash the whole parity check if robots.txt contains invalid UTF-8 (e.g., from an accidentally committed static/robots.txt). Using an explicit errors= keeps the gate reporting a normal failure instead of stack-tracing.

 advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", robots.read_text(encoding="utf-8"))

checks/README.md:91

• This sentence says the Sitemap: line is the only absolute URL a gate here can read, but earlier in this README it notes that the checks read absolute asset references too. Rewording to focus on the intended point (origin comparison) avoids contradicting the earlier section.

**The `Sitemap:` line is the only absolute URL in the build a gate here can read.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments, so an origin is normally joined rather than checked. This is the one place one is compared.

[ptr727]

Round 3's four suppressed findings, all accepted

No threads again, so answered here. Fixed in 29a9669. One of them is a real defect rather than wording.

The crash path.robots.read_text(encoding="utf-8") at strict, in the one gate written to report a bad robots.txt:

will raise UnicodeDecodeError and crash the whole parity check if robots.txt contains invalid UTF-8

Confirmed in both directions rather than reasoned about, against a file carrying 0xff:

strict would have raised: UnicodeDecodeError 'utf-8' codec can't decode byte 0xff in position 67
errors="replace": reports through the ordinary path, whole run exits 0

The file most likely to carry an invalid byte is a committed static/robots.txt, which is precisely the case this check exists for, so the gate would have stack-traced on its own subject. Now errors="replace", with the reason in a comment so nobody tightens it back.

Two claims narrowed. The docstring and the README both said the Sitemap: line is the only absolute URL this gate can read. It is not — site_origin reads the home page's canonical link and collect_refs reads absolute asset references, and the README says so a few paragraphs earlier, so the sentence contradicted its own file. The true and narrower point is that it is the only place an origin is compared rather than joined, and both now say that.

One stale count. "Two failures, one gate" where there are four. Now enumerated in the docstring, matching the README and the table in this description.

On coverage, for the record

Round 3 reports coverage=PARTIAL at the same 5-of-6 ratio as rounds 1 and 2, which is what the runbook says to expect: a re-request has never cleared a partial. Three rounds have now read this diff and the findings have moved from the description's claims, to the comments' claims, to an actual defect in the code, so the reviewer is reaching the substance even at 5 of 6. Reported rather than retried into, and the merge decision is the maintainer's knowing it.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

checks/check-url-parity.py:142

• The robots gate prints robots : built, advertising … before verifying that the advertised sitemap file(s) exist. If an advertised sitemap is missing, the check still fails, but the earlier log line is misleading and can confuse gate output interpretation.

 # Named rather than counted, since there is one today and the line is the thing being checked.
print(f"robots : built, advertising {advertised[0]}")
return [
f"{u} (advertised, but {u[len(origin) + 1:]} was not built)"
for u in advertised

TODO.md:38

• This TODO entry still says “Since this site emits none” in the present tense, but the preceding bullet states robots.txt is now built. Rewording to past tense keeps the history accurate while reflecting the current state after enabling enableRobotsTXT.

 - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit.

[ptr727]

Round 4's two suppressed findings, both accepted

Fixed in 0d964d9.

The misleading log line. The gate printed robots : built, advertising … before checking that the advertised sitemap exists, so the one failing case printed a success-shaped line immediately above its own failure. Every branch now prints exactly once, after deciding:

clean robots : built, advertising https://blog.insanegenius.com/sitemap.xml
sitemap missing robots : built, 1 advertised sitemap(s) not built
FAIL robots: 1 problems
https://blog.insanegenius.com/news-sitemap.xml (advertised, but news-sitemap.xml was not built)

Re-demonstrated failing after the change, exit 1.

The tense. The TODO.md paragraph describing what the cutover would have cost still read "since this site emits none", one bullet below the entry saying the file is now built. It reads as history now, which is what it is.

Where this leaves the pull request

Four rounds have read this diff and every round has found something real, so the reviewer is doing its job. What has not moved is coverage=PARTIAL at an identical 5-of-6 ratio in all four rounds, which the runbook says a re-request does not cure. The merge decision stays the maintainer's, taken knowing one of the six changed files has had no review in any round, and splitting this into a docs change and a code change is the remedy available if full coverage is wanted before it lands.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (1)

checks/check-url-parity.py:120

• The missing-file failure message currently asserts enableRobotsTXT is off, but public/robots.txt could be missing for other reasons (partial build, deleted file, wrong output dir). This can mislead debugging; make the message describe what was observed and point to enableRobotsTXT as a likely cause rather than a certainty.

 robots = public / "robots.txt"
if not robots.is_file():
print("robots : missing")
return ["robots.txt was not built - enableRobotsTXT is off in hugo.yaml"]

[ptr727]

Round 5's one finding, accepted

The missing-file failure message currently asserts enableRobotsTXT is off, but public/robots.txt could be missing for other reasons (partial build, deleted file, wrong output dir).

Fixed in bb7f10d. A partial build, a deleted file and the wrong output directory reach the identical state, so naming one as the cause sends a reader to verify a setting that is already correct. The message now names the path it looked for, offers the likely cause as likely, and says the alternatives are indistinguishable from here:

FAIL robots: 1 problems
<public>/robots.txt does not exist - likely enableRobotsTXT is unset in hugo.yaml,
though a partial build or the wrong output directory look identical here

That is the same shape check_orphans uses, and for the same reason: this repo's rule is that a message naming one of several causes sends a reader looking in the wrong place.

Re-demonstrated failing after the change, exit 1.

Standing state

Five rounds, five findings, all accepted — the description's claims, then the comments' claims, then a crash path, then misleading output, now a misleading message. coverage=PARTIAL has held at 5 of 6 in every one of the five, which the runbook says a re-request does not cure, so it is reported rather than retried into and the merge decision remains the maintainer's.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (1)

checks/check-url-parity.py:45

• The summary noun for the new "robots" check is set to "problems", which produces grammatically incorrect output when exactly one item is found (e.g., FAIL robots: 1 problems). Consider using a singular/plural-neutral noun to keep the failure output readable.

FAILURE_NOUN = {"gallery": "stray nodes", "robots": "problems"}

[ptr727]

Round 6's one finding, accepted, and it was not only the new noun

The summary noun for the new "robots" check is set to "problems", which produces grammatically incorrect output when exactly one item is found (e.g., FAIL robots: 1 problems).

Fixed in fa90708. The noun was a bare string and a count is always printed beside it, so every label with a plural noun had this, including the pre-existing gallery one — FAIL gallery: 1 stray nodes on a single stray node. The new noun was wrong in the same way rather than in a new way, so the pair is the fix and it covers both.

FAIL robots: 1 problem 3 problems
FAIL gallery: 1 stray node 27 stray nodes
FAIL render: 1 missing 5 missing

The default stays missing and needs no pair, being count-neutral already. Verified at 1 and at many for both labels, and the clean run is unchanged.

I am stopping the review loop here

Six rounds, six findings, all accepted and none declined: two overclaims in this description, four in the code comments, a UnicodeDecodeError that would have taken down the whole parity run, a summary line that printed above its own failure, a message asserting one of three indistinguishable causes, and this pluralization.

coverage=PARTIAL has held at an identical 5 of 6 files in every one of the six rounds, which is what the runbook predicts and says not to retry into. Continuing to push rounds that each find one more thing while coverage never moves is motion rather than progress, so this is where it stops.

The merge decision is the maintainer's, taken knowing one of the six changed files has had no review in any round. If full coverage is wanted first, splitting this into a documentation change and a code change is the available remedy.

[ptr727]

Closed by GitHub when its base branch record-log-pull-installed was deleted on the #58 merge, and it cannot be reopened because that branch is gone. Continued as #60, same seven commits rebased onto develop with #58's two dropped.

Two things from this pull request are worth carrying rather than losing here. It never ran CI — test-pull-request.yml triggers on pull_request: branches: [main, develop] and this was opened against a feature branch to stack it, so its checks=2/2 was two skipped jobs and the validation gate never fired. #60 targets develop, so it runs. And six review rounds here found six real things, all accepted and none declined, every fix carried into #60; the round-by-round exchange stays above as the record.