Promote develop to main: robots.txt, and the log pull recorded as installed

OPERATIONS.md

@@ -237,7 +237,9 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access
 
 **Retention on the local mirrors is a different question and is unsolved.** Those containers use Docker's `json-file` driver with the built-in defaults, so a mirror's log grows without bound and is discarded when its container is recreated. That matters less than it did on the VPS, because the mirrors serve no readers, and it belongs to the host rather than to this repository, the same split "Retention" and "Who Owns What" describe for release pruning.
 
-**The off-host copy of the access log is a design rather than a fact.** The VPS holds the only copy until the pull to the backup host runs, which makes the host's retention window the hard deadline rather than a comfortable margin. A review reads whatever survived and reports confidently on it either way, so settle this before the first review is expected to mean anything.
+**The off-host copy of the access log exists, and the schedule that maintains it is younger than the copy.** The pull to the backup host is installed as a `systemd` timer running daily at 09:00 UTC, chosen to sit behind both producers on the VPS rather than beside them, and its first copy was made by hand rather than by the timer. Read the unit and its last run on the backup host rather than trusting this paragraph, for the same reason retention is read from the VPS: a claim about a schedule is only worth what the machine says.
+
+**A rename on the VPS does not propagate to that copy, and nothing reports the divergence.** The pull passes no `--delete` for the logs, deliberately, since an append-only record must never be removed by a transfer. So a file **the VPS** renames, merges, or re-compresses after it has been pulled keeps its old name **on the backup host** forever, alongside the new one, and a count that walks that archive by filename double-counts the overlap. This has already happened once, to two archives whose names were a day ahead of their contents. **Read a date from a line's `StartUTC` rather than from the filename that holds it**, and treat a rename on the VPS as something the channel has to carry, because no transfer will.
 
 ## Who Owns What
 

checks/README.md

@@ -84,6 +84,18 @@ A count is all the check can observe, and two causes reach each direction: it ri
 
 **Both directions read absolute references as well as relative ones.** Hugo writes an absolute URL wherever a template resolves one against the base, which the entry-cover image on every list page does. Reading only rooted paths made those files look linked from nowhere while they were being displayed, and left a broken one unchecked in the other direction. The origin is read from the home page's canonical link rather than assumed, since staging and production build with different base URLs and a hardcoded host would check one environment's output against another's. No canonical link is a hard failure, because a guessed origin inflates the orphan count by exactly the pages that use one.
 
+## The robots check, and the one thing no gate here can do
+
+Hugo emits no `robots.txt` unless `enableRobotsTXT` is set, so this site served none until the flag was turned on, and the old platform serves one. The `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first full day of traffic, every request for `sitemap.xml` came from `curl` and none from a crawler, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted.
+
+**The `Sitemap:` line is the one place this gate compares an origin rather than joining one.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments. Other absolute URLs are read here, the home page's canonical link and the absolute asset references described above among them, but they are read to resolve a reference rather than to check one host against another.
+
+**What that comparison proves is internal consistency, and it is worth being exact about the limit.** The advertised origin must match the one read from the home page's canonical link, and both are derived from the same `baseURL`, so they agree whenever the build is coherent — including when `baseURL` held the wrong value for the environment being deployed to. **Nothing inside the artifact can detect that**, which is why the check belongs on the side that knows which host it is serving: the VPS side reads the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml` and reports the counts either way. A build baked with the wrong host still passes all 1,245 URLs here, and it passes this too.
+
+What the comparison does catch is an origin that was **written rather than derived** — a committed `static/robots.txt` shadowing the template is the way that happens, and pasting the old platform's `.com` sitemap line into one is the specific mistake it would catch — along with a sitemap advertised but never built.
+
+`/robots.txt/`, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolves through `slugs.map` like any other one-segment legacy URL, and the generator special-cases it to the real file rather than to the home page. `/osd.xml/` stays pointed at the home page deliberately: it was the old platform's OpenSearch description and this site emits no such file.
+
 ## The gallery check, which no direction above can reach
 
 Every check above reasons about a URL: whether it renders, whether it resolves, whether anything points at it. Content misplaced **inside** a gallery satisfies all of that. The file exists, the reference resolves, and something links it, so the media surface is green in both directions while the page is laid out wrong. The defect is one of structure, which is why three variants of it survived the conversion and every gate since.

checks/build-redirects.py

@@ -19,6 +19,15 @@
 # The WordPress importer registers the full slug, and both answer, so both are mapped.
 BLOGGER_SLUG_LIMIT = 40
 
+# A one-segment URL naming a file the site serves at the root, rather than a page slug.
+# The resolver below reads these as unresolvable attachment slugs and sends them to the home
+# page, which is the right answer for a slug nothing claims and the wrong one for a file that
+# exists: /robots.txt/ should reach /robots.txt. Named here rather than hand-edited into the
+# generated map, because the map is rewritten from the capture and a hand edit does not survive.
+# /osd.xml/ stays out deliberately. It was the old platform's OpenSearch description and this
+# site emits no such file, so the home page remains the honest destination for it.
+WELL_KNOWN = {"/robots.txt/": "/robots.txt"}
+
 
 def text(el, path):
  node = el.find(path, NS)
@@ -179,7 +188,9 @@ def main(argv):
  resolved, via_parent, via_media, orphan = [], 0, 0, []
  for u in needed:
  slug = u.strip("/")
- if slug in by_slug:
+ if u in WELL_KNOWN:
+ resolved.append((u, WELL_KNOWN[u]))
+ elif slug in by_slug:
  resolved.append((u, by_slug[slug]))
  via_parent += 1
  elif slug.lower() in file_to_post:

checks/check-url-parity.py

@@ -41,8 +41,15 @@
 # Every check above returns a list, and the shared summary called all of them "missing". That is
 # what a URL that did not build is, and it is not what a stray node inside a gallery is: those are
 # present, which is the whole complaint. The default stays "missing" so a check added later reads
-# the way the older ones do unless it says otherwise.
-FAILURE_NOUN = {"gallery": "stray nodes"}
+# the way the older ones do unless it says otherwise, and it needs no pair because it is already
+# count-neutral. A count is always printed beside the noun, so the pair is (singular, plural) and
+# "1 stray nodes" was the reason for making it a pair rather than a string.
+FAILURE_NOUN = {"gallery": ("stray node", "stray nodes"), "robots": ("problem", "problems")}
+
+
+def failure_noun(label, count):
+ forms = FAILURE_NOUN.get(label, ("missing", "missing"))
+ return forms[0] if count == 1 else forms[1]
 
 
 def load(name):
@@ -92,6 +99,65 @@ def check_media(public):
  return missing
 
 
+def check_robots(public):
+ """Check that robots.txt exists and advertises this build's own sitemap.
+
+ Four failures, one gate: the file absent, no Sitemap: line, a line naming another origin, and a
+ line advertising a sitemap that was not built. Hugo emits no robots.txt unless enableRobotsTXT
+ is set, which is why this site served none until that flag was turned on, and the file's only
+ load-bearing line points at the sitemap a crawler is otherwise unlikely to find, being told
+ rather than guessing.
+
+ Every contract list is path-only and check-live-urls.sh joins whatever base URL it is handed, so
+ this is the only place the gate *compares* an origin rather than joining one. Other absolute
+ URLs are read here - the home page's canonical link, and absolute asset references - but they
+ are read to resolve a reference rather than to check a host against another.
+
+ What the comparison proves is internal consistency: the advertised origin matches the one the
+ canonical link carries. Both come from baseURL, so this cannot tell that baseURL was the wrong
+ value for the environment being deployed to - nothing in the artifact can, which is why that
+ check belongs on the side that knows which host it is serving. It does catch an origin that was
+ written rather than derived, a static robots.txt shadowing the template being the way that
+ happens, and it catches a sitemap advertised but not built.
+ """
+ robots = public / "robots.txt"
+ if not robots.is_file():
+ # Naming one cause as the cause sends a reader to check a setting that is already correct.
+ # enableRobotsTXT is the likely one and a partial build or the wrong output directory reach
+ # the same state, which is the same reason the orphan messages name both of their causes.
+ print("robots : missing")
+ return [
+ f"{robots} does not exist - likely enableRobotsTXT is unset in hugo.yaml, "
+ "though a partial build or the wrong output directory look identical here"
+ ]
+
+ origin = site_origin(public)
+ # errors="replace" rather than strict, or invalid UTF-8 raises out of the whole parity run and a
+ # gate that exists to report a bad robots.txt stack-traces on one instead. A committed
+ # static/robots.txt is the file most likely to carry it, and it is the case this check is for.
+ text = robots.read_text(encoding="utf-8", errors="replace")
+ advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", text)
+ if not advertised:
+ print("robots : built, no Sitemap line")
+ return ["robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided"]
+
+ wrong = [u for u in advertised if not u.startswith(origin + "/")]
+ if wrong:
+ print(f"robots : built, {len(wrong)} Sitemap line(s) naming another origin")
+ return [f"{u} (this build's origin is {origin})" for u in wrong]
+
+ # The summary is printed after the last assertion rather than before it, or the missing-sitemap
+ # case reads as a pass on the line above its own failure. Every branch here prints exactly once.
+ unbuilt = [u for u in advertised if not (public / u[len(origin) + 1 :]).is_file()]
+ if unbuilt:
+ print(f"robots : built, {len(unbuilt)} advertised sitemap(s) not built")
+ return [f"{u} (advertised, but {u[len(origin) + 1:]} was not built)" for u in unbuilt]
+
+ # Named rather than counted, since there is one today and the line is the thing being checked.
+ print(f"robots : built, advertising {advertised[0]}")
+ return []
+
+
 def site_origin(public):
  """The site's own scheme and host, read from the artifact rather than assumed.
 
@@ -307,6 +373,7 @@ def main(argv):
  ("assets", check_assets(public, refs)),
  ("orphans", check_orphans(public, refs)),
  ("gallery", check_galleries(public)),
+ ("robots", check_robots(public)),
  ):
  if found:
  failures.append((label, found))
@@ -317,7 +384,7 @@ def main(argv):
 
  print()
  for label, found in failures:
- print(f"FAIL {label}: {len(found)} {FAILURE_NOUN.get(label, 'missing')}")
+ print(f"FAIL {label}: {len(found)} {failure_noun(label, len(found))}")
  for item in found[:20]:
  print(f" {item}")
  if len(found) > 20:

deploy/maps/slugs.map

@@ -100,7 +100,7 @@
 /jellyfish-lighting-ports-500x214-1/ /
 /osd.xml/ /
 /packet/ /2020/09/05/abandoning-greeneye-for-sense/
-/robots.txt/ /
+/robots.txt/ /robots.txt
 /sce_smartmeter/ /2020/09/05/abandoning-greeneye-for-sense/
 /ted-dashboard/ /2020/09/05/abandoning-greeneye-for-sense/
 /ted-graphing/ /2020/09/05/abandoning-greeneye-for-sense/

hugo.yaml

@@ -27,6 +27,15 @@ refLinksErrorLevel: ERROR
 # The deploy rebuilds the site rather than receiving it as an artifact, so the build stays reproducible.
 enableGitInfo: false
 
+# Off by default in Hugo, which is why this site answered 404 for a file the old platform served.
+# The Sitemap: line is the load-bearing part: across the interim hostname's first full day of
+# traffic, every request for sitemap.xml came from curl and none from a crawler, because a crawler
+# is told where a sitemap is rather than guessing, and the only thing telling them today is the
+# robots.txt the cutover deletes.
+# The theme's template derives that URL from the built baseURL, so it names .net during the
+# rehearsal and .com afterwards with no second edit to remember at the cutover.
+enableRobotsTXT: true
+
 params:
  description: My discoveries, frustrations, and delights
  defaultTheme: auto