Gate request-path discovery (new-Target creation) on bot name

[harper-joseph]

Problem

Discovery on the request path is a privilege currently granted to every user agent the CDN forwards. Any request that reaches the bot handler on a prerender route and misses the cache with a 200 HTML origin response creates a Target (maybeSchedule → handlePageScheduling in src/http_handlers/bot_request.js) — and a Target is a standing claim on render capacity, ~2 renders/day, forever.

That privilege is what #81 exploits by accident: third-party crawlers with broken link extractors (AhrefsBot alone accounted for 81.5% of the junk misses; DotBot and friends the rest) invent malformed URLs from our rendered HTML, the origin soft-404s them with a 200, and each becomes an immortal Target. The crawlers we actually serve SEO for are a small, well-known set; the long tail of scrapers gets the same power to grow our corpus and spends it almost entirely on garbage.

We already compute a stable identity for every request — request.botName via getBotName() (registry match, else derived self-identification, else 'other') — but scheduling ignores it today.

Measured funnel (production deployment, 24h audit-log sample, 2026-08-12)

Classifying every Target write in a 24h window by record shape (the audit log includes replicated writes, so this is cluster-wide):

Path	New Targets / day
Request-path discovery	27,219 distinct URLs (27,222 writes — replication-race duplicates are negligible)
Sitemap run	~3,300
Deletes (all causes)	~790

Discovery creates ~8× more Targets than the sitemap, against a corpus of only ~400k URLs — ~6%/day growth pressure, offset almost not at all by deletion.

Composition, and what it costs (measured against the live origin, 2026-08-12)

Alongside legitimately new product/facet pages, a large share of what discovery creates is variants of pages the corpus already has: the same catalog facet URL with + / %2B / %20 separator encodings, the same facets in a different order, the same page under a different (or truncated) slug, double-percent-encoded mutations, plus the outright malformed inventions of #81.

Probing the origin directly settles what those variants are worth. Every re-spelling returns the same page — same product set, same title — and declares a <link rel="canonical"> pointing at the sitemap spelling. The slug is decorative; the origin ignores it entirely:

variant of one catalog facet URL	origin	canonical it declares	our cache key
control (sitemap loc)	38 products	self	—
facet values reordered	38 products	→ control	different
wrong / truncated slug	38 products	→ control	different
separators as %2B	38 products	→ control	different
separators as %252B (double-encoded)	0 products, junk page	→ a fabricated junk URL	different
separators as %20	38 products	→ control	different

So each variant is a distinct Target for a page we already render — but not an immortal one, which corrects this issue's original cost claim. A canonical pointing elsewhere is a non-indexable verdict, so the variant is suppressed after one render, rechecked at render.suppression.recheckInterval (7d), and deleted after maxStrikes (4) — roughly 5 renders per 28 days, ~0.18/day, not ~2/day forever. The exception was the + ↔ %20 re-spelling in either direction (production's real shape is a space written as +: Silhouette:Bath+Rugs for Bath%20Rugs), which read as self-canonical and did render forever; #90 closes that.

Two things this measurement rules out:

• The encoding classes cannot be fixed in the cache key. + and %2B are interchangeable only in separator position. In value position they are different values (Brand:ACME%2BCO returns that brand's 52 products; Brand:ACME+CO returns an empty page canonicalizing elsewhere), so folding them would serve one facet's HTML under another facet's URL — the canonical half is also the origin-fetch/navigation URL. Re-keying would also orphan every cached page in the corpus.
• The double-encoded shapes feed Crawler-invented malformed URLs become immortal Targets: no validation at discovery + 403 exempt from suppression #81. The origin answers a %252B mutation with a 200 whose canonical is a newly fabricated junk URL derived from the mutation — so the origin manufactures fresh junk URLs from junk input, which crawlers then follow. That argues for rejecting %25xx escapes at discovery (in Crawler-invented malformed URLs become immortal Targets: no validation at discovery + 403 exempt from suppression #81's validation), and against ever adopting a page's canonical as a new Target.

What the corpus already does about it (production, 24h to 2026-08-12T22:52Z)

Suppression is firing at scale — from the four nodes' hdb.log, cluster-wide (two lines per URL, desktop + mobile):

reason	24h
canonical-mismatch	13,036
noindex	13,120
http-error	2,101

~6.5k distinct URLs/day get a canonical verdict on their first render and die within ~28 days. On one node, 85% of those distinct URLs are /product/prd-N/product.jsp — the placeholder-slug class.

Sizing the encoding class specifically, from a 30-minute read_audit_log window on render_service.Target (615 distinct discovery-created URLs, 93% catalog): 4.7% of discovered catalog URLs carry a space-written-as-+ value (Silhouette:Bath+Rugs where the canonical spells Bath%20Rugs), and 1.6% use %2B between facets. Probing all 27 signature URLs against the origin: 19 canonicalize to the %20 spelling and were, until #90, permanently self-canonical duplicate targets; 6 were already suppressed; 2 declare no canonical at all and are invisible to any canonical logic.

One caveat that matters for this issue's framing: only 5 of the 25 differing canonicals are in a sitemap. Suppressing the variant therefore does not hand coverage to a twin we already render — usually neither spelling is prerendered afterward and bots take the origin proxy. That is correct (the origin disowns the spelling), but it means canonical-based retirement shrinks waste, not the discovery inflow itself.

What remains, and what this issue is about, is the volume: ~27k new Targets/day, each costing at least a render and a suppression cycle before it dies, from crawlers we serve no SEO for.

Proposal

Gate Target creation (not serving) on bot name: a configurable allowlist consulted in maybeSchedule/handlePageScheduling against the already-computed request.botName.

• A non-allowlisted bot's request is otherwise untouched: it still serves from cache, still gets the origin proxy on a miss, still records analytics. It just can't mint a new Target.
• Existing Targets are unaffected — this gates creation only. Renders keep happening on schedule regardless of who visits.
• The sitemap ingestion path is unaffected (it's the trusted corpus source and sets sitemapUrl).

Config sketch (names open to bikeshedding — probably a discovery group so future validation from #81 has a home):

discovery:
 bots: ['Googlebot', 'Googlebot-Image', 'Bingbot', 'DuckDuckBot', 'Applebot', …] # names as produced by getBotName
 # default: '*' (current behavior) for backward compatibility; deployments opt in

Matching should be against the output of getBotName — registry display names — so the registry stays the single place a crawler's identity is defined. That also means derived names can be listed before they're promoted to the registry.

Decisions to make

1. Default: '*' (no behavior change on upgrade) vs. a curated search+AI default. Lean '*' — silently shrinking discovery on upgrade is a corpus-shape change an operator should make deliberately.
2. 'other' and derived names: excluded unless explicitly listed, presumably — but confirm we're comfortable that a misparsed UA of a bot we care about falls out of discovery (analytics still records it, so it's visible).
3. Demand ladder input (recordVisit): today every bot's visit feeds demand promotion for existing targets. Same allowlist, a separate one, or leave untouched? Leaving it untouched is defensible (it only re-paces pages we already own) but worth an explicit call.

Non-goals / caveats

• Not a security boundary. UA strings are trivially spoofable; upstream bot management owns authenticity. This is a junk-quality filter that shrinks who can grow the corpus.
• Not a substitute for Crawler-invented malformed URLs become immortal Targets: no validation at discovery + 403 exempt from suppression #81. Allowlisted crawlers can also follow invented URLs (Googlebot crawls links wherever it finds them), so discovery-time URL validation and letting 403 suppress under the never-rendered predicate are still needed. This issue removes the dominant funnel; Crawler-invented malformed URLs become immortal Targets: no validation at discovery + 403 exempt from suppression #81 removes the immortality.
• Not blocked by fix(browser): read the canonical against the cache key, not the document (v1.17.0) #90. That PR stops the one variant class that rendered forever; the remaining cost of a variant is a render plus a suppression cycle, which this gate is what actually avoids.

Sizing

Small: thread request.botName (or the request) into maybeSchedule, one allowlist check, one config schema entry (configSchema.js), tests. No schema/data migration.