Faceted discovery mints a combinatorial corpus the site never declared, and the demand ladder can't reach it

[harper-joseph]

What was measured

On a faceted retail deployment (2026-08-13), sampling the live target registry:

• 94.2% of catalog targets are discovered, not sitemap-listed — ~224k urls against ~38k sitemap locs, roughly 6x the declared corpus.
• The discovered set is a different SHAPE from the declared one. Facet-count distribution:

facets in CN	discovered	sitemap
1–3	8%	54%
4–6	66%	45%
7–9	25%	1%

Crawlers walk the facet navigation combinatorially (Size:2T+Gender:Boys+Color:Blue+Brand:…+AgeAppropriate:…), and every combination becomes a permanent target. The site's own sitemap tops out at 7 facets and is mostly ≤4 — it is declaring the shallow, valuable pages and deliberately omitting the deep tail.

Only 6.8% of these ever suppress. The origin self-canonicals them, so by every check the plugin makes they are legitimate pages. Nothing in the current pipeline distinguishes "page the site published" from "page a crawler assembled".

Why it is not self-limiting

Two mechanisms that look like they should bound this, and don't:

1. The demand ladder cannot reach them if the route's interval is at the ladder floor. render.demand's ladder only moves a target to a rung FASTER than the route's granted cadence — never slower. A route granted the fastest rung therefore has a one-entry ladder and no decision is possible. Measured before the fix, with a 6h catalog route against the default [6h, 12h, 24h, 48h] ladder:

{"decisions":246,"graded":116,"singleRung":130,"levels":{"6h":1,"12h":7,"24h":13,"48h":95}}

singleRung was over half of all decisions and levels contained no catalog at all; every one of 2,826 sampled discovered catalog targets had demandInterval unset. So the entire discovered corpus rendered at the route cadence forever, regardless of whether any bot had ever requested it.

That specific trap is worth surfacing in its own right — see "Ladder floor" below. It was worked around on that deployment by raising the route to 24h.

2. Nothing prunes on absence of demand. A discovered target is created on a single request and then renders on its cadence indefinitely. Suppression only fires on non-indexability, which these pages never trigger.

Why it matters

At a 6h cadence the discovered catalog corpus alone demanded ~75,000 renders/hr against a measured fleet ceiling of ~71,289/hr — i.e. the crawler-assembled tail oversubscribed the entire fleet, starving the declared corpus. Raising the route interval fixes the arithmetic but not the premise: the fleet is still spending renders on pages the site chose not to publish.

Possible directions

Not a proposal yet — these want discussion, and the right answer is likely site-contingent (which is the usual bar for a plugin option here):

1. A facet-depth / query-complexity cap at discovery. Cheap and effective on this shape — a cap of 4–5 segments would have excluded the 7–9 facet tail (25% of discovered, 1% of declared). Needs to be expressed generically, not as a CN-specific rule.
2. Sitemap-membership as a discovery gate for a route. "Prerender only what the sitemap declares; proxy the rest." Defensible for a route whose corpus is genuinely enumerable, and it makes the site's own declaration the authority. Pairs with Crawler-invented malformed URLs become immortal Targets: no validation at discovery + 403 exempt from suppression #81.
3. Prune on demonstrated absence of demand. The ladder already tracks visits; a target that reaches the slowest rung and still records no visit over N windows is a retirement candidate. This is the general fix and needs care: deleting is self-healing (rediscovery restores it), so the failure mode is churn rather than loss.
4. Let the ladder go SLOWER than the route grants. Currently the route interval is a hard floor on cadence. Allowing demotion past it would let a cold facet decay to 48h+ without changing the route, which is a smaller change than any of the above.

Ladder floor — the trap worth fixing regardless

Independent of the discovery question: a route whose renderInterval equals the fastest ladder rung silently disables the ladder for every url that route matches, and the only signal is the singleRung counter in a periodic histogram. That is very easy to configure by accident — the natural instinct is to set an aggressive route interval for an important route, which is exactly what switches the mechanism off.

Worth at least a config warning when a prerender route's interval is at or below render.demand.ladder[0] while render.demand.enabled is true.

Refs #81, #84