Trailing-slash folding serves pages the origin refuses (PDP 403, /catalog.jsp/ soft 404)

[harper-joseph]

Problem

canonicalizeUrl drops a trailing slash on any non-root path, so /a/ and /a become one cache
key. We then fetch, render and cache the slash-less form and serve those bytes for both — which
means we answer for a URL the origin never approved.

Measured against the live origin (bypass token), most recently 2026-08-13 across two passes ~20
minutes apart with two distinct URLs per shape:

URL	origin
/product/prd-<id>/<slug>.jsp	200, 1 product
/product/prd-<id>/<slug>.jsp/	403
/catalog.jsp?CN=<facets>	200, 37 products
/catalog.jsp/?CN=<facets>	200 — but the 404 page (canonical /feature/404.jsp, 0 products)
/catalog/<slug>.jsp?CN=<facets>	200, 37 products
/catalog/<slug>.jsp/?CN=<facets>	200, identical to the slash-less form

So on two of the four routed shapes the slashed URL is not the same resource. The product route
is a hard refusal (a WAF verdict on URL shape — the same class of rule as #81, not an auth failure).
/catalog.jsp/ answers with the site's 404 page under a 200 status, which is the more dangerous
of the two: a soft 404 is a 200, and a 200 is exactly what isPrerenderCandidate accepts. Only
/catalog/<slug>.jsp/ is genuinely equivalent.

A crawler that requests either non-equivalent form gets served our cached snapshot of a different
URL — one the origin answers 403 or 404 on.

Why this rule is different from the others

We serve on behalf of the origin, so every normalization rule is a claim about the origin's URL
semantics, and the requirement is that origin(N(U)) be the same resource as origin(U) for every
URL we accept. Most of the rules in canonicalizeUrl are settled at the parser boundary —
dropping the fragment, upper-casing escape hex, decoding %3A/%2C/%40, sorting params — where
the decoder erases the distinction before any application logic runs. Those hold for every URL and
can be established once.

Trailing slash is not like that. Whether /a/ and /a name one resource is a per-resource
question, decided by routing and by edge rules, and no parser property implies it. It is the one
rule in the set that cannot be justified by argument — and the probes above show it is in fact false
for two route classes on this deployment, including one that the first round of measurement (which
only covered /product/ and /catalog/<slug>) recorded as harmless.

Resolution

cacheKey.trailingSlash: 'strip' | 'preserve' shipped in plugin v0.45.0 / browser v1.17.0 (#94),
implemented in both canonicalizeUrl copies, defaulting to strip so an upgrade changes nothing,
with the measured 403 in the option's own description and pinned by shared test vectors.

This deployment is set to preserve — paired PRs, to be deployed in one window:

• component: HarperFast/kohls-pr#69
• render fleet: HarperFast/render-service#72

Both sides must move together: the fleet's mirrored cacheKey block decides which URLs are the same
key.

Properties of the change:

• No migration. Stored keys are already slash-less, since every sitemap loc and every canonical
  the origin emits is slash-less. Nothing re-keys.
• Self-correcting. A slashed URL keys separately, is discovered only if the origin answers it
  with a 200 (so on the product route it never becomes a target at all), and any that do slip
  through render, take the 403, and retire as http-error.
• The right answer for the crawler. It gets what the origin actually says for the URL it asked
  for.

Open — the cost side is still unmeasured

How many slashed requests actually arrive (i.e. how many cache hits become origin proxies) is not
known, and could not be measured: HTTP request logging produces no output on these nodes. The
configured http.logging.path creates no http.log file, and neither hdb.log nor container stdout
carries request lines — so there is no per-request record to count. Sizing this needs request logging
working first, and that is worth fixing independently of this issue.

The exposure is bounded regardless: slashed URLs are a crawler artifact, appearing in neither the
sitemap nor the origin's canonicals.

Caveat

Dropping the fold slightly widens the discovery surface (/a/ becomes eligible to be its own target
where the origin does answer 200 for both). The /catalog.jsp/ soft 404 is exactly that shape — it
returns 200, so it can become a target that renders the 404 page — which makes this an argument for
pairing with the discovery-side validation in #81 / the bot-name gate in #84.

Refs #84, #92, #94

[harper-joseph]

Code lands in #94. cacheKey.trailingSlash: 'strip' | 'preserve' is implemented in both canonicalizeUrl copies, defaulting to strip so an upgrade changes nothing, with the measured 403 in the option's own description and pinned by shared test vectors.

What remains here is the decision, not the implementation: set preserve on this deployment (the origin 403s a slashed product URL while serving the slash-less form 200, so stripping answers on its behalf with a page it refused), and count slashed requests in the http log before/after to size the change. Closing once that config change is deployed.

[harper-joseph]

Body rewritten against a 2026-08-13 re-measurement (two passes ~20 min apart, two distinct URLs per shape). Two changes to what this issue said:

1. /catalog.jsp/ is also broken, and worse than the 403. The first round only probed /product/ and /catalog/<slug>, and recorded catalog as harmless. The fourth routed shape, /catalog.jsp/?CN=<facets>, returns the site's 404 page under a 200 (canonical /feature/404.jsp, 0 products). A soft 404 is a 200, and a 200 is exactly what isPrerenderCandidate accepts — so unlike the PDP 403 it can mint a target that renders the 404 page. Two of four routed shapes are non-equivalent, not one.
2. The cost side cannot currently be measured. HTTP request logging produces no output on these nodes — the configured http.logging.path creates no http.log, and neither hdb.log nor container stdout carries request lines. So "count slashed requests before/after" is blocked on fixing request logging, which is worth doing independently.

Set to preserve in HarperFast/kohls-pr#69 + HarperFast/render-service#72, to deploy in one window.