docs(site): keep the agent-reader copies of every page out of the search index

[os-zhuang]

Fixes#12241

What was wrong

Every documentation page is reachable as machine-readable text at a crawlable URL with no robots directive of any kind. Measured against production before this change (2026-08-25, after the epic's P0 PRs deployed):

$ curl -sI https://objectstack.ai/docs/data-modeling/objects.mdx
HTTP/2 200
content-type: text/markdown
content-length: 28080
<- no x-robots-tag, no link
$ curl -sI https://objectstack.ai/docs/data-modeling/objects
HTTP/2 200
content-type: text/html; charset=utf-8
content-length: 440431
$ curl -sI https://objectstack.ai/llms.txt
HTTP/2 200
content-type: text/plain;charset=UTF-8
content-length: 53063
$ curl -sI https://objectstack.ai/llms-full.txt
HTTP/2 200
content-type: text/plain;charset=UTF-8
content-length: 8052587
$ curl -s https://objectstack.ai/robots.txt
User-Agent: *
Allow: /
Sitemap: https://objectstack.ai/sitemap.xml

The card understated it by one. The rewrite's destination is itself a real app-router route, so it answers directly too — each page has two markdown URLs, not one:

$ curl -sI https://objectstack.ai/llms.mdx/docs/data-modeling/objects
HTTP/2 200
content-type: text/markdown
content-length: 28080 <- byte-identical to the .mdx URL above

After

Same commands against the app with this change (dev server, next dev, port 3987):

/docs/data-modeling/objects.mdx 200 text/markdown X-Robots-Tag: noindex
/llms.mdx/docs/data-modeling/objects 200 text/markdown X-Robots-Tag: noindex
/llms.txt 200 text/plain;charset=UTF-8 X-Robots-Tag: noindex
/llms-full.txt 200 text/plain;charset=UTF-8 X-Robots-Tag: noindex
/docs/data-modeling/objects 200 text/html; charset=utf-8 (no X-Robots-Tag — unchanged)
$ curl -s http://localhost:3987/robots.txt
User-Agent: *
Allow: /
Allow: /llms.txt
Allow: /llms-full.txt
Sitemap: https://objectstack.ai/sitemap.xml

Body still served in full: /docs/data-modeling/objects.mdx returns 200 with 28042 bytes, first line # Object Metadata. Nothing is gated, redirected, renamed or content-negotiated — the endpoints are a deliberate feature and they behave exactly as before to anyone who fetches them.

Which directive, and why — noindex, not a canonical Link header

I built and measured both candidates before choosing, because "which one actually reaches the response" was the stated tiebreak. Both do; Next interpolates route params into header values, so the canonical variant produced a correct per-page target (Link header carrying https://objectstack.ai/docs/data-modeling/objects with rel="canonical" on the .mdx response). Mechanics did not decide it — substance did, on two counts. Coverage: only the per-page markdown has an HTML twin to canonicalise to. /llms.txt is an index of all 400+ pages and /llms-full.txt is all of them concatenated; neither is a duplicate of any single HTML URL, so a canonical header can say nothing honest about them and a canonical-based fix would leave the two largest parallel copies on the site — including an 8 MB one — with no directive at all, which is the exact gap this card exists to close. Strength: a canonical link is a hint a search engine weighs against other signals and may overrule, whereas noindex is a directive; what is wanted here is the strong form — keep the copy fetchable, keep it out of results. The two are also not additive: pairing noindex with a canonical pointing elsewhere is self-contradictory, so exactly one belongs. One claim I expected to make and had to drop after testing it: I assumed next.config.mjs (ESM JavaScript) could not import SITE_ORIGIN from lib/site.ts, which would have forced a second spelling of the origin. It imports fine under Next 16's config loader — verified by printing the value at config load. So that was not a reason, and noindex wins on coverage and strength alone.

The two files are one mechanism, and they point in opposite directions

robots.ts shipped in #12253 with no Disallow lines and left the question here. The answer is that it stays that way: a Disallow would defeat the header. A crawler told not to fetch a URL never sees the X-Robots-Tag on it, and a disallowed URL remains eligible for URL-only indexing from an inbound link — strictly worse than today. Allow the crawl, refuse the index. Both docblocks say so at the point where someone would otherwise add the wrong thing.

Allow: /llms.txt / Allow: /llms-full.txt are redundant under Allow: / and change no crawler's behaviour — that is deliberate and it is the whole job. robots.txt has exactly one discovery directive, Sitemap:, and these are not sitemaps; naming the paths in the file agents fetch first is the available way to make them findable on purpose rather than by guessing at a convention.

On /llms-full.txt being 8 MB (note, not scope)

Measured at 8,052,587 bytes today and growing linearly with the page count. My read is that it should eventually be split, but the trigger should be a measured consumer failure rather than the number itself: the file's whole value is being one fetch, and splitting it trades that for an index plus N requests, which is what /llms.txt plus the per-page .mdx URLs already provide. The thing to watch is that many agent HTTP clients cap response bodies well under 8 MB and truncate silently mid-document — if that starts happening the split is forced, and the natural seam is the existing top-level docs sections. Not touched here.

Verification

Gate union re-run at the final commit 6f8244cef (after merging origin/main at 5ce5f8c12), each verdict quoted from the gate's own output:

gate	verdict
check:docs-locale-catch-all	✓ 1 top-level dynamic segment(s), 1 guarded; dotted paths bypass proxy.ts: true
check:page-declaration-shape	OK — 34 page entries across 2185 sources … all reach the kernel through a discoverable declaration
check:published-files	✓ 69 publishable package(s) of 78 workspace member(s) declare a files whitelist …
check:test-source-alias	OK — 72 packages with tests scanned
check:type-source-resolution	OK — 93 tsc program(s) across 77 packages scanned
check:nul-bytes	OK (scanned 6815 text file(s) … no raw ASCII control bytes)
pnpm --filter @objectstack/docs typecheck	exit 0 (tsc --noEmit, silent)

Families derived by node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack, re-derived at the final commit; the list did not change.

Two honest limits on the above:

• next.config.mjs is not in the typecheck program.tsc --noEmit --listFiles in apps/docs enumerates 1239 files; app/robots.ts appears in it, next.config.mjs does not (it is .mjs). So typecheck says nothing about the file carrying the headers — the evidence for that file is the running server above, which loaded the config (✓ Running next.config.mjs took 135ms) and served the headers.
• Repo-wide pnpm lint was narrowed, deliberately, and the narrowing is declared. ESLint was run on the two edited files only: --format json reports exactly 2 results, both errorCount=0 warningCount=0, neither ignored. The narrowing is safe to a specific measured fact rather than an assumption: this repo runs one eslint.config.mjs which never enables type-aware linting for any file — eslint.config.mjs's own docblock records that (no parserOptions.project, no typed @typescript-eslint rules, with a planted-positive-control measurement behind it). With no cross-file type program, a two-file diff cannot move any untouched file's verdict. CI runs the full farm regardless.
• The shared verify lock was NOT taken.scripts/pm/os-verify-lock.sh reported UNLOCKED (declared) · no usable flock on this host, so the shared verify lock was NEVER taken and NOTHING was serialized on both runs, and instructs that this be declared here. Every command above still went through the wrapper.

Not done here

No changeset: docs-site only, publishes nothing — skip-changeset applied.

The card's second acceptance box reads "the HTML page is unaffected and still canonicalises to itself". The first half holds and is measured above. The second half is not true on main yet and is not this card's to fix: the HTML page emits no link rel=canonical today, in production or locally — that is #12234, in flight in the same round. This PR adds nothing to the HTML page, so whatever #12234 lands is unaffected by it.

Generated by Claude Code