feat(docs): serve a real /robots.txt and /sitemap.xml

[os-zhuang]

Fixes#12232

Part of the docs-site indexability epic #12243 (P0 lane).

/robots.txt and /sitemap.xml had no route, so app/[lang]/page.tsx matched them
as lang = "robots.txt" / lang = "sitemap.xml" and answered 200 text/html with
the homepage. A crawler asking for crawl rules got a web page; a sitemap submitted to
Search Console would have failed to parse.

What changed — 3 new files, 217 lines, nothing edited

File	Purpose
apps/docs/lib/site.ts	the canonical origin, declared once
apps/docs/app/robots.ts	text/plain robots response naming the sitemap
apps/docs/app/sitemap.ts	every indexable URL, derived from source / blog

The shared origin constant — exactly what is exported

This card owns creating it. apps/docs/lib/site.ts exports two names and nothing
else:

exportconstSITE_ORIGIN='https://objectstack.ai';exportfunctionabsoluteUrl(path: string): string;

• SITE_ORIGIN — the origin only, no trailing slash. Maintainer ruling, recorded in
  fix(docs): converge the docs host on the canonical origin, gate included #10659, verbatim and untranslated:

  这个仓的文档站规范 URL 是 https://objectstack.ai

• absoluteUrl(path) — joins a site-relative path onto that origin.
  path must start with a slash; anything else throws at build time rather than
  quietly emitting a URL on the wrong host. new URL(path, SITE_ORIGIN) alone would
  hand an already-absolute https://elsewhere/... straight back, and a sitemap
  listing a foreign host is discarded wholesale rather than reported.

Notes for the two cards that import this next round:

• docs site: no metadataBase and no canonical link on any page #12234 (metadataBase) — Next wants a URL object, not a string. Write
  metadataBase: new URL(SITE_ORIGIN). That is the intended spelling, and it is
  documented in the file so a second origin literal never appears. A module-level
  shared URL instance is deliberately not exported: URL is mutable, so one
  consumer assigning .pathname would silently move the origin for every other.
• docs site: no structured data (JSON-LD) anywhere #12240 (JSON-LD) — use absoluteUrl() for every @id / url.
• No environment-variable override, deliberately. A preview deployment deriving its
  own origin would emit canonical links and a sitemap pointing at the preview host,
  which is exactly the duplicate-content signal a canonical link exists to suppress.
• No SITE_NAME / title / description constants: nothing in this card needs them,
  and inventing surface for a consumer that has not landed is how a shared module
  starts collecting things nobody reads.

lastModified: git committer date, not build time

One git log pass over both content collections, run once at build:

git -c diff.relative=false log --format=commit-date:%cI --name-only --no-renames \
-- content/docs content/blog

Cost, measured, not assumed: 1.0s over 11,231 commits for all 406 content files.
The suggested alternative in the card — one git log per file — would be 406 git
processes; this is one. So the cheap honest signal and the good one are the same
signal here, and there was no trade to make.

Why not build time: it restamps 400+ pages on every deploy, which tells a crawler the
whole site changed whenever anything did — a signal discounted precisely because it is
never false.

When the date cannot be known, the entry ships with no lastmod element at all
(no git directory; or a clone shallow enough that no commit in the window touched the
file). lastmod is optional in the sitemap protocol, so omitting it is the honest
answer, and substituting build time would reintroduce the exact lie the git date
exists to avoid. It does not degrade silently, either: the build prints a counted
warning naming the remedy — a shallow-clone note, or a per-page count with the first
three paths. On this build that warning count is 0.

PM dispatch assumptions — all four verified, none falsified

1. No robots.ts / sitemap.ts in apps/docs/app/ — confirmed on origin/main.
2. Production serves both as 200 text/html — re-measured today against
   https://objectstack.ai, unchanged:

   $ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://objectstack.ai/robots.txt
   200 text/html; charset=utf-8
   $ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://objectstack.ai/sitemap.xml
   200 text/html; charset=utf-8
   

3. source.getPages() is the right source — confirmed; no list is hand-maintained.
   Note getPages() with no argument lists every language (fumadocs LoaderOutput);
   English is the only one today, and a future locale belongs in the sitemap under its
   own prefixed URL, so it is left unfiltered on purpose.
4. A literal app/robots.ts beats the single-segment [lang] dynamic route —
   confirmed, in both dev and a production build. Measurements below. This is the
   assumption the PM asked to have tested; it holds, so docs site: any single-segment path containing a dot renders the homepage with 200 (soft-404 class) #12233 is not a blocker for
   this card. The converse also holds and is worth recording for docs site: any single-segment path containing a dot renders the homepage with 200 (soft-404 class) #12233: an unrelated
   dotted path still renders the homepage after this change, so this PR fixes exactly
   two paths and does not touch that defect.

Measurements

Dev server (next dev -p 38412)

$ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" http://localhost:38412/robots.txt
200 text/plain
$ curl -s http://localhost:38412/robots.txt
User-Agent: *
Allow: /
Sitemap: https://objectstack.ai/sitemap.xml
$ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" http://localhost:38412/sitemap.xml
200 application/xml

Production build (next build, exit 0)

Route table — both are ○ (Static) prerendered as static content, so the git log
runs in the build process and never in a request:

├ ○ /robots.txt
└ ○ /sitemap.xml

Production server (next start -p 38413)

$ curl -sI http://localhost:38413/robots.txt | head -5
HTTP/1.1 200 OK
x-nextjs-cache: HIT
cache-control: public, max-age=0, must-revalidate
content-type: text/plain
$ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" http://localhost:38413/sitemap.xml
200 application/xml

The sitemap's first line is an XML declaration, not HTML.

Acceptance criteria

Criterion	Result
robots.txt is 200 text/plain and the body names the sitemap	pass
sitemap.xml starts with an XML declaration, not HTML	pass — and content-type: application/xml
URL count = find content/docs -name '*.mdx' | wc -l + non-docs routes	408 = 403 + 5 (403 docs, plus /, /blog, and 3 blog posts)
no absolute URL names a host other than objectstack.ai	pass — grep -o over every loc element yields exactly one host

lastmod element count is 406 — every one of the 403 docs pages and 3 blog posts.
The 2 URLs without one are / and /blog, which have no MDX file behind them. Three
dates spot-checked against git log -1 --format=%cI on the source file: all three match
to the second.

Neighbouring routes, unchanged

Path	Before and after
/en/docs	307 to /docs — the epic asks that this not break; it does not
/llms.txt	200 text/plain
/docs/upgrading.mdx	200 text/markdown
an unrelated dotted path	200 text/html — #12233's defect, untouched

Scope: what robots.txt deliberately does NOT contain

User-agent: * / Allow: / and the Sitemap: line, and nothing else. No Disallow
for /api, /og, /docs/**.mdx or llms*.txt:

• crawl hygiene for the .mdx parallel copy is docs site: /docs/**.mdx serves an indexable parallel copy of every page with no robots directive #12241, and two PRs editing the
  same handful of lines in the same file is a conflict for no gain — that card can add
  its directive on top of this file;
• /og in particular must stay crawlable, or the OG cards docs site: Open Graph images are generated for all 403 pages but no page references them #12235 is about would
  stop rendering for the crawlers that fetch them.

No priority or changeFrequency on any entry: Google ignores both, and inventing
values for 408 URLs would put numbers into a machine-readable surface that nothing
measured.

Verification

All at e02a77cc7, the final commit on this branch.

Command	Result
pnpm --filter '@objectstack/docs^...' build	VERDICT command-exit 0
pnpm --filter @objectstack/docs typecheck	VERDICT command-exit 0
next build (apps/docs)	exit 0
pnpm lint (repo-wide eslint . --no-inline-config)	VERDICT command-exit 0, 28s, no findings — the full farm scan, not a narrowed one
pnpm check:published-files	✓ ... 69 publishable package(s) ...
pnpm check:test-source-alias	check-test-source-alias OK — 72 packages with tests scanned
pnpm check:type-source-resolution	check-type-source-resolution OK — 93 tsc program(s) across 77 packages scanned
pnpm check:nul-bytes	check-nul-bytes: OK (scanned 6772 text file(s) ... no raw ASCII control bytes)

The first three gate families are the ones
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack derives for
these three paths, re-derived from a tree at origin/main after the branch was
fast-forwarded (the first derivation ran against a stale tree and said so).
check:nul-bytes is added because the diff is an edit.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock (the lock is
declared Linux-only; a stock macOS does not ship util-linux). Every wrapped command
above therefore ran directly, without the lock — a declared narrowing, not a silent
one. No serialization guarantee held for those runs.

Landing

Docs-site only, no published package changes, so no changeset — the
skip-changeset label carries that, applied to this PR.

---

Generated by Claude Code