epic(docs-site): the site is technically un-indexable — fix robots/sitemap/canonical/OG first, then the keyword shape

[os-zhuang]

Why this epic exists

The docs site at https://objectstack.ai is, as of 2026-08-25, technically un-indexable in the ways that matter: there is no sitemap and no robots.txt (both paths answer 200 with the homepage HTML), no canonical link on any page, and no metadataBase. 403 doc pages exist with nothing pointing a crawler at them. On top of that the on-page keyword signal is thin — half the pages ship two <h1>s, page titles have a median length of 14 characters, and half the meta descriptions are too short to be used as snippets.

The order matters: the P0 lane is the precondition. Rewriting 403 titles while /sitemap.xml returns HTML buys nothing.

Lane / territory (epic PM declaration)

• Epic PM session: f9f0958b-ab68-46cc-801c-216aa7ee2107
• Declared file territory:
  • apps/docs/** — app routes, proxy.ts, lib/, components/, public/
  • content/docs/** — frontmatter title / description, body headings
  • scripts/check-*.mjs + the Lint & Repo Gates step list — only for the new doc-shape gates two sub-issues ask for
• Not in territory: packages/**, anything under docs/adr/**, .claude/**.

Sub-issues

P0 — indexability (do these first; #12234 unblocks #12235)

• docs site: /robots.txt and /sitemap.xml do not exist — both answer 200 with the homepage HTML #12232 — /robots.txt and /sitemap.xml do not exist; both answer 200 with the homepage
• docs site: any single-segment path containing a dot renders the homepage with 200 (soft-404 class) #12233 — any single-segment dotted path renders the homepage with 200 (soft-404 class)
• docs site: no metadataBase and no canonical link on any page #12234 — no metadataBase, no canonical link on any page
• docs site: Open Graph images are generated for all 403 pages but no page references them #12235 — OG images generated for all 403 pages, referenced by none

P1 — keyword shape (maintainer-voice; PRs stay open for review)

• docs content: 205 pages render two <h1> — 129 of them the same text twice #12236 — 205 pages render two <h1>, 129 the same text twice
• docs content: page titles carry no search intent — median 14 characters, 325 of 403 under 20 #12237 — page titles carry no search intent (median 14 chars)
• docs content: meta descriptions too thin to serve as snippets — median 46 chars, 228 of 403 under 70 #12238 — descriptions too thin to serve as snippets (median 46 chars)
• docs site: homepage <title> and description are not keyword-shaped (description is 615 characters) #12239 — homepage title/description not keyword-shaped (615-char description)

P2 — structured data, crawl hygiene, weight

• docs site: no structured data (JSON-LD) anywhere #12240 — no JSON-LD anywhere (SoftwareApplication / TechArticle / BreadcrumbList)
• docs site: /docs/**.mdx serves an indexable parallel copy of every page with no robots directive #12241 — /docs/**.mdx serves an indexable parallel copy with no robots directive
• docs site: homepage hero cover is a 406 KB PNG served at ~1024 px #12242 — homepage hero cover is a 406 KB PNG (LCP candidate)

Shared facts every sub-issue inherits

• Canonical origin is https://objectstack.ai — maintainer ruling recorded in fix(docs): converge the docs host on the canonical origin, gate included #10659:

  这个仓的文档站规范 URL 是 https://objectstack.ai

• The origin belongs in one shared constant that the sitemap, metadataBase, canonical links and JSON-LD all read. Three sub-issues need it; the first one to land creates it and the rest import it.
• Locales: lib/i18n.ts is English-only (languages: ['en'], hideLocale: 'default-locale'). No hreflang work is in scope, and /en/docs correctly 307s to /docs today — do not break that.
• llms.txt / llms-full.txt / .mdx endpoints are a deliberate feature for agent readers. ⛔ Do not remove or gate them; the only change any sub-issue may make is a robots directive.
• Two sub-issues (docs content: page titles carry no search intent — median 14 characters, 325 of 403 under 20 #12237, docs content: meta descriptions too thin to serve as snippets — median 46 chars, 228 of 403 under 70 #12238) rewrite maintainer-voice copy across 403 files: the PR must carry the rule plus the full before/after table in its body, and must be left for maintainer review rather than self-merged.

Definition of done for the epic

• curl -sI https://objectstack.ai/robots.txt → 200 text/plain, naming the sitemap
• /sitemap.xml is XML and lists every doc page
• a random dotted path 404s
• every page type emits one absolute canonical and a working OG card
• no page renders more than one <h1>
• title/description length gates run in Lint & Repo Gates

Source

Filed from an SEO review of apps/docs on 2026-08-25 — every claim above was measured against the local dev server and against production, with the commands recorded in the individual sub-issues.