One-liner
The /docs/:path*.mdx rewrite (in next.config.mjs, for LLM/agent consumption) serves the full text of every page as text/markdown, 200, with no X-Robots-Tag and no canonical link. That is a second copy of all 403 pages at a crawlable URL.
Measured
$ curl -sI http://localhost:3999/docs/data-modeling/objects.mdx
content-type: text/markdown
(no x-robots-tag)
$ curl -s -o /dev/null -w "%{http_code} %{size_download}\n" http://localhost:3999/docs/data-modeling/objects.mdx
200 28080
Expected
Keep the endpoints — they are the point of the llms.txt / .mdx design and they are how agents read the docs. Just tell search engines which copy is canonical:
- add
X-Robots-Tag: noindex to the .mdx responses (and to /llms.txt, /llms-full.txt), or an HTTP Link: <https://objectstack.ai/docs/…>; rel="canonical" header pointing at the HTML page — pick one and say why in the PR; - name
llms.txt in robots.txt so agent crawlers find it deliberately (coordinate with the robots/sitemap card, which owns that file).
Note /llms-full.txt is currently 8 MB — worth a line in the PR on whether that should be split or streamed, but do not expand scope into rewriting it.
Acceptance
Source
Found in an SEO review of the docs site (apps/docs) run on 2026-08-25, measured against the local dev server and against production. The canonical origin is https://objectstack.ai — maintainer ruling recorded in #10659:
这个仓的文档站规范 URL 是 https://objectstack.ai
One-liner
The
/docs/:path*.mdxrewrite (innext.config.mjs, for LLM/agent consumption) serves the full text of every page astext/markdown, 200, with noX-Robots-Tagand no canonical link. That is a second copy of all 403 pages at a crawlable URL.Measured
Expected
Keep the endpoints — they are the point of the
llms.txt/.mdxdesign and they are how agents read the docs. Just tell search engines which copy is canonical:X-Robots-Tag: noindexto the.mdxresponses (and to/llms.txt,/llms-full.txt), or an HTTPLink: <https://objectstack.ai/docs/…>; rel="canonical"header pointing at the HTML page — pick one and say why in the PR;llms.txtinrobots.txtso agent crawlers find it deliberately (coordinate with the robots/sitemap card, which owns that file).Note
/llms-full.txtis currently 8 MB — worth a line in the PR on whether that should be split or streamed, but do not expand scope into rewriting it.Acceptance
curl -sI /docs/<page>.mdxshows the chosen directiverobots.txtnames both the sitemap andllms.txtSource
Found in an SEO review of the docs site (
apps/docs) run on 2026-08-25, measured against the local dev server and against production. The canonical origin ishttps://objectstack.ai— maintainer ruling recorded in #10659: