fix: add custom robots.txt with explicit AI-crawler Allow rules

[mpot37]

Summary

Fixes the "AI crawlers explicitly allowed (GPTBot / ClaudeBot / etc.)" severe finding from the check.outrun.at GEO audit, tracked internally as Linear REQ-271.

Confirmed with Mintlify support: a robots.txt at the project root overrides the auto-generated one, served as-is (it does not carry over Mintlify's default Content-Signal directive). So this file reproduces every directive currently being auto-generated in production:

User-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Disallow: /cdn-cgi/
Allow: /_next/image
Disallow: /_next/
Sitemap: https://docs.request.network/sitemap.xml

...and adds explicit Allow: / blocks for GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, and Google-Extended on top.

Test plan

• After merge + deploy, re-run npx tsx scripts/check-geo-signals.ts and confirm the robots.txt-related PLATFORM-GATED check can be upgraded (the script currently expects this to stay platform-gated per REQ-271 — worth updating once this lands)
• Diffed against the live https://docs.request.network/robots.txt to confirm no existing directive is dropped

[greptile-apps]

Greptile Summary

The PR adds a custom root-level crawler policy that preserves the existing site-wide directives while explicitly allowing six AI crawlers.

• Repeats the content-use signal and internal-path rules in every named crawler group.
• Publishes the existing sitemap location.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the named crawler groups now preserve the wildcard directives identified in the previous review.

Important Files Changed

Filename	Overview
robots.txt	Adds explicit AI-crawler groups and now correctly carries the wildcard group’s content signal and path-specific rules into each named group.

_{Reviews (2): Last reviewed commit: "fix: repeat wildcard exclusions in named..." | Re-trigger Greptile}