chore: block search engine indexing with noindex signals

[rodrigopavezi]

Why

We need request-scan out of Google and other search engines.

What

• src/app/layout.tsx — robots: { index: false, follow: false, googleBot: { … } } in the root metadata: every route inherits <meta name="robots" content="noindex, nofollow"> (covers the four "use client" routes that can't export metadata themselves).
• next.config.mjs — global X-Robots-Tag: noindex, nofollow response header on /:path* (covers non-HTML responses a meta tag can't reach; served for real since the app runs next start).
• src/app/robots.ts (new) — deliberately allow-all. To get already-indexed pages removed, crawlers must be able to fetch pages so they can see the noindex signal (Google's documented removal process). A Disallow: / would hide the noindex and leave stale entries stuck in the index. Once the site has dropped out of the index (weeks), a follow-up can flip this to Disallow to also cut crawl traffic.
• .github/workflows/build-and-lint.yml — the smoke test now also asserts the X-Robots-Tag header and /robots.txt are served, so de-indexing is regression-proof in CI.

Notes

• This deliberately supersedes the crawler-facing intent of feat: server-render the request page so crawlers get the request #118 (request pages stay server-rendered — still useful for link previews/unfurls).
• Effect starts on deploy: staging on merge (existing webhook), production on the next release.
• Optional accelerator once deployed to prod: Google Search Console → Removals → temporary site-prefix removal while the noindex propagates. Full de-indexing takes days–weeks as Google re-crawls.

Verification

• npm run check, npm run build, npm test green locally
• Local next start: curl -sI localhost:3000/ → X-Robots-Tag: noindex, nofollow; /robots.txt → 200 allow-all; page HTML contains <meta name="robots" content="noindex, nofollow"/>

[greptile-apps]

Greptile Summary

The PR adds site-wide crawler directives intended to remove Request Scan from search indexes while keeping pages crawlable long enough for crawlers to observe those directives.

• Adds global noindex, nofollow metadata and response headers.
• Serves an allow-all robots.txt.
• Extends CI smoke coverage for the response header and crawler-access policy.
• Bumps the package version to 0.4.1.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/build-and-lint.yml	The smoke test now verifies both header directives and the required allow-all robots policy, resolving the prior partial-policy finding.
next.config.mjs	Adds a global static X-Robots-Tag: noindex, nofollow response header.
src/app/layout.tsx	Adds inherited Next.js metadata preventing indexing and link following.
src/app/robots.ts	Adds an allow-all robots route so crawlers can fetch pages and observe their noindex signals.
package.json	Bumps the application version from 0.4.0 to 0.4.1.
package-lock.json	Synchronizes the root package version with package.json.

_{Reviews (2): Last reviewed commit: "chore: bump version to 0.4.1" | Re-trigger Greptile}

[bassgeta]

Yup, nice and sensible 👍

[rodrigopavezi]

Merge activity

• Aug 17, 2:09 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Aug 17, 2:09 PM UTC: @rodrigopavezi merged this pull request with Graphite.