From 23dd9aa103db17cbc69c3bbc6970c7c449838af0 Mon Sep 17 00:00:00 2001 From: Divya Date: Fri, 29 May 2026 16:36:33 +0530 Subject: [PATCH] fix: block bots from crawling Mintlify static assets Re-applied on top of current main (original branch fix/robots-txt-block-static-assets was cut from an old initial-commit state and rebasing produced unrelated conflicts in favicon/logo/.mintignore/docs.json). Clarity data shows bots (Apple, OpenAI, Google) spending ~200 requests/week on /mintlify-assets/_next/static/ JS/CSS chunks. These have zero SEO value. Adds custom robots.txt that blocks /mintlify-assets/ while keeping the existing /cdn-cgi/ block and sitemap reference. Co-Authored-By: Claude Opus 4.7 (1M context) --- robots.txt | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 robots.txt diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000..a2a4eae --- /dev/null +++ b/robots.txt @@ -0,0 +1,5 @@ +User-agent: * +Disallow: /cdn-cgi/ +Disallow: /mintlify-assets/ + +Sitemap: https://docs.tracebloc.io/sitemap.xml