crawlability

Here are 3 public repositories matching this topic...

nimajafari / faceted-nav-seo

A framework-agnostic reference for handling faceted navigation without wrecking crawl budget: a five-tier classifier (index / canonicalize / noindex / robots-block) driven by one facet registry, with sitemap, robots.txt, and parameter normalization.

python sitemap seo indexing robots-txt url-parameters noindex canonical-urls technical-seo ecommerce-seo crawlability crawl-budget faceted-navigation

Updated May 24, 2026
Python

Wolfnicos / openseo-lens

Star

AI Search Readiness & Web Discoverability Toolkit — Analyze how well websites are prepared for AI-powered search engines

python cli open-source seo geo schema-org ai-search nlnet generative-engine-optimization tdm-reservation crawlability eu-dsm-directive

Updated Apr 26, 2026
Python

justhandledlabs / robots-sitemap-validator

Star

Local-only robots.txt and sitemap.xml validation for AI coding agents

sitemap seo robots-txt skill-md ai-agent-skills openclaw-skill crawlability

Updated Jul 28, 2026
Python

Improve this page

Add a description, image, and links to the crawlability topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the crawlability topic, visit your repo's landing page and select "manage topics."

Learn more