robots-exclusion-protocol

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

Generate an XML sitemap for a GitHub Pages site using GitHub Actions

Manage the robots.txt from the Kirby config file

Alternative robots parser module for Python

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R

Robots for Kirby CMS

Chrome extension that audits which paths a site disallows in robots.txt and how many of them Google has indexed anyways.

RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.

GIT SOURCE CODE . this is a license certificate agreement code repository for full license package you can use https://github.com/git/git/pull/2379 for application submission turning the certificate into files using the unlimited [PACK] once channel is open for license purchase use pull 2379 via LICENSEPACK@outlook.com

Parse robots.txt and sitemaps using dotnet

User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

Typescript robots.txt parser with support for wildcard (*) matching.

python binding for Google robots.txt parser C++ library