BranchesTags

Repository files navigation

Google Robots.txt Parser and Matcher Library

The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++14).

About the library

The Robots Exclusion Protocol (REP) is a standard that enables website owners to control which URLs may be accessed by automated clients (i.e. crawlers) through a simple text file with a specific syntax. It's one of the basic building blocks of the internet as we know it and what allows search engines to operate.

Because the REP was only a de-facto standard for the past 25 years, different implementers implement parsing of robots.txt slightly differently, leading to confusion. This project aims to fix that by releasing the parser that Google uses.

The library is slightly modified (i.e. some internal headers and equivalent symbols) production code used by Googlebot, Google's crawler, to determine which URLs it may access based on rules provided by webmasters in robots.txt files. The library is released open-source to help developers build tools that better reflect Google's robots.txt parsing and matching.

For webmasters, we included a small binary in the project that allows testing a single URL and user-agent against a robots.txt.

Building the library

Quickstart

We included with the library a small binary to test a local robots.txt against a user-agent and URL. Running the included binary requires:

  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers are supported.
  • Git for interacting with the source code repository. To install Git, consult the Set Up Git guide on GitHub.
  • Although you are free to use your own build system, most of the documentation within this guide will assume you are using Bazel. To download and install Bazel (and any of its dependencies), consult the Bazel Installation Guide

Building with Bazel

Bazel is the official build system for the library, which is supported on most major platforms (Linux, Windows, MacOS, for example) and compilers.

To build and run the binary:

$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES

Building with CMake

CMake is the community-supported build system for the library.

To build the library using CMake, just follow the steps below:

$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES

Notes

Parsing of robots.txt files themselves is done exactly as in the production version of Googlebot, including how percent codes and unicode characters in patterns are handled. The user must ensure however that the URI passed to the AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter of the robots tool, follows the format specified by RFC3986, since this library will not perform full normalization of those URI parameters. Only if the URI is in this format, the matching will be done according to the REP specification.

Also note that the library, and the included binary, do not handle implementation logic that a crawler might apply outside of parsing and matching, for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.

License

The robots.txt parser and matcher C++ library is licensed under the terms of the Apache license. See LICENSE for more information.

Links

To learn more about this project:

About

The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).

Resources

Readme
Apache-2.0 license

Contributing

Contributing
Activity

Stars

2 stars

Watchers

2 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library
(compliant to C++14).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to
control which URLs may be accessed by automated clients (i.e. crawlers) through
a simple text file with a specific syntax. It's one of the basic building blocks
of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different
implementers implement parsing of robots.txt slightly differently, leading to
confusion. This project aims to fix that by releasing the parser that Google
uses.
The library is slightly modified (i.e. some internal headers and equivalent
symbols) production code used by Googlebot, Google's crawler, to determine which
URLs it may access based on rules provided by webmasters in robots.txt files.
The library is released open-source to help developers build tools that better
reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a
single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a
user-agent and URL. Running the included binary requires:
  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are
fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers
are supported.
  • Git for interacting with the source code repository.
To install Git, consult the
Set Up Git guide on
GitHub.
  • Although you are free to use your own build system, most of the
documentation within this guide will assume you are using
Bazel. To download and install Bazel (and any of its
dependencies), consult the
Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library,
which is supported on most major platforms (Linux, Windows, MacOS, for example)
and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the
library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Notes
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter
of the robots tool, follows the format specified by RFC3986, since this library
will not perform full normalization of those URI parameters. Only if the URI is
in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the
Apache license. See LICENSE for more information.
Links
To learn more about this project:
About
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
2 stars
Watchers
2 watching
Forks
1 fork
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library
(compliant to C++14).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to
control which URLs may be accessed by automated clients (i.e. crawlers) through
a simple text file with a specific syntax. It's one of the basic building blocks
of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different
implementers implement parsing of robots.txt slightly differently, leading to
confusion. This project aims to fix that by releasing the parser that Google
uses.
The library is slightly modified (i.e. some internal headers and equivalent
symbols) production code used by Googlebot, Google's crawler, to determine which
URLs it may access based on rules provided by webmasters in robots.txt files.
The library is released open-source to help developers build tools that better
reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a
single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a
user-agent and URL. Running the included binary requires:
  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are
fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers
are supported.
  • Git for interacting with the source code repository.
To install Git, consult the
Set Up Git guide on
GitHub.
  • Although you are free to use your own build system, most of the
documentation within this guide will assume you are using
Bazel. To download and install Bazel (and any of its
dependencies), consult the
Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library,
which is supported on most major platforms (Linux, Windows, MacOS, for example)
and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the
library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Notes
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter
of the robots tool, follows the format specified by RFC3986, since this library
will not perform full normalization of those URI parameters. Only if the URI is
in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the
Apache license. See LICENSE for more information.
Links
To learn more about this project:
About
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
2 stars
Watchers
2 watching
Forks
1 fork
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library
(compliant to C++14).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to
control which URLs may be accessed by automated clients (i.e. crawlers) through
a simple text file with a specific syntax. It's one of the basic building blocks
of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different
implementers implement parsing of robots.txt slightly differently, leading to
confusion. This project aims to fix that by releasing the parser that Google
uses.
The library is slightly modified (i.e. some internal headers and equivalent
symbols) production code used by Googlebot, Google's crawler, to determine which
URLs it may access based on rules provided by webmasters in robots.txt files.
The library is released open-source to help developers build tools that better
reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a
single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a
user-agent and URL. Running the included binary requires:
  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are
fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers
are supported.
  • Git for interacting with the source code repository.
To install Git, consult the
Set Up Git guide on
GitHub.
  • Although you are free to use your own build system, most of the
documentation within this guide will assume you are using
Bazel. To download and install Bazel (and any of its
dependencies), consult the
Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library,
which is supported on most major platforms (Linux, Windows, MacOS, for example)
and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the
library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Notes
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter
of the robots tool, follows the format specified by RFC3986, since this library
will not perform full normalization of those URI parameters. Only if the URI is
in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the
Apache license. See LICENSE for more information.
Links
To learn more about this project:
About
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
2 stars
Watchers
2 watching
Forks
1 fork
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library
(compliant to C++14).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to
control which URLs may be accessed by automated clients (i.e. crawlers) through
a simple text file with a specific syntax. It's one of the basic building blocks
of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different
implementers implement parsing of robots.txt slightly differently, leading to
confusion. This project aims to fix that by releasing the parser that Google
uses.
The library is slightly modified (i.e. some internal headers and equivalent
symbols) production code used by Googlebot, Google's crawler, to determine which
URLs it may access based on rules provided by webmasters in robots.txt files.
The library is released open-source to help developers build tools that better
reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a
single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a
user-agent and URL. Running the included binary requires:
  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are
fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers
are supported.
  • Git for interacting with the source code repository.
To install Git, consult the
Set Up Git guide on
GitHub.
  • Although you are free to use your own build system, most of the
documentation within this guide will assume you are using
Bazel. To download and install Bazel (and any of its
dependencies), consult the
Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library,
which is supported on most major platforms (Linux, Windows, MacOS, for example)
and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the
library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Notes
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter
of the robots tool, follows the format specified by RFC3986, since this library
will not perform full normalization of those URI parameters. Only if the URI is
in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the
Apache license. See LICENSE for more information.
Links
To learn more about this project:
About
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
2 stars
Watchers
2 watching
Forks
1 fork
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library
(compliant to C++14).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to
control which URLs may be accessed by automated clients (i.e. crawlers) through
a simple text file with a specific syntax. It's one of the basic building blocks
of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different
implementers implement parsing of robots.txt slightly differently, leading to
confusion. This project aims to fix that by releasing the parser that Google
uses.
The library is slightly modified (i.e. some internal headers and equivalent
symbols) production code used by Googlebot, Google's crawler, to determine which
URLs it may access based on rules provided by webmasters in robots.txt files.
The library is released open-source to help developers build tools that better
reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a
single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a
user-agent and URL. Running the included binary requires:
  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are
fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers
are supported.
  • Git for interacting with the source code repository.
To install Git, consult the
Set Up Git guide on
GitHub.
  • Although you are free to use your own build system, most of the
documentation within this guide will assume you are using
Bazel. To download and install Bazel (and any of its
dependencies), consult the
Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library,
which is supported on most major platforms (Linux, Windows, MacOS, for example)
and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the
library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Notes
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter
of the robots tool, follows the format specified by RFC3986, since this library
will not perform full normalization of those URI parameters. Only if the URI is
in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the
Apache license. See LICENSE for more information.
Links
To learn more about this project:
About
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
2 stars
Watchers
2 watching
Forks
1 fork
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library
(compliant to C++14).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to
control which URLs may be accessed by automated clients (i.e. crawlers) through
a simple text file with a specific syntax. It's one of the basic building blocks
of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different
implementers implement parsing of robots.txt slightly differently, leading to
confusion. This project aims to fix that by releasing the parser that Google
uses.
The library is slightly modified (i.e. some internal headers and equivalent
symbols) production code used by Googlebot, Google's crawler, to determine which
URLs it may access based on rules provided by webmasters in robots.txt files.
The library is released open-source to help developers build tools that better
reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a
single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a
user-agent and URL. Running the included binary requires:
  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are
fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers
are supported.
  • Git for interacting with the source code repository.
To install Git, consult the
Set Up Git guide on
GitHub.
  • Although you are free to use your own build system, most of the
documentation within this guide will assume you are using
Bazel. To download and install Bazel (and any of its
dependencies), consult the
Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library,
which is supported on most major platforms (Linux, Windows, MacOS, for example)
and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the
library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Notes
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter
of the robots tool, follows the format specified by RFC3986, since this library
will not perform full normalization of those URI parameters. Only if the URI is
in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the
Apache license. See LICENSE for more information.
Links
To learn more about this project:
About
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
2 stars
Watchers
2 watching
Forks
1 fork
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library
(compliant to C++14).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to
control which URLs may be accessed by automated clients (i.e. crawlers) through
a simple text file with a specific syntax. It's one of the basic building blocks
of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different
implementers implement parsing of robots.txt slightly differently, leading to
confusion. This project aims to fix that by releasing the parser that Google
uses.
The library is slightly modified (i.e. some internal headers and equivalent
symbols) production code used by Googlebot, Google's crawler, to determine which
URLs it may access based on rules provided by webmasters in robots.txt files.
The library is released open-source to help developers build tools that better
reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a
single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a
user-agent and URL. Running the included binary requires:
  • A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are
fully supported.
  • A compatible C++ compiler supporting at least C++14. Most major compilers
are supported.
  • Git for interacting with the source code repository.
To install Git, consult the
Set Up Git guide on
GitHub.
  • Although you are free to use your own build system, most of the
documentation within this guide will assume you are using
Bazel. To download and install Bazel (and any of its
dependencies), consult the
Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library,
which is supported on most major platforms (Linux, Windows, MacOS, for example)
and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the
library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build &&cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Notes
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter
of the robots tool, follows the format specified by RFC3986, since this library
will not perform full normalization of those URI parameters. Only if the URI is
in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the
Apache license. See LICENSE for more information.
Links
To learn more about this project:
About
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
2 stars
Watchers
2 watching
Forks
1 fork
Report repository
Releases
Packages
Contributors
Languages