Toimik

Pinned Loading

1. WarcProtocolWarcProtocolPublic

   Parser for WARC (aka WebArchive) files

   C# 16 5

2. CommonCrawlCommonCrawlPublic

   Common Crawl's processing tools

   C# 11

3. UrlNormalizationUrlNormalizationPublic

   URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical

   C# 5

4. SitemapsProtocolSitemapsProtocolPublic

   Parsers for sitemap / sitemap index (aka Sitemaps Protocol)

   C#

5. RobotsProtocolRobotsProtocolPublic

   Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

   C#