Table of Contents

• Overview
  • Design Considerations
• Features
• Usage
  • Minimal Example
    • With Dependency Injection
    • Without Dependency Injection
  • Web Crawler Example
    • With Dependency Injection
    • Without Dependency Injection
  • Retrieving the Sitemap
  • Checking a Rule
  • Getting Preferred Host
  • Getting Crawl Delay
• Contributing

Overview

Parse robots.txt and sitemaps using dotnet. Supports the proposed RFC9309 standard, as well as the following common, non-standard directives:

• Sitemap
• Host
• Crawl-delay

Design Considerations

This library is based upon HttpClient, making it very familiar, easy to use and adaptable to your needs. Since you have full control over the HttpClient, you are able to configure custom message handlers to intercept outgoing requests and responses. For example, you may want to add custom headers on a request, configure additional logging or set up a retry policy.

Some websites can have very large sitemaps. For this reason, async streaming is supported as the preferred way of parsing sitemaps.

There is also the possibility to extend this library to support protocols other than HTTP, such as FTP.

Features

Name	Supported	Priority
HTTP/HTTPS	✔️
FTPS/FTPS	❌	0.1
Wildcard (*) User-agent	✔️
Allow & disallow rules	✔️
End-of-match ($) and wildcard (*) paths	✔️
Sitemap entries	✔️
Host directive	✔️
Crawl-delay directive	✔️
RSS 2.0 feeds	❌	0.8
Atom 0.3/1.0 feeds	❌	0.8
Sitemaps XML format	✔️
Simple text sitemaps	✔️
Async streaming of sitemaps	✔️
Cancellation token support	✔️
Memory management	✔️

Usage

Install the package via NuGet.

dotnet add package Robots.Txt.Parser

Minimal Example

First, create an instance of RobotWebClient.

With Dependency Injection

publicvoidConfigureServices(IServiceCollectionservices){services.AddHttpClient<IRobotWebClient,RobotWebClient>();}

Without Dependency Injection

usingvarhttpClient=newHttpClient();varrobotWebClient=newRobotWebClient(httpClient);

Web Crawler Example

Optionally, specify message handlers to modify the HTTP pipeline. For example, you may want to throttle the rate of your requests, to responsibily crawl a large sitemap. You can achieve this by adding a custom HttpMessageHandler to the pipeline.

publicclassResponsibleCrawlerHttpClientHandler:DelegatingHandler{protectedoverrideasyncTask<HttpResponseMessage>SendAsync(HttpRequestMessagerequest,CancellationTokencancellationToken){varresponse=awaitbase.SendAsync(request,cancellationToken);awaitTask.Delay(TimeSpan.FromSeconds(1),cancellationToken);returnresponse;}}

With Dependency Injection

publicvoidConfigureServices(IServiceCollectionservices){services.TryAddTransient<ResponsibleCrawlerHttpClientHandler>();services.AddHttpClient<IRobotWebClient,RobotWebClient>().AddPrimaryHttpMessageHandler<ResponsibleCrawlerHttpClientHandler>();}

Without Dependency Injection

varhttpClientHandler=newResponsibleCrawlerHttpClientHandler(){InnerHandler=newHttpClientHandler{AutomaticDecompression=DecompressionMethods.All}};usingvarhttpClient=newHttpClient(httpClientHandler);varrobotWebClient=newRobotWebClient(httpClient);

Retrieving the Sitemap

varrobotsTxt=awaitrobotWebClient.LoadRobotsTxtAsync(newUri("https://github.com"));// providing a datetime only retrieves sitemap items modified since this datetimevarmodifiedSince=newDateTime(2023,01,01);// sitemaps are iterated asynchronously// even if robots.txt does not contain sitemap directive, looks for a sitemap at {url}/sitemap.xmlawaitforeach(variteminrobotsTxt.LoadSitemapAsync(modifiedSince)){}

Checking a Rule

varrobotsTxt=awaitrobotWebClient.LoadRobotsTxtAsync(newUri("https://github.com"));// if rules for the specific robot are not present, it falls back to the wildcard *varhasAnyRulesDefined=robotsTxt.TryGetRules(ProductToken.Parse("SomeBot"),outvarrules);// even if no wildcard rules exist, an empty rule-checker is returnedvarisAllowed=rules.IsAllowed("/some/path");

Getting Preferred Host

varrobotsTxt=awaitrobotWebClient.LoadRobotsTxtAsync(newUri("https://github.com"));// host value will fall back to provided host, if no directive existsvarhasHostDirective=robotsTxt.TryGetHost(outvarhost);

Getting Crawl Delay

varrobotsTxt=awaitrobotWebClient.LoadRobotsTxtAsync(newUri("https://github.com"));// if rules for the specific robot are not present, it falls back to the wildcard *// if no Crawl-delay directive exists, crawl delay will be 0varhasCrawlDelayDirective=robotsTxt.TryGetCrawlDelay(ProductToken.Parse("SomeBot"),outvarcrawlDelay);

Contributing

Issues and pull requests are encouraged. For large or breaking changes, it is suggested to open an issue first, to discuss before proceeding.

If you find this project useful, please give it a star.