fetchkit

AI-friendly web content fetching tool designed for LLM consumption. Rust library with CLI, MCP server, and Python bindings.

Features

• HTTP fetching - GET and HEAD methods with streaming support
• Pluggable fetchers - URL-aware dispatch to specialized handlers for repos, docs, feeds, videos, papers, and more
• HTML-to-Markdown - Built-in conversion optimized for LLMs, with fetched relative links/images resolved to absolute URLs
• Agent content focus - Optional low-noise extraction mode for AI agents
• Crawl discovery - Optional bounded same-origin page discovery for AI agents
• HTML-to-Text - Plain text extraction with clean formatting
• Content processors - Post-download extraction for text PDFs, with an extensible registry
• Binary detection - Returns metadata only for unsupported binary formats
• Timeout handling - 1s first-byte, 30s body with partial content on timeout
• Safety limits - 10 MB default decompressed body cap with truncation
• URL filtering - URL-aware allow/block lists for controlled access
• SSRF protection - Resolve-then-check blocks private IPs by default
• MCP server - Model Context Protocol support for AI tool integration

Built-in Fetchers

Fetchkit routes each request through an ordered fetcher registry. Specialized fetchers match first; the default fetcher handles everything else.

• GitHubCodeFetcher - GitHub source file URLs (/blob/...)
• GitHubIssueFetcher - GitHub issue and pull request URLs
• GitHubRepoFetcher - GitHub repository home pages
• TwitterFetcher - X/Twitter status URLs
• StackOverflowFetcher - Stack Overflow and Stack Exchange question URLs
• PackageRegistryFetcher - PyPI, crates.io, and npm package pages
• WikipediaFetcher - Wikipedia article URLs
• YouTubeFetcher - YouTube watch and youtu.be URLs
• ArXivFetcher - arXiv abstract and PDF URLs
• HackerNewsFetcher - Hacker News item threads
• RSSFeedFetcher - RSS and Atom feed URLs
• DocsSiteFetcher - docs sites with llms.txt/llms-full.txt support
• DefaultFetcher - all remaining HTTP/HTTPS URLs with HTML conversion, streaming, timeout handling, and binary detection

Built-in Content Processors

Content processors run after a fetcher retrieves a bounded response body. They select by final URL, response media type, and requested output, then turn documents into LLM-friendly content without performing their own network requests.

• HtmlProcessor - metadata and focused-content extraction followed by native Markdown or text conversion; accepts a custom HtmlToMarkdownConverter
• PdfProcessor - text-based PDF classification and Markdown extraction via pdf-inspector; scanned or image-only pages are reported as requiring OCR

Custom processors implement ContentProcessor and can be registered in a ContentProcessorRegistry. Pass that registry to FetcherRegistry::with_content_processors to retain the built-in fetchers while customizing post-download processing.

Installation

From crates.io (recommended)

cargo install fetchkit-cli

From Git

cargo install --git https://github.com/everruns/fetchkit fetchkit-cli

From Source

git clone https://github.com/everruns/fetchkit
cd fetchkit
cargo install --path crates/fetchkit-cli

CLI Usage

# Fetch URL (outputs markdown with frontmatter)
fetchkit fetch https://example.com
# Output as JSON instead
fetchkit fetch https://example.com -o json
# Custom user agent
fetchkit fetch https://example.com --user-agent "MyBot/1.0"# Hardened outbound policy for cluster/data-plane use
fetchkit fetch https://example.com --hardened
# Discover a small same-origin page map for an agent
fetchkit fetch https://example.com --content-focus agent --crawl --max-pages 5
# Optional JS/DOM rendering for simple SPAs/docs (requires render-rakers feature)
fetchkit fetch https://example.com/app --render-rakers
# Show full documentation
fetchkit --llmtxt

Default output is markdown with YAML frontmatter:

---url: https://example.comstatus_code: 200source_content_type: text/html; charset=UTF-8source_size: 1256quality_score: 1.00extraction_method: "full"---# Example Domain
This domain is for use in illustrative examples in documents...

JSON output (-o json):

{
"url": "https://example.com",
"status_code": 200,
"content_type": "text/html",
"size": 1256,
"format": "markdown",
"content": "# Example Domain\n\nThis domain is for use in illustrative examples..."
}

MCP Server

Run as a Model Context Protocol server:

fetchkit mcp
# Hardened profile for cluster/data-plane use
fetchkit mcp --hardened

Exposes fetchkit tool over JSON-RPC 2.0 stdio transport. Returns markdown with frontmatter (same format as CLI). Compatible with Claude Desktop and other MCP clients.

Library Usage

Add to Cargo.toml:

[dependencies]
fetchkit = "0.2"

Optional rendered fetching:

[dependencies]
fetchkit = { version = "0.2", features = ["render-rakers"] }

render-rakers is not enabled by default. It is lightweight partial rendering: inline JavaScript can update the DOM before markdown/text conversion, but it is not a full browser engine. FetchKit blocks rakers-initiated subresource network access in this mode; the initial page still uses FetchKit's normal URL, DNS, proxy, timeout, and size policies.

Basic Fetch

use fetchkit::{fetch,FetchRequest};#[tokio::main]asyncfnmain(){let request = FetchRequest::new("https://example.com").as_markdown();let response = fetch(request).await.unwrap();println!("{}", response.content.unwrap_or_default());}

With Tool Builder

use fetchkit::{FetchRequest,ToolBuilder};let tool = ToolBuilder::new().enable_markdown(true).enable_text(false).user_agent("MyBot/1.0").allow_prefix("https://docs.example.com").block_prefix("https://internal.example.com").build();let request = FetchRequest::new("https://example.com");let response = tool.execute(request).await.unwrap();

Toolkit Contract Surface

use fetchkit::ToolBuilder;let builder = ToolBuilder::new().enable_save_to_file(true);let tool = builder.build();assert_eq!(tool.name(),"web_fetch");assert_eq!(tool.display_name(),"Web Fetch");let definition = builder.build_tool_definition();letmut service = builder.build_service();

Hardened Tool Profile

use fetchkit::Tool;let tool = Tool::builder().hardened().allow_prefix("https://docs.example.com").build();

Python Bindings

pip install fetchkit

fromfetchkit_pyimportfetch, FetchRequest, FetchkitTool# Simple fetchresponse=fetch("https://example.com", as_markdown=True)
print(response.content)
# With configurationtool=FetchkitTool(
enable_markdown=True,
user_agent="MyBot/1.0",
allow_prefixes=["https://docs.example.com"]
)
response=tool.fetch("https://example.com")

Request Fields

Field	Type	Description
url	string	URL to fetch (required, http:// or https://)
method	enum?	GET (default) or HEAD
as_markdown	bool?	Convert HTML to markdown
as_text	bool?	Convert HTML to plain text
save_to_file	string?	Non-blank destination path; validated by FileSaver before fetching
content_focus	string?	"full"/unset returns everything; "main" strips semantic boilerplate; "readable" selects article-like content; "agent" selects the best low-noise strategy for AI agents
crawl	bool?	Fetch the seed URL, then discover and fetch bounded same-origin pages
max_pages	int?	Maximum crawl pages, including the seed; default 5, max 20
if_none_match	string?	ETag for conditional If-None-Match
if_modified_since	string?	Timestamp for conditional If-Modified-Since
render	string?	"rakers" to opt into rendered fetch when enabled

Response Fields

Field	Type	Description
url	string	Fetched URL
status_code	int	HTTP status code
content_type	string?	Content-Type header
size	int?	Content size in bytes
last_modified	string?	Last-Modified header
etag	string?	ETag header (use for next conditional request)
filename	string?	From Content-Disposition
format	string?	"markdown", "text", "raw", or a fetcher-specific format
content	string?	Page content
truncated	bool?	True if content was cut off
method	string?	"HEAD" for HEAD requests
error	string?	Error message if failed
saved_path	string?	Filesystem path when save_to_file succeeded
bytes_written	int?	Bytes saved to file
metadata	object?	Structured PageMetadata (title, description, links, headings, extraction method, …)
quality	object?	Agent-facing PageQuality (score, warnings, link density, suggested next action)
crawl	object?	Bounded crawl discovery result with visited page summaries
word_count	int?	Word count of returned content
redirect_chain	string[]	URLs visited during redirects (empty if none)
is_paywall	bool?	Heuristic paywall signal (soft, not guaranteed)
rendered_by	string?	Rendering backend used before conversion, e.g. "rakers"

Error Handling

Errors are returned in the error field:

• InvalidUrl - Malformed URL
• UrlBlocked - URL blocked by filter
• NetworkError - Connection failed
• Timeout - Request timed out
• HttpError - 4xx/5xx response
• ContentError - Failed to read body
• BinaryContent - Binary content has no registered processor

Security

Fetchkit blocks connections to private/reserved IP ranges by default, preventing SSRF attacks when used in server-side or AI agent contexts.

Blocked by default: loopback, private networks (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x including cloud metadata), IPv6 equivalents, multicast, and other reserved ranges.

// Default: private IPs blocked (safe for production)let tool = Tool::default();// Explicit opt-out for local development onlylet tool = Tool::builder().block_private_ips(false).build();

DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canonicalized before validation. Redirects are followed manually in the default fetcher so each hop is revalidated against scheme and DNS policy. Allow/block prefixes are matched against parsed URLs rather than raw strings, which prevents lookalike host overmatches such as allowed.example.com.evil.test. Proxy environment variables are ignored by default. Use the hardened profile for cluster-facing deployments and opt in with ToolBuilder::respect_proxy_env(true) only when it is part of an intentional egress design.

See the knowledge/security/threat-model.md concept for the full threat model. See docs/hardening.md for deployment guidance.

Configuration

Timeouts And Limits

• First-byte: 1 second (connect + initial response)
• Body: 30 seconds total
• Body size: 10 MB decompressed content by default

Partial content is returned on body timeout or body-size limit with truncated: true.

PDF And Binary Content

When Markdown is requested, text-based PDFs are downloaded within the configured body limit and converted to Markdown. PDF parsing runs locally; no OCR service or additional network request is used. Scanned/image-only PDFs report use_ocr as the suggested next action.

Other binary content returns metadata only:

• Images, audio, video, fonts
• Archives (zip, tar, rar, 7z)
• Office documents

HTML Conversion

HTML is automatically converted to markdown:

• Headers: h1-h6 → # to ######
• Lists: Proper nesting with 2-space indent
• Code: Language-aware, collision-safe fences and inline backticks
• Links/images: Titles, relative-URL resolution, and highest-resolution srcset
• Tables: Valid Markdown with formatted cells and escaped pipes
• Rich content: Footnotes, LaTeX math, callouts, figures, highlights, strikethrough, details, and task checkboxes
• Strips: document head, scripts, styles, templates, iframes, SVGs
• Adds a bounded Agent resources navigation appendix when discoverable resources are available

License

MIT. See Third-Party Notices for adapted components.