Autonomous SEO audit engine with AI-driven analysis.
RaptorProto is a proof of concept for a fully autonomous SEO pipeline: crawl any website, render JavaScript-heavy pages via Cloudflare Browser Rendering, and feed structured audit data directly to Claude for intelligent analysis, recommendations, and ongoing monitoring -- all orchestrated through a REST API and MCP (Model Context Protocol) integration.
Built by Nick Mangubat, founder of Ravana Solutions, a Norfolk, Virginia web design and SEO studio. This prototype grew into Raptor, the audit toolkit behind every Ravana engagement — read the full story in the Raptor case study.
┌─────────────────────────────────────────────────────────────────┐
│ │
│ Website ──► Raptor Crawler ──► Structured SEO Data │
│ │ │ │
│ Cloudflare Browser │ │
│ Rendering (JS/SPA) ▼ │
│ Claude API / MCP │
│ │ │ │
│ REST API MCP Server │
│ (direct) (agentic) │
│ │ │ │
│ Prioritized Autonomous │
│ Fix Plans Monitoring │
│ │
└─────────────────────────────────────────────────────────────────┘
- What This Proves
- The Crawler
- Cloudflare Browser Rendering
- Claude Integration: REST API + MCP
- Architecture
- Current Status
- Getting Started
- License
A Rust-native crawler can produce structured, machine-readable SEO audit data at scale. 114+ analyzers, health scoring, and multi-format reports -- all in seconds, not minutes.
Cloudflare Browser Rendering closes the SPA gap. JS-heavy sites that return empty shells to traditional crawlers are fully rendered before analysis. Screenshots, entity extraction, and static-vs-rendered comparisons come free.
The output is purpose-built for AI consumption. Raptor's structured JSON is designed to be fed directly into Claude via REST API for on-demand analysis, or exposed as an MCP server for agentic workflows where Claude autonomously crawls, analyzes, and acts.
The core engine is written in Rust using async I/O (tokio) for high-throughput concurrent crawling.
Seed URL
│
▼
┌──────────────────────────────────────────────────┐
│ Crawl Queue (BFS, depth-bounded) │
│ ┌──────────┐ ┌───────────┐ ┌───────────────┐ │
│ │ Fetcher │ │ Parser │ │ URL Resolver │ │
│ │ (reqwest │──│ (scraper │──│ (normalize, │ │
│ │ async) │ │ HTML) │ │ dedup, scope)│ │
│ └──────────┘ └───────────┘ └───────────────┘ │
└──────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Analysis Pipeline (114+ checks) │
│ │
│ Technical On-Page Structured Data │
│ ───────── ──────── ─────────────── │
│ Canonicals Titles JSON-LD │
│ Directives Meta desc Microdata │
│ Status codes Headings Open Graph │
│ Security Content │
│ URL issues Images Links & Hreflang │
│ Sitemaps Readability Internal/External │
└──────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Scoring Engine │
│ 100-point scale, per-issue deductions with caps │
│ Grades: A (90+), B (80+), C (70+), D (60+), F │
└──────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Report Generation │
│ │
│ JSON ─── machine-readable, Claude-ready │
│ HTML ─── self-contained, offline-capable │
│ CSV ─── spreadsheet export │
│ SVG ─── site structure visualization │
└──────────────────────────────────────────────────┘
| Feature | Detail |
|---|---|
| Async concurrent crawling | Configurable concurrency (default 3), semaphore-bounded |
| robots.txt compliance | Honors rules, crawl-delay, disallow patterns |
| Sitemap discovery | Parses sitemap.xml/sitemap index for URL seeding |
| Depth + page limits | Configurable max depth (10) and max pages (500) |
| 114+ SEO analyzers | Titles, meta, headings, images, links, structured data, security headers, canonicals, hreflang, content quality |
| Health scoring | 0-100 scale with per-issue weighted deductions and category caps |
| Self-contained HTML reports | Single file, zero external dependencies, works offline |
| Event-driven architecture | CrawlEvent channel decouples engine from UI -- reusable as a library |
raptor crawl https://example.com --max-pages 100 --concurrency 5
# Output:# JSON crawl-example.com-2026-04-09.json (structured audit data)# HTML crawl-example.com-2026-04-09.html (visual report)Modern SPAs (React, Vue, Angular, Next.js) return skeleton HTML to crawlers. The real content only exists after JavaScript executes. Raptor integrates Cloudflare Browser Rendering to close this gap -- entirely optional, zero local dependencies.
| Command | What It Does |
|---|---|
raptor crawl --backend cloudflare | Full JS-rendered crawl -- pages loaded in headless Chromium on Cloudflare's edge, then analyzed by all 114+ checks |
raptor screenshot <url> | Pixel-perfect full-page captures at any viewport size |
raptor inspect <url> --compare | Side-by-side static vs. JS-rendered audit -- reveals content invisible to traditional crawlers |
raptor extract <url> --preset seo | AI-powered entity extraction (business name, phone, address, services, hours) using Workers AI |
┌──────────────────────────────┐
│ Raptor submits seed URL to │
│ CF /crawl endpoint │
└──────────────┬───────────────┘
│
┌──────────────▼───────────────┐
│ Cloudflare renders pages in │
│ headless Chromium (JS runs) │
│ Discovers + follows links │
└──────────────┬───────────────┘
│
┌──────────────▼───────────────┐
│ Raptor retrieves rendered │
│ HTML via cursor pagination │
└──────────────┬───────────────┘
│
┌──────────────▼───────────────┐
│ Parallel security header │
│ fetch (HSTS, CSP, XFO) │
└──────────────┬───────────────┘
│
┌──────────────▼───────────────┐
│ Same 114+ analyzers run on │
│ rendered HTML. Same reports. │
└──────────────────────────────┘
The inspect --compare command demonstrates why this matters:
┌────────────────────┬──────────────┬──────────────┐
│ Check │ Static │ Rendered │
├────────────────────┼──────────────┼──────────────┤
│ Title │ (empty) │ My SPA App │
│ Meta description │ (empty) │ Welcome to… │
│ H1 count │ 0 │ 1 │
│ Word count │ 12 │ 847 │
└────────────────────┴──────────────┴──────────────┘
A traditional crawler sees 12 words. Cloudflare rendering reveals 847. That gap is exactly what search engines penalize, and exactly what Raptor surfaces.
Cloudflare's free plan includes 10 minutes of browser time per day -- enough for 100-200 pages per crawl session. No credit card required.
This is where Raptor becomes more than a crawler. The structured JSON output is designed to be consumed directly by Claude for AI-driven SEO analysis.
Raptor's JSON output feeds directly into the Claude API (Anthropic REST) for on-demand, intelligent analysis that goes far beyond rule-based checks.
┌─────────────┐ ┌──────────────────┐ ┌──────────────┐
│ raptor │ │ Orchestrator │ │ Claude API │
│ crawl │──────►│ (service layer) │──────►│ (Anthropic) │
│ --json │ JSON │ │ REST │ │
└─────────────┘ │ Sends crawl │ │ Returns: │
│ data + prompt │ │ - Analysis │
│ template │◄──────│ - Fix plan │
└──────────────────┘ │ - Priority │
└──────────────┘
What Claude adds on top of Raptor's raw audit:
| Raptor Produces | Claude Interprets |
|---|---|
| "47 images missing alt text" | "The product gallery on /shop uses decorative images that don't need alt text, but 12 hero images on landing pages are missing descriptive alt -- those are the priority." |
| "Health score: 72 (C)" | "The score is dragged down by 3 redirect chains on your highest-traffic pages. Fixing those alone would push you to B+." |
| "Missing meta descriptions on 23 pages" | "These 23 pages share a /blog/draft URL pattern -- they may be unpublished drafts indexed by mistake. Verify they should be public before writing descriptions." |
| "Duplicate title tags on 8 pages" | "All 8 are paginated category pages (/shoes?page=2, /shoes?page=3). Append the page number to each title to differentiate them." |
Implementation pattern:
// Raptor outputs structured JSONlet crawl_result:CrawlResult = raptor_core::crawler::crawl(config, tx).await?;let json = serde_json::to_string(&crawl_result)?;// Feed into Claude API with a domain-specific promptlet response = anthropic_client
.messages().create(MessageRequest{model:"claude-sonnet-4-20250514",messages:vec![Message{
role:"user",
content: format!("Analyze this SEO audit and produce a prioritized action plan. \ Focus on fixes with the highest traffic impact first.\n\n{}",
json
),}],max_tokens:4096,}).await?;The more powerful pattern: expose Raptor as an MCP (Model Context Protocol) server, giving Claude direct access to crawl, inspect, and extract as tools it can invoke autonomously.
┌───────────────────────────────────────────────────────┐
│ Claude (MCP Client) │
│ │
│ "Audit example.com, then check if the top 3 issues │
│ are also present on their staging site." │
│ │
│ Tool calls: │
│ ┌─────────────────────────────────────────────────┐ │
│ │ 1. raptor_crawl(url: "example.com") │ │
│ │ 2. raptor_inspect(url: "staging.example.com", │ │
│ │ compare: true) │ │
│ │ 3. raptor_extract(url: "example.com/contact", │ │
│ │ preset: "seo") │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ Claude reasons over the combined results and │
│ produces a unified analysis without human prompting. │
└───────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────┐
│ Raptor MCP Server │
│ │
│ Tools exposed: │
│ ───────────── │
│ raptor_crawl Full site crawl → JSON │
│ raptor_inspect Single-page audit (3 modes) │
│ raptor_screenshot Full-page capture → image │
│ raptor_extract Entity extraction → JSON │
│ raptor_compare Diff two crawl snapshots │
│ │
│ Resources exposed: │
│ ───────────────── │
│ Recent crawl results (read-only) │
│ Issue category definitions │
│ Scoring algorithm reference │
│ │
│ Transport: stdio or SSE │
└─────────────────────────────────────────────────────┘
MCP unlocks agentic workflows like:
| Workflow | What Happens |
|---|---|
| Competitive audit | "Compare our site against 3 competitors" -- Claude crawls all 4, cross-references scores, identifies gaps |
| Regression monitoring | "Crawl weekly and alert if score drops" -- Claude diffs snapshots, flags regressions, explains root cause |
| SPA pre-launch check | "Verify the React app is crawlable" -- Claude runs inspect --compare, checks for content gaps between static and rendered |
| Local SEO extraction | "Pull business data from these 50 competitor pages" -- Claude calls extract in a loop, normalizes results into a spreadsheet |
| Fix verification | "Re-crawl after deploy and confirm issues are resolved" -- Claude compares before/after, confirms fixes, flags new issues |
MCP server definition (planned):
{
"mcpServers": {
"raptor": {
"command": "raptor",
"args": ["mcp-server"],
"env": {
"CLOUDFLARE_ACCOUNT_ID": "...",
"CLOUDFLARE_API_TOKEN": "..."
}
}
}
}RaptorProto/
├── raptor-core/ Core Rust library (zero CLI coupling)
│ └── src/
│ ├── crawler/ Async fetcher, parser, queue, robots.txt, sitemaps
│ ├── analysis/ 114+ SEO analyzers + scoring engine
│ ├── cloudflare/ CF Browser Rendering client (crawl, screenshot, inspect, extract)
│ ├── report/ JSON, HTML, CSV, SVG report generators
│ ├── models/ CrawlConfig, CrawlResult, PageData, CrawlEvent
│ └── utils/ Readability scoring, text analysis
│
├── raptor-cli/ CLI binary (clap-based, dispatches to core)
│ └── src/main.rs Dashboard, subcommands, progress bars
│
├── templates/ Minijinja HTML templates (compiled into binary)
│ ├── crawl-report.html Full report with sidebar, tabs, charts
│ ├── components/ Overview, Issues, Pages, Structure tabs
│ ├── styles/ Light theme CSS
│ └── mascot.svg Raptor mascot (mood changes with score)
│
├── python/ Companion analysis tools
│ ├── raptor-entity/ NER + Semantic Density Index (spaCy)
│ ├── raptor-link/ Internal link graph visualization
│ ├── raptor-fossil/ Historical SERP tracking (Wayback Machine)
│ └── raptor-common/ Shared console styling
│
└── docs/
└── cloudflare.md Cloudflare integration setup guide
- Library-first.
raptor-coreis a standalone Rust library with no CLI coupling. It communicates viaCrawlEventchannels, making it embeddable in any service -- including an MCP server. - Structured output. Every crawl produces machine-readable JSON following a consistent schema. This is what makes the Claude integration viable without any data transformation layer.
- Cloudflare is optional. The core crawler, all 114 analyzers, scoring, and reporting work without it. CF integration is behind a Cargo feature flag (
cloudflare, enabled by default). - Self-contained reports. HTML reports are single files with all CSS, JS, and SVG inlined. No CDN. Works offline.
- Responsible crawling. Honors robots.txt, polite delays, exponential backoff, rate limiting by default.
| Component | Status |
|---|---|
| Rust crawler engine | Production-ready (v0.2.0) |
| 114+ SEO analyzers | Complete |
| Health scoring + grading | Complete |
| HTML/JSON/CSV/SVG reports | Complete |
| Cloudflare Browser Rendering | Complete (crawl, screenshot, inspect, extract) |
| Python NER/link/fossil tools | Complete |
| Claude REST API integration | Designed, ready to implement |
| MCP server | Designed, ready to implement |
- Rust 1.70+ (for building from source)
- Python 3.10+ (for entity, link, and fossil analysis tools)
- Cloudflare account (free tier, optional -- only for JS rendering features)
git clone <repo-url> RaptorProto
cd RaptorProto
cargo build --release# Crawl a site
./target/release/raptor crawl https://example.com
# JS-rendered crawl (requires Cloudflare credentials)export CLOUDFLARE_ACCOUNT_ID="your-id"export CLOUDFLARE_API_TOKEN="your-token"
./target/release/raptor crawl https://example.com --backend cloudflare
# Single-page audit: static vs rendered comparison
./target/release/raptor inspect https://example.com --compare
# Screenshot
./target/release/raptor screenshot https://example.com
# Entity extraction
./target/release/raptor extract https://example.com --preset seoMIT