Sparrow Testbed (500k+ Scale Web Ecosystem)

A production-grade, local-first mock website designed to generate and serve over 503,000 deterministic pages across 10 diverse verticals. Built with Hono v4, Bun, bun:sqlite, and Tailwind CSS.

Created specifically as a high-performance benchmark testbed for Black Sparrow to test its asynchronous crawler, AIMD adaptive congestion controller, internal PageRank calculator, streaming HTML parser, and 120 technical SEO audit rules.

---

⚡ Quickstart

Prerequisites

• Bun (v1.2+ installed)

# 1. Install dependencies
bun install

# 2. Seed the core taxonomy database (generates SQLite & JSON in ~20ms)
bun run seed

# 3. Compile production Tailwind CSS
bun run build:css

# 4. Start local high-speed server (runs on http://localhost:3001)
bun run dev

The server will be live at http://localhost:3001!

---

🏗️ Architecture: The Hybrid Virtual Database

Generating 500,000 static .html files on disk would take 5–10 GB of storage, hours to build, and strictly violate Vercel's ~15,000 file upload cap.

Instead, this project uses a Hybrid Virtual Database:

1. The Backbone (~600 KB on disk): A lightweight SQLite database (src/db/taxonomy.sqlite) stores the core taxonomy: 50 categories, 500 subcategories, 1,000 brands, 500 cities, 500 authors, and 200 topics.
2. The Virtual Leaves (0 MB on disk): The 500,000+ leaf pages (products, locations, questions, articles) are deterministically derived from their URL ID via a mathematical PRNG seed (src/lib/prng.ts).

Why Determinism Matters for SEO:

• When a crawler visits /p/48291 today or six months from now, it always receives the exact same title, price, brand, breadcrumbs, specifications, and related cross-links.
• Any request is computed in < 0.05 milliseconds with zero database locks.

---

📊 10 Verticals Breakdown (~503,000 Pages)

Vertical	URL Pattern	Page Count	Primary Schema.org Structured Data
E-Commerce Products	/p/:id	120,000	Product, Offer, AggregateRating
Retail Store Locations	/locations/:country/:state/:city/:id	100,000	LocalBusiness, PostalAddress, Geo tags
Community Q&A Forums	/community/questions/:id	100,000	QAPage, Question, Answer
Hardware Comparisons	/compare/:id	50,000	Product, Review, Comparison Table
Technical Documentation	/docs/:id	50,000	TechArticle, HowTo, Code blocks
Editorial Newsroom	/news/:year/:month/:id	30,000	NewsArticle, Article, Author byline
Curated Buying Guides	/best/:id	20,000	ItemList, Ranked awards
A-Z Technical Glossary	/glossary/:letter/:id	15,000	DefinedTerm, Phonetic pronunciation
Careers & Job Board	/careers/jobs/:id	10,000	JobPosting, Salary brackets
Events & Podcasts	/events/:id & /podcasts/:id	5,000	Event, PodcastEpisode, Full Transcripts
Taxonomy Hubs & Legal	/, /category/:cat, /team, /faq	~3,000	Organization, WebSite, FAQPage
Total Live Pages		~503,000	12+ Distinct Schemas

---

🎯 SEO Flaw & Fault Injection ("Chaos Mode")

A mock website where every page is 100% flawless cannot properly test an audit engine. src/lib/chaos.ts deterministically injects controlled SEO defects:

• HTTP 404 Broken Pages (1.0% of pages): id % 100 === 13 returns 404.
• HTTP 301 Permanent Redirects (0.5% of pages): id % 200 === 7 redirects to canonical sibling.
• HTTP 302 Temporary Redirects (0.2% of pages): id % 500 === 23 temporary redirect.
• Missing <title> Tags (0.67% of pages): id % 150 === 1 omits <title>.
• Title Too Long (>70 Chars) (2.0% of pages): id % 50 === 3 triggers length warnings.
• Missing Meta Description (1.25% of pages): id % 80 === 5 omits meta description.
• Robots Directives (0.5% of pages): id % 200 === 19 injects noindex, nofollow.
• Canonical Mismatches (0.83% of pages): id % 120 === 11 points canonical to external/http URL.
• Artificial Latency (1.5s) (0.2% of pages): id % 500 === 0 tests the crawler's AIMD backoff.

---

🗺️ XML Sitemaps & Search Protocol

• Sitemap Index: Accessible at /sitemap.xml
• 11 Chunked Urlsets: Accessible at /sitemaps/sitemap-1.xml through /sitemaps/sitemap-11.xml (each containing up to 50,000 URLs).
• Robots Directives: Accessible at /robots.txt.

---

🧪 Testing with Black Sparrow

Once the server is running locally on port 3001:

# Run Black Sparrow crawler against the local testbed (first 5,000 pages)
blacksparrow audit http://localhost:3001 --max-pages 5000 --concurrency 32

# Check crawl telemetry and PageRank convergence
blacksparrow report --format markdown

---

☁️ Deploying to Vercel

This repository is preconfigured for 1-click Vercel deployment:

• api/index.ts exposes the Hono serverless handler via @hono/vercel-adapter.
• vercel.json routes all wildcard traffic to the serverless function.
• In serverless environments where bun:sqlite is unavailable, src/db/taxonomy.ts automatically falls back to bundled src/db/taxonomy.json (~590 KB).

Deploy via the Vercel CLI:

bunx vercel

---

🤝 Contributing

We welcome contributions of new verticals, chaos fault heuristics, and architectural improvements! Please review our Contributing Guide before submitting a pull request.

---

📄 License

Dual-licensed under either of:

• MIT License (LICENSE-MIT)
• Apache License, Version 2.0 (LICENSE-APACHE)

at your option.