Resolve queries or URLs into compact, LLM-ready Markdown — intelligent cascade routing across free and paid providers. Zero-config by default: works out of the box with no API keys.
- Zero-key mode — Works out of the box with no API keys; free providers are used by default
- Intelligent cascade — Routes through providers in priority order, stopping at the first successful result
- Self-healing — Circuit breakers and per-domain routing memory recover from failures automatically
- LLM-optimized output — Compact, deduplicated Markdown ready for direct injection into prompts
- Three interfaces — Python library, Rust CLI, and Next.js Web UI — one core, any workflow
- Quick Start
- Architecture
- Features
- Installation
- Configuration
- Usage
- Agent Skills
- Data Layer
- Testing
- Repository Structure
- Contributing
- License
# Clone and install (no API keys needed)
git clone https://github.com/d-oit/do-web-doc-resolver.git
cd do-web-doc-resolver
pip install -r requirements.txt
# Resolve a URL
python -m scripts.cli "https://docs.example.com"# Resolve a search query (uses free providers)
python -m scripts.cli "your search query"Or try the live demo →
The resolver uses a cascade pattern, trying providers in order until one succeeds:
- Semantic Cache (free, instant) — Cached results from previous queries
- Free Providers (no key needed) — Exa MCP, Exa SDK, Tavily, DuckDuckGo
- Paid Providers (API key required) — Serper, Mistral
- Fallback (free) — Jina Reader, Firecrawl, Direct HTTP
Query providers: Semantic Cache → Exa MCP → Exa SDK → Tavily → Serper → DuckDuckGo → Mistral URL providers: Semantic Cache → llms.txt → Jina Reader → Firecrawl → Direct HTTP → Mistral Browser → DuckDuckGo
| Feature | Description |
|---|---|
| Cascade Routing | Automatic provider fallback with configurable priority order |
| Semantic Cache | In-memory similarity lookup with configurable TTL per provider |
| Circuit Breakers | Per-provider failure detection with automatic recovery |
| Routing Memory | Remembers which providers succeed for each domain |
| Quality Scoring | Ranks results by content density and relevance |
| Multi-interface | Python API, Rust CLI (do-wdr), and Next.js Web UI |
| Zero-config | Works without API keys using free providers by default |
| LLM-ready output | Compact Markdown optimized for prompt injection |
Requires Python 3.10 or higher.
git clone https://github.com/d-oit/do-web-doc-resolver.git
cd do-web-doc-resolver
pip install -r requirements.txtcd cli
cargo build --release
# Binary: cli/target/release/do-wdrcd web
npm install --legacy-peer-deps
npm run dev
# Open http://localhost:3000All API keys are optional. The tool works with zero configuration using free providers.
| Variable | Provider | Required | Notes |
|---|---|---|---|
EXA_API_KEY | Exa SDK | No | Enables Exa search with highlights |
TAVILY_API_KEY | Tavily Search | No | Enables broad web search |
SERPER_API_KEY | Serper (Google) | No | Enables Google search |
FIRECRAWL_API_KEY | Firecrawl | No | Enables deep content extraction |
MISTRAL_API_KEY | Mistral AI | No | Enables AI-powered search/browse |
# Linux/macOSexport EXA_API_KEY="your-key"# Windows PowerShell$env:EXA_API_KEY="your-key"Configuration file: config.toml — routing thresholds, cache TTLs, rate limits.
fromscripts.resolveimportresolve# Resolve a URLresult=resolve("https://docs.python.org/3/library/json.html")
print(result["content"])
# Resolve a search queryresult=resolve("Python json module documentation")
print(result["content"])python -m scripts.cli "your search query"
python -m scripts.cli "https://example.com"./cli/target/release/do-wdr resolve "https://docs.example.com"
./cli/target/release/do-wdr resolve "your search query"cd web && npm run dev
# Open http://localhost:3000 and enter a URL or queryThe resolver ships as a skill for AI agents under .agents/skills/. It provides a self-contained SKILL.md with references, tests, and a portable Python module.
| Skill | Interface | Purpose |
|---|---|---|
do-web-doc-resolver | Python | Full cascade resolver — importable module or CLI |
do-wdr-cli | Rust | Compiled do-wdr binary for fast resolution |
# As a CLI (from project root)
python3 -m scripts.cli "https://docs.example.com"# As a Python module
from scripts.resolve import resolve
result = resolve("your search query")
print(result["content"])
# Via the Rust CLIcd cli && cargo build --release
./target/release/do-wdr resolve "https://docs.example.com"The resolver uses two complementary storage systems — both in-memory by default, with optional persistent backends.
Provides similarity-based caching using local embeddings. Identifies semantically equivalent queries (not just exact matches) and returns cached results instantly.
| Component | Detail |
|---|---|
| Engine | SQLite + sqlite-vec vector extension |
| Embeddings | all-MiniLM-L6-v2 via sentence-transformers (~80MB, runs locally) |
| Storage | ~/.cache/do-web-doc-resolver/semantic/semantic_cache.db |
| Similarity | Cosine distance, threshold 0.85 (configurable) |
| Eviction | LRU with max 10,000 entries (configurable) |
| TTL | Per-provider, 1–24 hours (see config.toml) |
# Optional: install sqlite-vec for high-performance vector search
pip install sqlite-vecConfiguration:
| Variable | Default | Description |
|---|---|---|
DO_WDR_SEMANTIC_CACHE | 1 | Set to 0 to disable |
DO_WDR_CACHE_THRESHOLD | 0.85 | Minimum similarity for cache hits |
DO_WDR_CACHE_MAX_ENTRIES | 10000 | Max entries before LRU eviction |
Learns which providers work best for each domain. Ranks providers by success rate, quality score, latency, and recency — so repeated requests to the same domain go to the fastest, most reliable provider first.
| Component | Detail |
|---|---|
| Storage | In-memory (defaultdict), thread-safe |
| Ranking | Weighted: success rate × quality × recency / latency |
| Decay | Recency factor decays over 7 days |
| Base score | 0.5 for unknown provider/domain pairs |
Protects against cascading failures. When a provider fails 3 consecutive times, it is skipped for 5 minutes before retry.
| Setting | Value |
|---|---|
| Failure threshold | 3 consecutive failures |
| Cooldown | 300 seconds (5 minutes) |
| Reset | Successful call resets failure count |
All shared state is managed via a singleton ResolverState object (scripts/state.py):
fromscripts.stateimportget_statestate=get_state()
# state.circuit_breakers — CircuitBreakerRegistry# state.routing_memory — RoutingMemory (per-domain learning)# state.semantic_cache — SemanticCache (sqlite-vec)python -m pytest tests/ -v -m "not live"cd cli && cargo testcd web && npx playwright test --project=desktop./scripts/quality_gate.sh├── scripts/ # Python resolver core
│ ├── resolve.py # Main entry point
│ ├── _cascade.py # Cascade routing engine
│ ├── _query_resolve.py # Query resolution providers
│ ├── _url_resolve.py # URL resolution providers
│ ├── semantic_cache.py # SQLite-vec semantic cache
│ ├── routing_memory.py # Per-domain provider learning
│ ├── circuit_breaker.py # Provider failure protection
│ ├── state.py # Shared resolver state singleton
│ └── quality.py # Content quality scoring
├── cli/ # Rust CLI (do-wdr)
│ └── src/
├── web/ # Next.js Web UI
│ └── app/
├── tests/ # Python test suite
├── .agents/skills/ # Agent skill definitions
│ ├── do-web-doc-resolver/ # Core resolver skill
│ ├── do-wdr-cli/ # Rust CLI skill
│ ├── do-wdr-release/ # Release management
│ └── ... # 11 skills total
├── docs/ # Project documentation
├── agents-docs/ # Agent-specific reference
├── assets/ # Logo, screenshots, visual assets
├── config.toml # Routing, cache, and rate config
├── CONTRIBUTING.md # Contribution guidelines
├── LICENSE # MIT License
└── README.md # This file
Contributions are welcome!
- Fork the repository
- Create a feature branch (
git checkout -b feat/my-feature) - Add tests for new functionality
- Run the quality gate:
./scripts/quality_gate.sh - Submit a pull request
See CONTRIBUTING.md for detailed guidelines (Python linting, Rust clippy, Web typecheck).
MIT License — see LICENSE for details.