Oley v3

Open-Source Web Intelligence & Data Extraction Engine (Firecrawl & Perplexity Alternative)

A self-hostable Web Intelligence & Data Extraction Engine designed for LLMs, RAG pipelines, and agentic workflows. It provides a clean, unified API for scraping, crawling, structured data extraction, web search, screenshots, PDFs, and deep research synthesis.

🇬🇧 English • 🇪🇸 Español • 🇨🇳 中文 • 🇯🇵 日本語

---

📖 Table of Contents

• About Oley
• Key Features
• Quick Start
• Docker Deployment
• Environment Variables
• Unified API Endpoint: POST /api/fire
• Usage Examples
• Specialty & Streaming Routes
• SDK Clients
• Directory Structure
• License

---

🔍 About Oley

Oley is a production-ready, open-source alternative to Firecrawl and Perplexity. It acts as a bridge between the raw web and Large Language Models (LLMs). It solves common challenges in web data harvesting, including client-side JS rendering, rate limiting, anti-bot protections, and parsing unstructured HTML into LLM-ready formats (like clean GFM Markdown or structured JSON).

Why Oley?

• All-in-One Engine: No need for separate microservices for crawling, scraping, searching, or generating screenshots.
• Agent-Friendly: Automatically cleans up noise (ads, tracking scripts, navigation elements) to reduce LLM context token usage.
• Deep Research Capability: Built-in multi-agent planner that acts like Perplexity's deep search, returning structured facts with precise source citations.

---

🚀 Key Features

• Unified /api/fire Endpoint: One simple interface to rule all actions—auto-detects targets from your payload.
• LLM Schema Extraction (Firecrawl Parity): Extract structured JSON data using custom schemas (e.g. products, reviews, pricing) with natural language prompt guidance.
• Deep Research Engine (Perplexity Parity): Expands queries, executes parallel web searches, scrapes relevant sources, and synthesizes citation-backed answers.
• Anti-Bot Stealth: Rotates user-agents, spoofs canvas/timezone fingerprints, connection parameters, and WebGL context using a randomized pool of real GPUs.
• Advanced Scraper Options: Injects custom cookies/headers, clicks/fills inputs, evaluates custom JS snippets, and waits for selectors before extraction.
• GFM Markdown Engine: Produces clean GitHub-Flavored Markdown tables, nested lists, and checkboxes from dynamic HTML pages.
• $O(1)$ LRU Cache: Memory-efficient response cache with TTL configuration and route-specific cache invalidation support.
• Live SSE Streaming: Stream tokens or research milestones in real-time.
• Zero Dependencies SDKs: Pre-packaged clients for TypeScript and Python.

---

⚡ Quick Start

Running Locally (Node 18+)

1. Clone the repository and navigate into it:

   git clone https://github.com/Asno-dev/asno-ai.git oley &&cd oley

2. Copy and customize the environment configurations:

   cp .env.example .env

3. Install dependencies and launch the developer environment:

   npm install
   npm run dev

   Your unified server is now running at http://localhost:3000.

---

🐳 Docker Deployment

To spin up the service in a containerized environment (which automatically sets up Chrome/Playwright dependencies):

docker compose up --build

---

🌐 Environment Variables (api/.env)

Customize your server behaviors by defining these keys inside your .env file:

PORT=3000# Web server portOLEY_API_KEY=your-api-key# Optional: Enforces Bearer token authRATE_LIMIT_PER_MIN=60# Max requests per minute per IPPUBLIC_URL=http://localhost:3000# LLM Providers (Required for AI actions, schema extraction & deep research)OPENAI_API_KEY=your_keyANTHROPIC_API_KEY=your_keyGOOGLE_API_KEY=your_keyGROQ_API_KEY=your_keyMISTRAL_API_KEY=your_key# Local LLM Support (Ollama)OLLAMA_BASE_URL=http://localhost:11434/v1OLLAMA_MODEL=llama3.1

---

🔥 The Unified API Endpoint: POST /api/fire

Submit requests to /api/fire to automatically route and run any intelligence job.

Parameter Reference

Parameter	Type	Default	Description
url	string	-	Target URL for scraping, screenshot, crawl, or extraction.
query	string	-	Query term for web search or deep research.
action	string	-	Explicit task: scrape, crawl, extract, llmExtract, search, screenshot, pdf, render, research, translate, etc. (Auto-detected if left empty).
formats	array	["markdown"]	Desired outputs: markdown, html, text, metadata, links, images, screenshot, pdf.
renderJs	boolean	false	Enable Playwright JS rendering context.
stealth	boolean	false	Apply Canvas, GPU, and timezone anti-fingerprint blocks.
blockResources	boolean	false	Block css/fonts/images from loading for speed.
waitForSelector	string	-	Wait for CSS selector to appear in DOM before scraping.
actions	array	-	Interactivity script: [{"type": "click", "selector": "#btn"}].
cookies	array	-	Inject context cookies: [{"name": "a", "value": "b", "domain": "..."}].
extractSchema	object	-	Shorthand JSON schema for structured extraction.
prompt	string	-	Instructs the extraction parser in plain English.
cache	boolean	true	Enable $O(1)$ LRU response caching.
ttl	number	300000	Custom cache duration in milliseconds.

---

🛠 Usage Examples

1. Simple Scrape to Markdown (cURL)

curl -X POST http://localhost:3000/api/fire \
-H "Content-Type: application/json" \
-d '{"url": "https://github.com/Asno-dev/asno-ai", "formats": ["markdown"]}'

2. Structured LLM Data Extraction

curl -X POST http://localhost:3000/api/fire \
-H "Content-Type: application/json" \
-d '{ "url": "https://github.com/Asno-dev/asno-ai", "action": "llmExtract", "extractSchema": { "repositoryName": "string", "starCount": "number", "author": "string" } }'

3. Deep Research Synthesis

curl -X POST http://localhost:3000/api/fire \
-H "Content-Type: application/json" \
-d '{ "query": "superconductor news updates", "action": "research", "depth": "deep" }'

4. PowerShell Usage (unified RestMethod)

Invoke-RestMethod-Uri "http://localhost:3000/api/fire"`-Method Post `-ContentType "application/json"`-Body '{"url": "https://github.com/Asno-dev/asno-ai", "formats": ["markdown"], "stealth": true}'

---

📡 Specialty & Streaming Routes

Real-Time Research Streams (POST /api/ai/research/stream)

Allows your clients to trace deep research progress iteratively.

curl -N -X POST http://localhost:3000/api/ai/research/stream \
-H "Content-Type: application/json" \
-d '{"query": "solid state battery breakthroughs"}'

Emits live Event Source records (event: start, event: sources_found, event: answer, event: done).

Real-Time Chat Stream (POST /api/ai/chat/stream)

Direct LLM completion streaming.

curl -N -X POST http://localhost:3000/api/ai/chat/stream \
-H "Content-Type: application/json" \
-d '{"prompt": "Write a Python script for binary search."}'

Cache Management

Keep scrapers synchronized and discard stale cache entries instantly:

• Check Stats: GET /api/cache/stats
• Invalidate URL: POST /api/cache/invalidate (payload: {"url": "https://url-to-invalidate.com"})
• Flush Cache: POST /api/cache/clear

---

📦 Zero-Dependency SDK Clients

Oley includes clean client SDK wrappers for TS and Python under the sdks/ directory.

TypeScript SDK

import{Oley}from'oley';constoley=newOley({baseUrl: 'http://localhost:3000'});constresult=awaitoley.fire({url: 'https://example.com',formats: ['markdown'],stealth: true});

Python SDK

fromoleyimportOleyoley=Oley(base_url="http://localhost:3000")
res=oley.fire(
url="https://example.com",
formats=["markdown"],
stealth=True
)

---

🏗 Directory Architecture

api/src/
index.ts # System Entry point & browser pool
routes.ts # REST router & SSE stream controllers
openapi.ts # OpenAPI 3.1.0 generator schema
core/
fire.ts # Unified request engine
scraper.ts # Playwright & Turndown GFM compiler
cache.ts # O(1) LRU caching layer
stealth.ts # Browser anti-fingerprint spoofing
research.ts # Multi-query deep research builder
synthesis.ts # BM25 ranker & Citation synthesize formatter
stream.ts # Server-Sent Events formatter
extract/
llm_extract.ts # LLM JSON-schema mapper
ai.ts # Summarization, translations, entities pipelines

---

License

MIT © asno-dev