Browser Scraper 🚀

Multi-agent browser automation framework based on page-agent. Uses DOM-based manipulation for fast and reliable web interaction.

✨ Features

• 🤖 Multi-Agent Workflow: Planner → Browser Agent → Extractor pipeline
• 📄 DOM-Based: Direct DOM manipulation (no screenshots needed)
• 🎯 Flexible: Works in user's real browser or headless
• ⚡ Fast: Direct DOM operations, no image processing
• 🔄 Caching: Built-in result caching for performance
• 🔌 MCP Server: External control via Model Context Protocol
• 🎨 Full Toolset: All page-agent tools enabled (javascript, user interaction, etc.)

🏗️ Architecture

┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ PLANNER │────▶│ SCRAPER │────▶│ EXTRACTOR │
│ (qwen3.5) │ │ (glm-5.1) │ │ (gemini) │
└──────────────┘ └──────────────┘ └──────────────┘
│
▼
┌──────────────┐
│ PAGE-AGENT │
│ (DOM-based) │
└──────────────┘
│
▼
┌──────────────┐
│ BROWSER │
│ (User's) │
└──────────────┘

📦 Installation

# Clone repository
git clone https://github.com/mamidevs/browser-scraper.git
cd browser-scraper
# Install dependencies
npm install
# Setup environment variables
cp .env.example .env
# Edit .env and add your OpenRouter API key

🔧 Configuration

Create .env file:

# OpenRouter API KeyOPENROUTER_API_KEY=your_api_key_here# Model Configuration (optional - defaults shown)PLANNER_MODEL=qwen/qwen3.5-27bSCRAPER_MODEL=z-ai/glm-5.1EXTRACTOR_MODEL=google/gemini-2.5-flash# Base URL (optional)OPENROUTER_BASE_URL=https://openrouter.ai/api/v1

🚀 Usage

Basic Scraping (Single Agent)

import{ScrapingAgent}from'browser-scraper';constagent=newScrapingAgent({model: 'z-ai/glm-5.1',apiKey: process.env.OPENROUTER_API_KEY,});awaitagent.initialize();// Navigate and extractawaitagent.navigate('https://example.com');constresult=awaitagent.execute('Extract all product names and prices');console.log(result.data);

Multi-Agent Workflow

import{ScrapingOrchestrator}from'browser-scraper';constorchestrator=newScrapingOrchestrator({planner: {model: 'qwen/qwen3.5-27b',apiKey: '...'},scraper: {model: 'z-ai/glm-5.1',apiKey: '...'},extractor: {model: 'google/gemini-2.5-flash',apiKey: '...'},});awaitorchestrator.initialize();constresult=awaitorchestrator.scrape({task: 'Find all laptops under $1000 with ratings above 4.0',url: 'https://amazon.com',schema: {type: 'array',items: {type: 'object',properties: {name: {type: 'string'},price: {type: 'number'},rating: {type: 'number'}}}}});console.log('Plan:',result.plan);console.log('Data:',result.data);

Batch Scraping

consturls=['https://example.com/page1','https://example.com/page2','https://example.com/page3',];constresults=awaitorchestrator.scrapeBatch(urls.map(url=>({task: 'Extract product names and prices',
url,schema: {/* ... */}})),3// concurrency);

MCP Server

import{ScrapingMCPServer}from'browser-scraper';constserver=newScrapingMCPServer();awaitserver.initialize({planner: {model: 'qwen/qwen3.5-27b',apiKey: '...'},scraper: {model: 'z-ai/glm-5.1',apiKey: '...'},extractor: {model: 'google/gemini-2.5-flash',apiKey: '...'},});// Available tools:// - scrape-url: Single URL scraping// - scrape-batch: Multiple URLs in parallel// - extract-schema: Schema-based extraction// - clear-cache: Clear all caches// - get-stats: Get statistics// Call from external MCP clientconstresult=awaitserver.callTool({name: 'scrape-url',arguments: {url: 'https://example.com',task: 'Extract all links',}});

📚 Examples

Run the examples:

# Basic scraping
npm run example:basic
# Multi-agent workflow
npm run example:multi
# MCP client
npm run example:mcp

🎯 Use Cases

1. E-commerce Price Monitoring

constresult=awaitorchestrator.scrape({task: 'Extract product name, price, availability, and ratings',url: 'https://amazon.com/dp/B08N5KWB9H',schema: {type: 'object',properties: {name: {type: 'string'},price: {type: 'number'},currency: {type: 'string'},availability: {type: 'string'},rating: {type: 'number'},reviews: {type: 'integer'}}}});

2. Job Board Aggregation

constresult=awaitorchestrator.scrape({task: ` 1. Search for "software engineer" jobs 2. Filter by location "Remote" 3. Extract job title, company, salary, and application URL `,url: 'https://linkedin.com/jobs',schema: {type: 'array',items: {type: 'object',properties: {title: {type: 'string'},company: {type: 'string'},salary: {type: 'string'},location: {type: 'string'},applyUrl: {type: 'string'}}}}});

3. News Article Collection

consturls=['https://techcrunch.com','https://theverge.com','https://arstechnica.com',];constresults=awaitorchestrator.scrapeBatch(urls.map(url=>({task: 'Extract article title, author, publish date, and summary',
url,schema: {type: 'array',items: {type: 'object',properties: {title: {type: 'string'},author: {type: 'string'},date: {type: 'string'},summary: {type: 'string'},url: {type: 'string'}}}}})),3);

🔧 API Reference

ScrapingAgent

Single-agent wrapper around page-agent.

Methods:

• initialize(): Initialize the agent (must be called first)
• navigate(url): Navigate to a URL
• execute(task): Execute a natural language task
• extractData(schema): Extract structured data using schema
• clearCache(): Clear cached results

ScrapingOrchestrator

Multi-agent orchestration.

Methods:

• initialize(): Initialize all agents
• scrape(taskConfig): Execute scraping with multi-agent workflow
• scrapeBatch(tasks, concurrency): Execute multiple scraping tasks in parallel
• clearCache(): Clear all caches

ScrapingMCPServer

MCP server for external control.

Tools:

• scrape-url: Single URL scraping
• scrape-batch: Batch scraping
• extract-schema: Schema-based extraction
• clear-cache: Clear cache
• get-stats: Get statistics

🏃 Running

# Build
npm run build
# Run examples
npm run example:basic
npm run example:multi
npm run example:mcp
# Test
npm test

📁 Project Structure

browser-scraper/
├── src/
│ ├── agents/
│ │ └── orchestrator.ts # Multi-agent coordinator
│ ├── page-agent/
│ │ └── wrapper.ts # Page-agent integration
│ ├── storage/
│ │ └── cache.ts # Caching layer
│ ├── mcp/
│ │ └── server.ts # MCP server
│ └── index.ts # Main entry
├── examples/
│ ├── basic.ts # Basic example
│ ├── multi-agent.ts # Multi-agent workflow
│ └── mcp-client.ts # MCP client
├── tests/
├── package.json
├── tsconfig.json
└── README.md

🔒 Privacy & Ethics

• Respect robots.txt: Always check website's robots.txt
• Rate limiting: Use caching and respect rate limits
• Terms of Service: Ensure scraping is allowed by ToS
• Data usage: Use scraped data responsibly and legally

🤝 Contributing

Contributions welcome! Please read CONTRIBUTING.md first.

📄 License

MIT License - see LICENSE file for details.

🔗 Related Projects

• page-agent - DOM-based browser automation
• OpenRouter - Multi-model LLM API
• Hermes Agent - Multi-agent framework

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Built with ❤️ using page-agent and Hermes Agent