Scrapester

Documentation | Python SDK | JavaScript SDK | Playground

Turn any website into LLM-ready clean data.

Overview

Scrapester is a powerful web scraping tool that converts website content into clean, markdown-formatted data perfect for LLM processing. With support for both single-page scraping and full website crawling, Scrapester makes it easy to gather web content in a structured, consistent format.

Features

• 🔍 Smart Content Extraction: Automatically removes noise and extracts meaningful content
• 📝 Markdown Output: Clean, structured content perfect for LLMs
• 🕷️ Website Crawling: Scrape entire websites with configurable depth and limits
• 🚀 Multiple SDKs: Official Python and JavaScript support
• ⚡ High Performance: Built for speed and reliability
• 🛡️ Error Handling: Robust error handling and rate limiting protection

Installation

Python

pip install scrapester

JavaScript/TypeScript

npm install scrapester
# or
yarn add scrapester

Quick Start

Python

fromscrapesterimportScrapesterApp# Initialize the clientapp=ScrapesterApp(api_key="your-api-key")
# Scrape a single pageresult=app.scrape("https://example.com")
print(result.markdown)
# Crawl an entire websiteresults=app.crawl(
"https://example.com",
options={
"max_pages": 10,
"max_depth": 2
}
)

JavaScript/TypeScript

import{ScrapesterApp}from'scrapester';// Initialize the clientconstapp=newScrapesterApp('your-api-key');// Scrape a single pageconstresult=awaitapp.scrape('https://example.com');console.log(result.markdown);// Crawl an entire websiteconstresults=awaitapp.crawl('https://example.com',{maxPages: 10,maxDepth: 2});

Response Format

Scrapester returns clean, structured data in the following format:

interfaceCrawlerResponse{url: string;// The scraped URLmarkdown: string;// Clean, markdown-formatted contentmetadata: {// Page metadatatitle: string,description: string,// ... other meta tags};timestamp: string;// ISO timestamp of when the page was scraped}

API Reference

ScrapesterApp

Constructor

newScrapesterApp(apiKey: string,baseUrl?: string,// default: "http://localhost:8000"timeout?: number // default: 600 seconds)

Methods

scrape(url: string)

Scrapes a single URL and returns clean, markdown-formatted content.

crawl(url: string, options?)

Crawls a website starting from the given URL. Options include:

• maxPages: Maximum number of pages to crawl
• maxDepth: Maximum crawl depth
• includePatterns: URL patterns to include
• excludePatterns: URL patterns to exclude

Error Handling

Scrapester provides detailed error information through the APIError class:

classAPIErrorextendsError{statusCode?: number;response?: object;}

Common error scenarios:

• 429: Rate limit exceeded
• 400: Invalid request
• 401: Invalid API key
• 500: Server error

Development

Running Tests

# Python
pytest tests/
# JavaScript
npm test

Building from Source

# Python
pip install -e ".[dev]"# JavaScript
npm install
npm run build

Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Support

• 📖 Documentation
• 💬 Discord Community
• 📧 Email Support

License

This project is licensed under the MIT License - see the LICENSE file for details.