A robust, developer-centric Python web scraper designed to transform complex web content into clean, high-fidelity GitHub Flavored Markdown (GFM).
Built for archival, LLM context gathering, research, and technical documentation. It handles everything from static blogs to complex, JavaScript-heavy Single Page Applications (SPAs).
- Intelligent Extraction: Heuristically identifies main content, stripping navigation, ads, and footers.
- Performance-First: Optimized core loops and pre-compiled regex patterns ensure high throughput (2.6x faster link extraction).
- Recursive Crawling: Spiders through links to a specified depth and page limit.
- High-Fidelity GFM: Preserves tables, code blocks (with language detection), and rich text formatting.
- Asset Management:
- Images: Keep as remote URLs, convert to Base64, or download locally.
- SVGs: Render as images, preserve code, strip, or save to file.
- Local & Remote Sources: Scrape live URLs or local HTML files.
- Dual Engines:
- Static: Fast
requests+BeautifulSoup4for simple sites. - Dynamic: Full
Playwrightintegration for JS-heavy sites.
- Static: Fast
- Multiple Interfaces:
- CLI: Powerful terminal tool with rich flags.
- Web UI: Flask-based interface for browser workflows.
- Library: Clean Python API for integration.
- Remote Offloading: Delegate heavy lifting (Playwright) to a remote Cloud Run instance (ideal for Termux/mobile).
- Core: Python 3.13+
- Parsing:
BeautifulSoup4,lxml - Conversion:
markdownify - Browser Automation:
Playwright - CLI:
click - Web:
Flask,gunicorn,Pico.css - Testing:
pytestecosystem
- Python 3.13+
- Poetry (Recommended) or
pip - Playwright Browsers (for dynamic scraping)
git clone https://github.com/yourusername/md-scraper.git
cd md-scraper
poetry installRequired only if you plan to use --dynamic mode locally:
poetry run playwright install chromiumpip install --editable .If running on Android via Termux, local Playwright is not supported. You have two options:
- Remote Mode: Use the
--serverflag to offload processing to a deployed instance. - Static Mode: Use the default static scraper (no JS execution).
- Chromium (Experimental): Install
pkg install chromiumand setCHROMIUM_PATH.
The scraper command is your primary tool.
# Scrape a single URL to stdout
scraper scrape https://example.com/article
# Save to file
scraper scrape https://example.com/article -o article.md
# Scrape a local file
scraper scrape path/to/local/file.html -o output.mdControl how assets are processed:
# Download images and SVGs to an 'assets' folder
scraper scrape https://example.com \
--image-action file \
--svg-action file \
--assets-dir ./my-assets \
-o ./output/page.md
# Convert images to Base64 (inline)
scraper scrape https://example.com --image-action base64
# Strip all images and SVGs
scraper scrape https://example.com --strip img --svg-action strip| Flag | Options | Description |
|---|---|---|
--image-action | remote (default), base64, file | How to handle <img> tags. |
--svg-action | image (default), preserve, strip, file | How to handle inline <svg> tags. |
--assets-dir | <path> | Directory to save assets when file action is used. |
Crawl a documentation site or blog:
scraper scrape https://tailscale.com/kb/ \
--crawl \
--depth 2 \
--max-pages 20 \
--only-subpaths \
-o ./tailscale-docs--crawl: Enable crawling.--depth <int>: How deep to follow links (default: 3).--only-subpaths: Only follow links that are children of the starting URL.
For Single Page Applications (React, Vue, etc.):
# Local Dynamic (requires Playwright)
scraper scrape https://spa-site.com --dynamic
# Remote Offloading (Recommended for Termux/Low-resource)
scraper scrape https://spa-site.com \
--server https://scraper-751660269987.us-central1.run.appThe scraper-go.sh script provides a user-friendly wizard for batch jobs.
./scraper-go.shFeatures:
- Enter URLs manually or provide a
.txtlist. - Auto-detects page titles for filenames.
- Organizes output into folders.
- Defaults to remote server for reliability.
Run the lightweight Flask UI for development:
poetry run python src/md_scraper/web/app.pyAlternatively, run a production-ready Gunicorn server:
poetry run gunicorn --bind 0.0.0.0:8080 --workers 1 --threads 8 md_scraper.web.app:appTo run Gunicorn in the background (detached mode) using nohup:
nohup poetry run gunicorn --bind 0.0.0.0:8080 --workers 1 --threads 8 md_scraper.web.app:app > output.log 2>&1&Access the UI at http://127.0.0.1:8080.
Integrate into your own scripts:
frommd_scraper.scraperimportScraper# Initializescraper=Scraper()
# Scraperesult=scraper.scrape(
"https://example.com",
dynamic=True,
image_action='base64'
)
# Access Dataprint(f"Title: {result['metadata']['title']}")
print(f"Markdown:\n{result['markdown']}")Isolate the environment with Docker.
# Build
docker build -t scraper .# Run
docker run -p 8080:8080 scraperRun the test suite to ensure reliability.
# Run all tests
poetry run pytest
# Check coverage
poetry run pytest --cov=md_scrapersrc/md_scraper/
├── cli.py # Command-line entry point
├── scraper.py # Core extraction & cleaning logic
├── crawler.py # Recursive crawling engine
├── utils.py # Helper functions (sanitization, headers)
└── web/
├── app.py
└── templates/
The project is optimized for Google Cloud Run.
Live Demo:https://scraper-751660269987.us-central1.run.app
Deploy Command:
gcloud run deploy scraper \
--source . \
--region us-central1 \
--allow-unauthenticated- Fork the repository.
- Create a feature branch (
git checkout -b feature/amazing-feature). - Commit changes (
git commit -m 'Add amazing feature'). - Push to branch (
git push origin feature/amazing-feature). - Open a Pull Request.
Distributed under the MIT License. See LICENSE for more information.