Scrappy Python Crawler

Scrappy Python Crawler is a typosquat and phishing reconnaissance scanner for authorized security research. It generates likely lookalike domains, checks DNS and WHOIS intelligence, fetches active pages, analyzes phishing indicators, scores risk, and writes JSON, Markdown, screenshots, and SQLite evidence.

Use it only on domains you own or are explicitly authorized to assess.

Features

• Typosquat generation: omissions, duplications, swaps, homoglyphs, wrong TLDs, compound login/security domains, and bitsquats.
• Domain validation with IDNA normalization, so invalid DNS names are filtered before scanning.
• DNS intelligence for A, MX, and NS records.
• WHOIS metadata collection.
• HTTP fetching with aiohttp and optional Playwright rendering.
• HTML analysis for password forms, hidden forms, iframes, phishing keywords, obfuscated JavaScript, external links, and JavaScript files.
• Explainable 0-100 risk scoring with individual weighted signals.
• SQLite persistence for scans, domains, findings, and evidence.
• JSON and Markdown reports.
• Screenshot gallery as a local HTML report.
• Live CLI progress while domains complete.
• Certificate Transparency lookup through crt.sh and Cert Spotter for hostnames issued under the target domain.
• CLI with fast and deep scan modes.

Install

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
playwright install chromium

If you do not want browser rendering or screenshots, skip the Playwright browser install and run scans with --no-browser --no-screenshots.

Usage

Deep scan with browser rendering:

scrappy-crawler scan example.com --max-concurrent 10 --limit 250

Fast scan without Playwright:

scrappy-crawler scan example.com --no-browser --no-screenshots --limit 250

Disable Certificate Transparency lookup:

scrappy-crawler scan example.com --no-ct

Limit CT hostnames included in reports:

scrappy-crawler scan example.com --ct-limit 100

Custom output location:

scrappy-crawler scan example.com --output-dir runs/example --db runs/example/threats.db

Read the latest saved scan:

scrappy-crawler report

Read a specific scan as JSON:

scrappy-crawler report --scan-id 1 --json

The old script name still works as a compatibility entrypoint:

python threat_crawler2.py scan example.com --no-browser

Outputs

By default, scan artifacts are written to threat_data/:

• threat_data/threat_intel.db: SQLite scan history and evidence.
• threat_data/report_<scan_id>.json: machine-readable report.
• threat_data/report_<scan_id>.md: human-readable report.
• threat_data/gallery_<scan_id>.html: visual screenshot gallery and CT hostname list.
• threat_data/screenshots/*.png: screenshots when browser mode is enabled.
• threat_data/ct_cache.json: cached CT results, reused for 24 hours when available.

Development

Run tests:

pytest

Run lint:

ruff check .

Project layout:

src/scrappy_crawler/
analyzer.py HTML extraction and phishing indicators
cli.py command line interface
config.py runtime configuration
crawler.py scan orchestration
dns_client.py DNS lookups
domain.py typosquat generation and validation
fetcher.py aiohttp and Playwright fetching
reporting.py JSON and Markdown reports
scoring.py risk scoring
storage.py SQLite persistence
whois_client.py WHOIS lookup wrapper
tests/

Notes

Risk scoring is heuristic. Treat results as triage signals, not final proof of malicious activity. Confirm suspicious domains manually before escalation or takedown.