Lead Mining + Posting System

A reusable, provider-neutral system for discovering, qualifying, enriching, exporting, and approval-gated posting of leads.

This repository is the general lead-mining/posting system. It is intentionally separate from vertical applications such as event-business lead operations. Events can be implemented as a connector/use case without changing this core repository.

Live Dashboard

https://web-production-73aaee.up.railway.app

The dashboard is a registry-backed control plane: 19 tools across Discover / Collect / Enrich / Process, chained workflows (SKOOL, Craigslist, social comment mining, LinkedIn post contacts), a chat tab that proxies to a Hermes agent, and a live logs view.

The live instance runs with the owner's provider keys. If you want your own, fork and deploy — see below.

Deploy Your Own (1-Click Fork → Railway)

This is a single-tenant system. Every deployment needs its own provider API keys. Here's how to spin up your own:

Step 1 — Fork the repo

gh repo fork jbellsolutions/lead-mining-posting --clone
cd lead-mining-posting

Step 2 — Create a Railway project

npm install -g @railway/cli
railway login
railway init --name my-lead-orchestrator

Step 3 — Connect the web service to your fork

railway add --service web
railway service source connect --repo <your-github-username>/lead-mining-posting --branch main --service web

Step 4 — Set your environment variables

railway service link web

# Provider keys (set the ones you have — leave the rest empty)
railway variable set FIRECRAWL_API_KEY=your_key
railway variable set SCRAPECREATORS_API_KEY=your_key
railway variable set SUPER_BROWSER_REPO_ROOT=/path/to/super-browser
railway variable set SUPER_BROWSER_BASE_URL=https://your-super-browser-host/mcp
railway variable set SUPER_BROWSER_TOKEN=your_token
railway variable set BRIGHTDATA_API_KEY=your_key
railway variable set APIFY_TOKEN=your_token
railway variable set FULLENRICH_API_KEY=your_key
railway variable set BROWSERBASE_API_KEY=your_key
railway variable set BROWSERBASE_PROJECT_ID=your_project_id

# Optional: gate access with a Bearer token (leave unset for open access)
railway variable set ORCHESTRATOR_ACCESS_TOKEN=your_secret_token

# Optional: read-only mode (blocks tool/workflow/chat execution)
railway variable set ORCHESTRATOR_READ_ONLY=true

Step 5 — (Optional) Add the Hermes chat agent

The chat tab proxies to a separate Hermes agent service. To enable it:

railway add --service agent
railway service source connect --repo <your-github-username>/lead-mining-posting --branch main --service agent
railway variable set --service agent RAILWAY_DOCKERFILE_PATH=deploy/agent/Dockerfile
railway variable set --service agent API_SERVER_ENABLED=true
railway variable set --service agent API_SERVER_HOST=0.0.0.0
railway variable set --service agent API_SERVER_KEY=$(openssl rand -hex 32)
railway variable set --service agent OLLAMA_API_KEY=your_ollama_key

# Point the web service at the agent
railway variable set --service web HERMES_AGENT_URL=http://agent.railway.internal:8642/v1/chat/completions
railway variable set --service web HERMES_AGENT_TOKEN=same_value_as_API_SERVER_KEY

Step 6 — Generate a domain and verify

railway domain --service web
# Open the URL in your browser

Every push to main auto-deploys. The healthcheck hits /health/live.

Environment Variables Reference

Variable	Required	Description
PORT	Railway sets this	HTTP port (Railway injects automatically)
FIRECRAWL_API_KEY	Discover tools	Firecrawl API key for directory/source crawling
SCRAPECREATORS_API_KEY	Social/creator tools	ScrapeCreators API key for social platform search
SUPER_BROWSER_REPO_ROOT	Collect tools	Local path to the Super Browser repo root
SUPER_BROWSER_BASE_URL	Collect tools	Super Browser MCP endpoint URL
SUPER_BROWSER_TOKEN	Collect tools	Super Browser Bearer token
BRIGHTDATA_API_KEY	Collect tools	Bright Data API key
APIFY_TOKEN	Collect tools	Apify API token
FULLENRICH_API_KEY	Enrich tools	FullEnrich API key
BROWSERBASE_API_KEY	Collect tools	Browserbase API key
BROWSERBASE_PROJECT_ID	Collect tools	Browserbase project ID
ORCHESTRATOR_ACCESS_TOKEN	Optional	Bearer token to gate dashboard access
ORCHESTRATOR_READ_ONLY	Optional	true blocks all execution (tools, workflows, chat)
HERMES_AGENT_URL	Chat tab	Hermes agent gateway URL (/v1/chat/completions)
HERMES_AGENT_TOKEN	Chat tab	Bearer token for the Hermes agent gateway

Dashboard Overview

The dashboard has four main areas:

• Tools — 19 registry-backed tools across Discover / Collect / Enrich / Process. Each card shows readiness (which env vars are missing), bounded limits, and suggested next tools. Click a card to see its input form and run it.
• Workflows — Named bounded workflows that chain tools in order (SKOOL Community Mining, Craigslist Directory Mining, Social Comment Mining, LinkedIn Post Contacts). Each stops at the first honest failure.
• Chat — Proxies messages to the configured Hermes agent gateway. Each tool card has a Chat button that pre-fills tool context.
• Logs — Append-only event log for every tool run, workflow run, and chat turn in the current process.

Provider readiness

The provider strip at the top of the dashboard shows which providers are ready and which env vars are missing. Tools that need a missing key will fail honestly with the exact variable name — no fabricated data.

What it does

• Multi-source lead discovery through Super Browser data lanes, Apify, Bright Data, Firecrawl, and public APIs.
• Normalization, canonicalization, cross-source deduplication, scoring, and provenance.
• Public contact-surface extraction first; enrichment and verification are separate, measurable stages.
• LinkedIn post-commenter → company → contact enrichment pipeline.
• Directory → structured lead CSV pipeline with fleet support.
• Cost-gated pilots, provider run IDs, resumable datasets, and coverage reporting.
• Scrape Creators provider adapter, 175-route OpenAPI snapshot, 183 route aliases, and social lead strategy registry.
• Owned lead-mining MCP server with bounded YouTube creator search, public channel profiles, link-in-bio contact surfaces, and provenance-oriented prospect records.
• Draft generation and external posting through an approval-gated browser/action layer. Mining does not silently publish.

Architecture

sources/connectors
        ↓
discovery artifacts + provenance
        ↓
normalize → dedupe → qualify/score
        ↓
public contact evidence → optional enrichment → verification
        ↓
CSV/JSON/Sheet exports + review dashboard
        ↓
approved draft → exact-scope posting action → evidence + reconciliation

The reusable system owns the pipeline contracts. A vertical use case supplies only its sources, fields, filters, copy, account/profile configuration, and fixtures.

Local development

python3 -m venv .venv
.venv/bin/pip install -e '.[test,mcp]'
cp .env.example .env
# Edit .env with your provider keys

# Run the dashboard locally
.venv/bin/lead-orchestrator
# Open http://localhost:8000

# Run tests
.venv/bin/pytest

# Directory sourcing (read-only; produces a local CSV)
.venv/bin/python scripts/lead_pipeline.py

# LinkedIn post commenters → enriched contacts (read-only)
.venv/bin/python scripts/post_to_leads.py "https://www.linkedin.com/posts/example" output.csv

# Owned lead-mining MCP (stdio for Claude/Cursor/Codex/etc.)
lead-mining-mcp

# Owned lead-mining MCP (local Streamable HTTP)
lead-mining-mcp --http --host 127.0.0.1 --port 8765

The owned MCP is our control-plane interface. It is distinct from the vendor's scrape-creators MCP, which remains the provider/data-plane interface. See docs/lead-mining-mcp.md.

The scripts are callable modules as well as command-line utilities. Configure SUPER_BROWSER_REPO_ROOT when the Super Browser engine is installed outside this repository. Never commit .env, cookies, browser profiles, private lead exports, or provider tokens.

Posting safety

Posting, commenting, replying, messaging, CRM writes, and similar external actions are not implied by discovery or enrichment. They require:

1. an exact draft and target scope;
2. an unexpired explicit approval;
3. a durable one-use execution record;
4. evidence of the submitted action or an explicit ambiguous outcome;
5. reconciliation instead of blind retry when the outcome is uncertain.

Use the Super Browser approval/runtime layer for execution. This repo does not bypass platform controls or turn a scrape into an automatic post.

Repository split

• This repo — general system: reusable lead mining, enrichment, posting contracts, connectors, artifacts, and operator skill.
• Events repo — vertical use case: event-lead-ops-agent, with event-specific authorization, pilot sources, and operations scaffolding.

Do not describe the events repository as the general lead-mining platform, and do not add event-specific account data to this repo.

Outputs and evidence

Every production export should include:

• requested target and achieved unique count;
• source/coverage ledger and provider run or dataset IDs;
• schema and field definitions;
• canonicalization and dedupe rules;
• contact evidence URLs and verification statuses;
• known gaps, blocked lanes, and cost summary;
• local CSV/JSON artifacts plus a README or manifest.

A pilot is a preflight, not the final deliverable. Do not invent missing fields or silently downscope a requested list.

License

MIT. Provider terms, platform policies, privacy obligations, and applicable outreach rules remain the operator's responsibility.