Shared HTTP engine and web crawler for CommonHuman-Lab tools — session management, injection helpers, BFS crawling, and passive recon primitives. One place. No duplication.
pip install commonhuman-core
pip install commonhuman-core[browser] # + headless Chromium crawler (requires selenium)
pip install commonhuman-core[openapi] # + YAML OpenAPI/Swagger support (requires pyyaml)
pip install commonhuman-core[websocket] # + WebSocket injection helpers (requires websocket-client)
pip install commonhuman-core[ntlm] # + NTLM HTTP authentication (requires requests-ntlm)Every CommonHuman-Lab scanner needs to speak HTTP: proxy routing, cookie injection, rate-limit back-off, and injection helpers for query params, POST bodies, path segments, headers, and cookies. Every scanner also needs to crawl — BFS traversal, form discovery, same-origin enforcement.
commonhuman-core is the single source of truth for that layer. Tools that use it get:
- Battle-tested session handling — automatic retry on connection errors, 429 back-off with
Retry-Aftersupport, configurable per-request delay. - A complete injection toolkit — GET params, form POST, JSON POST, path segments (by index), cookies, and custom headers through one consistent interface.
- BFS crawling with exclude patterns — multi-threaded, depth and page limits, HTML form extraction, URL parameter discovery, regex-based URL filtering.
- A single place to improve — a new injection method or crawler feature lands in every tool at once.
fromcommonhuman_core.httpimportHttpClientfromcommonhuman_core.crawlerimportcrawl, CrawlResultfromcommonhuman_core.passiveimportfetch_seedfromcommonhuman_core.authimportform_login, bearer_loginfromcommonhuman_core.openapiimportload_openapi, discover_openapi, ApiEndpointfromcommonhuman_core.browser_crawlerimportbrowser_crawlfromcommonhuman_core.dorkerimportdork, DorkEnginefromcommonhuman_core.wsimportws_inject, discover_ws_urls, WsResultfromcommonhuman_core.source_mapimportfetch_source_maps, SourceMapResult| Module | Purpose |
|---|---|
commonhuman_core.http.HttpClient | HTTP session wrapper — proxy, cookies, SSL, retry, rate limiting, injection helpers |
commonhuman_core.http.parse_cookie_string | Parse name=value; ... or JSON cookie strings |
commonhuman_core.http.parse_post_data | Parse urlencoded or JSON POST bodies into a flat dict |
commonhuman_core.crawler | BFS web crawler — link + form discovery, page source storage |
commonhuman_core.passive | Passive recon helpers — fetch_seed() |
commonhuman_core.auth | Form login, OAuth2 bearer, HTTP Basic/Digest/NTLM — returns auth objects or cookies + headers |
commonhuman_core.openapi | OpenAPI 2.x / 3.x spec parser — expands paths to scannable ApiEndpoint list |
commonhuman_core.browser_crawler | Headless Chromium BFS URL discovery for JS-rendered SPAs (optional: selenium) |
commonhuman_core.dorker | Multi-engine URL discovery (DDG, Bing, Yahoo) — returns URLs with query parameters |
commonhuman_core.ws | WebSocket injection helpers — send payloads, collect frames, detect marker reflection (optional: websocket-client) |
commonhuman_core.source_map | JavaScript source-map fetcher — recovers original source files from .js.map links |
Thin wrapper around requests.Session with everything a scanner needs built in.
fromcommonhuman_core.httpimportHttpClientclient=HttpClient(
timeout=15,
proxy="http://127.0.0.1:8080",
headers={"X-Custom": "value"},
cookies="session=abc; token=xyz",
verify_ssl=False,
delay=0.5, # seconds between requestsauth=None, # requests auth object — e.g. from http_auth()
)resp=client.get("https://target.com/search?q=test")
resp=client.post("https://target.com/login", data={"user": "admin"})
resp=client.head("https://target.com/")
print(client.request_count) # total requests sent (including retries)client.close()# Replace or add a query parameterclient.inject_get("https://target.com/search?q=original", "q", "PAYLOAD")
# → GET /search?q=PAYLOAD# Inject into a form POST bodyclient.inject_post("https://target.com/login", "user", "PAYLOAD", base_data={"csrf": "tok"})
# → POST body: user=PAYLOAD&csrf=tok# Inject into a JSON POST bodyclient.inject_post_json("https://target.com/api/search", "query", "PAYLOAD", base_data={"page": 1})
# → POST body: {"query": "PAYLOAD", "page": 1}# Replace a path segment by index (0-based after splitting on "/")client.inject_path("https://target.com/api/user/123", 3, "PAYLOAD")
# → GET /api/user/PAYLOAD# Pass -1 to append a new trailing segmentclient.inject_path("https://target.com/page", -1, "PAYLOAD")
# → GET /page/PAYLOAD# Inject a cookie for a single requestclient.inject_cookie("https://target.com/", "session", "PAYLOAD")
# Inject a custom header for a single requestclient.inject_header("https://target.com/", "X-Forwarded-For", "PAYLOAD")HttpClient.get_params("https://target.com/?a=1&b=2") # → ["a", "b"]HttpClient.get_base_url("https://target.com/path?q=1") # → "https://target.com"HttpClient.same_origin("https://target.com/a", "https://other.com/b") # → FalseAutomatic 429 back-off with Retry-After header support. Up to 2 retries per request, 5-second default back-off.
# Handled transparently — no extra code neededresp=client.get("https://target.com/api/")Multi-threaded BFS crawler. Discovers pages, forms, and URL parameters within a target origin.
fromcommonhuman_core.httpimportHttpClientfromcommonhuman_core.crawlerimportcrawl, CrawlResult, FormTargetclient=HttpClient(delay=0.2)
result: CrawlResult=crawl(
"https://target.com/",
client,
max_pages=50,
max_depth=3,
threads=5,
same_origin=True,
exclude_patterns=[r"/logout", r"\.pdf$"],
)
result.visited_urls# list of all crawled URLsresult.form_targets# list of FormTarget — each a discovered HTML formresult.url_params# list of (url, [param_names]) for URLs with query paramsresult.page_sources# dict of {url: html} — raw page contentFormTarget carries everything needed to replay a form submission:
forforminresult.form_targets:
print(form.method, form.action)
print(form.params) # {"username": "", "password": ""} — injectable fieldsprint(form.base_data) # {"csrf": "abc", "_submit": "Login"} — non-injectableexclude_patterns accepts a list of regex strings. Any URL matching one is silently skipped before fetching.
fromcommonhuman_core.passiveimportfetch_seedfromcommonhuman_core.httpimportHttpClientclient=HttpClient()
resp=fetch_seed(client, "https://target.com/")
# Returns None on connection error or 4xx/5xx — safe to call without a try/exceptifresp:
print(resp.text)Useful for a single passive check before starting an active scan — confirms the target is reachable and returns a response worth analysing.
Authenticate against a login form, OAuth2 token endpoint, or via HTTP protocol-level auth. Returns either a requests-compatible auth object or cookies and headers to forward to any HttpClient.
fromcommonhuman_core.authimportform_login, bearer_login, http_auth# Form-based login — GET page, extract CSRF, POST credentialsauth=form_login(
login_url="https://target.com/login",
username="admin",
password="secret",
# username_field="username", # default# password_field="password", # default
)
print(auth.cookies) # "session=abc; csrf=xyz"print(auth.headers) # {"Authorization": "Bearer ..."} if JSON token returned# OAuth2 client-credentialsauth=bearer_login(
token_url="https://target.com/oauth/token",
client_id="my-client",
client_secret="my-secret",
)
# HTTP protocol-level auth — returns a requests auth object for HttpClient(auth=...)basic_auth=http_auth("basic", "admin:secret")
digest_auth=http_auth("digest", "user:pass")
ntlm_auth=http_auth("ntlm", "DOMAIN\\user:pass") # requires commonhuman-core[ntlm]client=HttpClient(auth=basic_auth)auth.is_empty() returns True when form/bearer login produced no usable credentials.
http_auth() raises ValueError for unknown auth types or malformed credentials, and ImportError when NTLM is requested without requests-ntlm installed.
Parse an OpenAPI 2.x (Swagger) or 3.x spec and expand every path into a list of ready-to-scan URLs. Path parameters like {id} are substituted with sensible placeholders (1 for integers, a fixed UUID for UUID params).
fromcommonhuman_core.openapiimportload_openapi, discover_openapi# Probe a target for a spec — returns the spec URL or Nonespec_url=discover_openapi("https://target.com")
# Probes /openapi.json, /swagger.json, /api/openapi.json, ... (19 paths)# Accepts a file path, a URL, or a raw JSON/YAML stringendpoints=load_openapi("https://target.com/openapi.json", base_url="https://target.com")
endpoints=load_openapi("/path/to/swagger.yaml") # requires pyyamlforepinendpoints:
print(ep.method, ep.url) # GET https://target.com/users/1print(ep.query_params) # ["filter", "page"]print(ep.body_params) # ["name", "email"]YAML support requires the optional pyyaml dependency:
pip install commonhuman-core[openapi]Headless Chromium BFS crawler that discovers URLs from JavaScript-rendered pages — invisible to HTTP-layer crawlers. Returns a flat list of same-origin URLs found across rendered DOM links.
fromcommonhuman_core.browser_crawlerimportbrowser_crawlurls=browser_crawl(
start_url="https://target.com/",
max_pages=50,
max_depth=3,
cookies="session=abc", # injected before crawlingheadless=True,
# chromium_path="/usr/bin/chromium", # auto-detected by default# chromedriver_path="/usr/bin/chromedriver",spa_wait_s=1.5, # seconds to wait for JS to settle per page
)
forurlinurls:
print(url)Requires the optional selenium dependency:
pip install commonhuman-core[browser]Multi-engine URL discovery via DuckDuckGo, Bing, and Yahoo. Returns URLs that carry query parameters — the candidates most likely to have injectable surfaces. No API key required.
fromcommonhuman_core.dorkerimportdork, DorkEngine# Default engine is DDGurls=dork("site:example.com inurl:search")
urls=dork("inurl:q= filetype:php", max_results=50)
urls=dork("site:example.com", proxy="http://127.0.0.1:8080", timeout=30)
# Query all three engines and deduplicateurls=dork("site:example.com", engine=DorkEngine.ALL)
forurlinurls:
print(url) # https://example.com/search?q=hellodork() returns an empty list on any network or parse failure — safe to call without a try/except.
WebSocket injection helpers — discover WebSocket endpoints in HTML/JS, send payloads, collect response frames, and detect marker reflection.
fromcommonhuman_core.wsimportws_inject, discover_ws_urls, WsResult, WEBSOCKET_AVAILABLE# Discover WebSocket endpoints from page sourcews_urls=discover_ws_urls(html, base_url="https://target.com")
# → ["wss://target.com/ws/feed", "wss://chat.target.com/socket"]# Inject payloads and collect responsesresults: list[WsResult] =ws_inject(
url="wss://target.com/ws",
payloads=["<img src=x onerror=alert('xss')>"],
cookies="session=abc",
marker="StingXSS_marker",
timeout=10,
)
forrinresults:
print(r.reflected, r.responses, r.error)Requires the optional websocket-client dependency:
pip install commonhuman-core[websocket]WEBSOCKET_AVAILABLE is False when the package is not installed; ws_inject() returns [] in that case, so no try/except needed at the call site.
Fetches JavaScript source maps and recovers original source files. Useful for passive recon — production bundles often leak full application source trees via .js.map files.
fromcommonhuman_core.source_mapimportfetch_source_maps, SourceMapResultdeffetcher(url: str) ->str:
returnrequests.get(url, timeout=10).textresult: SourceMapResult=fetch_source_maps(
js_urls=["https://target.com/static/bundle.js"],
fetcher=fetcher,
base_url="https://target.com",
max_maps=10,
)
forpath, sourceinresult.sources.items():
print(path) # src/components/App.jsxprint(source[:200]) # const App = () => ...# result.mapping maps each JS URL to the source paths it containedprint(result.mapping["https://target.com/static/bundle.js"])fetch_source_maps ignores noise paths (node_modules, webpack runtime, test files) automatically. Inline data: source maps are decoded without a network request.
HttpClient is designed to be subclassed. stingxss adds XSS reflection probing on top:
fromcommonhuman_core.httpimportHttpClientclassInjector(HttpClient):
defprobe_reflection(self, url, param, marker, method="GET"):
...
defprobe_header_reflection(self, url, header_name, marker):
...breachsql uses HttpClient directly — no subclass needed.
- Transport only —
commonhuman-corehandles HTTP, crawling, and passive recon. Vulnerability detection, payload generation, and result analysis belong in the tools that use it. - One consistent interface — every injection method follows the same call shape:
(url, target, payload). No special cases per method type. - Threaded by default, deterministic when needed —
crawl()uses a thread pool; passthreads=1for single-threaded sequential crawling. - 100% branch coverage enforced —
pytest --covwithfail_under=100in CI. Every branch in every module is tested.
git clone https://github.com/commonhuman-lab/commonhuman-core.git
cd commonhuman-core
python -m venv .venv &&source .venv/bin/activate
pip install -e ".[dev]"
pytest
pytest tests/unit/ # isolated unit tests only
pytest tests/regression/ # API surface contracts (requires stingxss + breachsql installed)Licensed under the AGPLv3. You are free to use, modify, and distribute this software. If you run it as a service or distribute it, the source must remain open.
For commercial licensing, contact the author.