Production-grade IOC extraction, enrichment, persistence, and pipeline tooling for threat intelligence workflows
IOCParser extracts Indicators of Compromise from reports, feeds, URLs, stdin, and directory trees. It supports refanging, MISP warning-list enrichment, structured renderers, persisted run history, IOC search, run diffs, and queue-backed distributed processing.
| Feature | Description |
|---|---|
| Multi-source ingestion | Parse PDF, HTML, text, stdin, URLs, URL lists, multi-file batches, and directories |
| IOC extraction | Detect hashes, network indicators, Windows artifacts, threat-intel IDs, crypto addresses, YARA, and more |
| Warning-list enrichment | MISP warning-list matching with normal/warning separation and evidence context |
| Structured outputs | Render text, summary, JSON, JSONL, CSV, and STIX 2.1 |
| Persistence | Store runs in SQLite or MariaDB-compatible SQLAlchemy backends |
| Search and diff | Query persisted IOCs, export runs, diff runs, and compare against latest successful source runs |
| Batch operations | URL retries, backoff, rate limiting, concurrency, and failed-item replay reports |
| Distributed pipeline | Filesystem, RabbitMQ, SQS, and Celery queue adapters with persisted job lifecycle |
| Plugin surface | Custom renderers, enrichers, extractors, postprocessors, and IOC types |
Human reports text, summary
Data formats JSON, JSONL, CSV
Threat intel STIX 2.1 bundles
Persistence run exports, IOC search pages, structured run diffs
Operations URL batch reports, pipeline job results, schema artifacts
Hashes MD5, SHA1, SHA256, SHA512, SSDEEP, IMPHASH
Network Domains, Hosts, IPv4, IPv6, URLs, Emails, ASNs
Windows Registry keys, mutexes, named pipes, service names
Artifacts Filenames, filepaths, certificate serials, JWT, user agents
Threat intel CVEs, MITRE ATT&CK techniques, YARA rules
Crypto Bitcoin, Ethereum, Monero
Other MAC addresses
pip install iocparser-toolgit clone https://github.com/seifreed/IOCParser.git
cd IOCParser
python3 -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -e .pip install -e ".[dev]"# Initialize warning lists once
iocparser --init
# Extract from file, URL, or stdin
iocparser -f report.pdf
iocparser -u https://example.com/report.html
cat report.txt | iocparser --stdin --json
# Persist and query later
iocparser -f report.txt --persist --db-uri "sqlite:///iocparser.db"
iocparser --list-runs --db-uri "sqlite:///iocparser.db"# Single inputs
iocparser -f report.pdf
iocparser https://example.com/report.html
iocparser --stdin < report.txt
# Batch files and URL feeds
iocparser -m report1.txt report2.txt report3.txt
iocparser -d reports --recursive --glob "*.html"
iocparser --url-file feeds.txt --url-workers 8 --url-retries 2 --batch-report-json batch.json
# Output formats
iocparser -f report.txt --json
iocparser -f report.txt --jsonl
iocparser -f report.txt --csv
iocparser -f report.txt --stix --stix-types domains,urls,ips
iocparser -f report.txt --summary
# Analyst filters
iocparser -f report.txt --only urls,domains --severity medium --with-context
iocparser -f report.txt --exclude yara,registry --only-normal
iocparser -f report.txt --sort-by severity --max-evidence 1| Workflow | Description |
|---|---|
-f, --file | Parse a single file or - for stdin |
-u, --url | Download and parse one URL |
--stdin | Read IOC text from stdin |
-m, --multiple | Parse multiple files and merge results |
-d, --directory | Parse files from a directory, with --recursive and --glob |
--url-file | Parse a URL feed with workers, retries, backoff, and rate limiting |
--streaming | Process large files in chunks |
--persist | Save extraction run metadata and IOCs to a database |
--list-runs | List persisted runs |
--search-ioc | Search persisted IOC values with auto, fts, or like backends |
--export-run | Export a persisted run as text, JSON, JSONL, CSV, or STIX |
--diff-runs | Compare two persisted runs |
--diff-latest | Compare a run with the latest successful run from the same source |
--retry-failed-from | Replay failed URL items from a previous batch report |
--schema-version, --migrate | Inspect or migrate the persistence schema |
# Search persisted IOCs
iocparser --search-ioc evil.example --db-uri "sqlite:///iocparser.db"
iocparser --search-ioc evil.example --ioc-type urls --severity informational --tag warning-list-match
# Export and diff runs
iocparser --export-run 42 --json --db-uri "sqlite:///iocparser.db"
iocparser --diff-runs 40 42 --diff-only added --json --db-uri "sqlite:///iocparser.db"
iocparser --diff-latest 42 --summary --db-uri "sqlite:///iocparser.db"# Maintenance
iocparser --delete-run 42 --db-uri "sqlite:///iocparser.db"
iocparser --prune-before 2026-01-01T00:00:00 --keep-latest 10 --db-uri "sqlite:///iocparser.db"| Option | Description |
|---|---|
--url-workers | Number of concurrent URL workers |
--url-retries | Per-URL retry attempts |
--url-backoff | Backoff between URL retries |
--rate-limit | Delay between URL fetches |
--user-agent | Custom HTTP user agent |
--header, --cookie, --proxy | HTTP request customization |
--allow-redirects, --tls-verify, --tls-cert, --ca-bundle | Redirect and TLS policy |
--connect-timeout, --read-timeout | HTTP timeout policy |
fromiocparserimportextractionnormal_iocs, warning_iocs=extraction.extract_iocs_from_file("report.pdf")
normal_iocs, warning_iocs=extraction.extract_iocs_from_text("evil.example 198.51.100.10")
normal_iocs, warning_iocs=extraction.extract_iocs_from_url(
"https://example.com/report.html",
only="urls,domains",
exclude="registry",
)
result=extraction.extract_result_from_file("report.pdf")
print(result.total_count())fromiocparserimportpersistencedb_uri="sqlite:///iocparser.db"runs=persistence.list_persisted_runs(db_uri=db_uri, limit=10)
hits=persistence.search_persisted_iocs(
db_uri=db_uri,
value="evil.example",
ioc_type="urls",
min_severity="medium",
tag="network",
)
exported=persistence.export_persisted_run(db_uri=db_uri, run_id=42)
diff=persistence.diff_persisted_runs(db_uri=db_uri, left_run_id=40, right_run_id=42)fromiocparserimportpipelineclient=pipeline.DistributedPipelineClient(
db_uri="sqlite:///iocparser.db",
queue_backend="filesystem",
queue_path=".iocparser-queue",
)
job=client.submit(
pipeline.PipelineJobRequest(
input_kind="text",
source_value="IOC hxxp://evil.example",
persist=True,
db_uri="sqlite:///iocparser.db",
check_warnings=False,
),
queue_name="ingest",
)
client.process_next(queue_name="ingest")
state=client.get_job(job_id=job.job_id)fromiocparser.infrastructure.extractionimportIOCExtractorfromiocparser.infrastructure.file_parserimportPDFParserfromiocparser.infrastructure.warninglistsimportMISPWarningListstext=PDFParser("report.pdf").extract_text()
raw_iocs=IOCExtractor(defang=True).extract_all(text)
warning_lists=MISPWarningLists()IOCParser resolves configuration in this order:
- CLI arguments
- Environment variables
- INI file
export IOCPARSER_PERSIST=1
export IOCPARSER_DB_URI="sqlite:///iocparser.db"[database]persist = true
uri = sqlite:///iocparser.db
[defaults]only = urls,domains
exclude = yara
output_format = json
with_context = true
severity = medium,high
[network]url_workers = 8
url_retries = 2
url_backoff = 0.25
rate_limit = 0.10Included deployment profiles:
- deploy/iocparser.local.example.ini
- deploy/iocparser.scale.example.ini
- deploy/iocparser.production.example.ini
IOCParser exposes versioned machine-readable contracts for batch reporting and queue-backed processing.
| Document | Scope |
|---|---|
| docs/PIPELINE_CONTRACT.md | Worker input/result contracts and resource limits |
| docs/DISTRIBUTED_PIPELINE.md | Queue-backed execution with filesystem, RabbitMQ, SQS, and Celery |
| docs/WORKER_DEPLOYMENT.md | Worker deployment guidance |
| docs/SCHEMA_ARTIFACTS.md | JSON schema artifacts and release publication |
| docs/SECURITY_OPERATIONS.md | Secret handling and operational guidance |
Standalone worker:
IOCPARSER_WORKER_QUEUE_BACKEND=filesystem \
IOCPARSER_WORKER_QUEUE_PATH=.iocparser-queue \
IOCPARSER_WORKER_QUEUE_NAME=ingest \
IOCPARSER_WORKER_DB_URI=sqlite:///iocparser.db \
iocparser-worker- Python 3.13 or 3.14
- libmagic runtime support for file type detection
- See pyproject.toml for dependencies and optional pipeline extras
If this project is useful in your workflows, you can support development:
This project is licensed under the MIT license. See LICENSE.
Attribution
- Author: Marc Rivero López | @seifreed
- Repository: github.com/seifreed/IOCParser
Built for practical IOC extraction, threat-intelligence automation, and security operations