AutoPYara

Automated, Cluster-Driven YARA Rule Generation

Automatically discover malware families and generate high-quality, tightly scoped YARA rules using probabilistic clustering and Bloom-filtered n-gram analysis.

Documentation · PyPI · Report a Bug · Changelog · Releasing

---

📌 Overview

AutoPYara is a Python framework for automated YARA rule generation from collections of malware samples. It combines:

• Variational Bayesian Gaussian Mixture Models (VBGMM)
• Augmented DBSCAN with centroid refinement
• Malicious/benign Bloom filter isolation
• Byte-level n-gram feature extraction

The result: cluster-aware, precision-engineered YARA signatures with minimal manual effort.

🧠 How it works

flowchart TD
    A[Malware Samples] --> B[Byte n-gram Extraction]
    B --> C["Bloom Filter Isolation<br/>(benign removal + malicious focus)"]
    C --> D["Clustering Engine<br/>(VBGMM or Augmented DBSCAN)"]
    D --> E[Cluster-Specific Signature Construction]
    E --> F[High-Quality YARA Rules]

Loading

✨ Features

• Automated Clustering — group similar malware samples together automatically to create concise, targeted rules.
• Two Core Presets — the standard AutoYara (VBGMM) approach, or the enhanced AutoPYara (Augmented DBSCAN) pipeline.
• Built-in Bloom Filters — ships with pre-trained EMBER and AutoPYara filters to efficiently filter out benign n-grams.
• Multiple Output Formats — raw strings, compiled yara-python objects, or yaramod parsed objects.
• Custom Training — train your own Bloom filters on proprietary datasets.

🚀 Installation

Requirements

• Python >= 3.9
• A Java Runtime Environment (JRE 11+) on PATH or pointed to by JAVA_HOME. AutoPYara's clustering/rule-generation backend runs inside a JVM. pip install itself doesn't need Java, but AutoPYara() will raise a clear error the first time you construct it without one — install a JRE before you actually use the tool. On Debian/Ubuntu: sudo apt install default-jre.

pip install autopyara

Install from a local build instead

python -m build
pip install dist/autopyara-*.whl

Note on first run: the Bloom filter data (~600MB)

To keep the initial install lightweight, the package needs about 600MB of pre-trained Bloom filter data that isn't bundled in the distribution. You don't need to fetch this manually — the first time you import autopyara and the data is missing, it's downloaded automatically from the data-branch branch of this repository. To trigger it explicitly (e.g. to pre-warm a Docker image):

autopyara-download

⚡ Quick Start

Generating your first YARA rule is as simple as pointing the tool at a directory of malware samples.

from autopyara import AutoPYara

# 1. Initialize the tool
tool = AutoPYara()

# 2. Generate a rule using the AutoPYara preset
results = tool.generate(
    input_files="/path/to/malware/directory",
    preset="AutoPYara",
    rule_name="my_custom_rule",
    output_format="string"
)

# 3. Print the results
print(f"Discovered {results['k_clusters']} distinct malware clusters.")
print("\nGenerated YARA Rule:")
print(results['rule_string'])

⚙️ Core presets

preset="AutoYara" (Standard)

Algorithm: Variational Bayesian Gaussian Mixture Model (VBGMM)

Behavior: Automatically infers the number of clusters ($K$) probabilistically.

Best for: General-purpose rule generation where the structural diversity of the input directory is completely unknown.

preset="AutoPYara" (Enhanced)

Algorithm: Augmented DBSCAN combined with KMeans soft clustering

Behavior: Uses a custom Augmented DBSCAN to calculate $K$ prior to centroid optimization.

Best for: Producing more tightly bound rules for closely related malware families.

🛠 Advanced usage

Defining a custom $K$

If you want to manually force the algorithm to split your samples into a specific number of clusters, you can override the presets:

# Force exactly 4 clusters using the AutoPYara augmented pipeline
results = tool.generate(
    input_files="/path/to/malware",
    preset="AutoPYara",
    augmented_target_k=4  # Forces the optimizer to find 4 clusters
)

Output formats

By default, AutoPYara returns a raw string. You can integrate it directly into existing analysis pipelines by requesting Python objects instead:

# Returns a compiled yara-python object ready for immediate scanning
results = tool.generate(
    input_files="/path/to/malware",
    output_format="yara-python"
)

compiled_rule = results["output"]
matches = compiled_rule.match("/path/to/suspicious/file.exe")

Supported formats: 'string', 'yara-python', and 'yaramod'.

Custom Bloom filters

generate() defaults to the built-in "ember" Bloom filters for both benign and malicious data. You can switch to the "autopyara" defaults, or provide absolute paths to your own retrained filters:

results = tool.generate(
    input_files="/path/to/malware",
    bloom_malicious="/absolute/path/to/custom/malicious_bloom",
    bloom_benign="/absolute/path/to/custom/benign_bloom",
)

Training new Bloom filters

Train custom Bloom filters on your own proprietary benign or malicious datasets with train():

tool = AutoPYara()

# Extract 8-grams from a directory of benign software
tool.train(
    input_dir="/path/to/benign/software",
    output_dir="/path/to/save/new/bloom",
    ngram_size=8
)

📚 Full API reference: generate()

Parameter	Type	Default	Description
input_files	str | list	Required	Path to input directory or list of sample file paths.
preset	str	None	'AutoYara' or 'AutoPYara'. Auto-configures the clustering pipeline.
bloom_malicious	str	'ember'	Built-in flag ('ember', 'autopyara') or path to custom malicious Bloom filters.
bloom_benign	str	'ember'	Built-in flag ('ember', 'autopyara') or path to custom benign Bloom filters.
output_format	str	'string'	'string', 'yara-python', or 'yaramod'. Determines output rule format.
rule_name	str	'autoyara_rule'	Base string used to name the generated rules.
k_cluster	int	0	Hardcode $K$ for VBGMM. Do not use with preset="AutoPYara".
augmented_target_k	int	None	Hardcode target $K$ for the Augmented DBSCAN pipeline.
verbose	bool	False	Enable detailed logging during cluster generation.

📖 Documentation

Full documentation lives at botacin-s-lab.github.io/AutoPYaraPyPI. It's intentionally basic for now — installation, quick start, and the API reference — with more material (including the accompanying paper, once published) landing there over time.

🧪 Development

pip install -e ".[test]"
pytest tests/

tests/test_core_helpers.py and tests/test_augmented_dbscan.py are pure-Python unit tests (no JVM/network needed). tests/test_smoke_generate.py runs the real pipeline end-to-end against small synthetic dummy files (not real malware) using the built-in Bloom filters.

See RELEASING.md for how versioning and PyPI publishing work.

🤝 Contributing

We're accepting contributions — if you run into an issue or have a fix, fork the repo, open a PR against main, and we'll take a look. PRs are automatically built and tested; once checks pass and a maintainer approves, it gets merged. main itself isn't open to direct pushes from anyone (including maintainers) — everything goes through review. See CONTRIBUTING.md for details.

License

MIT — see LICENSE.

---

Maintained by Mabon Ninan, Texas A&M University — ninanmm@tamu.edu