Apache DataSketches Rust provides stochastic streaming algorithms for answering queries over large data sets with compact, mergeable summaries. It is the core Rust component of Apache DataSketches and currently implements a subset of the algorithms available in the other language components.
Sketch implementations are opt-in Cargo features; the crate enables none by default. For example, add the HyperLogLog implementation with:
cargo add datasketches --features hllThen build a sketch and query its distinct-count estimate:
use datasketches::hll::HllSketch;use datasketches::hll::HllType;letmut sketch = HllSketch::new(12,HllType::Hll8);for user in["alice","bob","alice","carol"]{
sketch.update(user);}assert!(sketch.estimate() >= 3.0);Enable multiple algorithms by listing their features together, such as features = ["hll", "theta"] in Cargo.toml.
| Feature | Main types | Use case |
|---|---|---|
bloom | BloomFilter | Space-efficient probabilistic set membership with a configurable false-positive rate. |
countmin | CountMinSketch | Approximate point-frequency queries over a stream. |
cpc | CpcSketch, CpcUnion, CpcWrapper | Highly compact distinct-count estimation and unions. |
frequencies | FrequentItemsSketch | Heavy-hitter discovery with upper and lower frequency bounds. |
hll | HllSketch, HllUnion | Fast distinct-count estimation and unions. |
tdigest | TDigestMut, TDigest | Quantile and rank estimation, with high accuracy near distribution tails. |
theta | ThetaSketch and set operations | Distinct counts, set expressions, and Jaccard similarity. |
tuple | TupleSketch and set operations | Theta-style keys with user-defined summaries attached to retained entries. |
See the API documentation for configuration, accuracy guarantees, serialization, and examples for each algorithm.
The minimum supported Rust version is 1.86.0. The crate currently supports little-endian targets only.
Supported serialization formats are tested with fixtures produced by Apache DataSketches Java, C++, and Go through the DataSketches TCK. When values must hash identically across language implementations, use the compatibility wrappers in hash::value.
See the changelog for release notes and migration guidance.
Apache DataSketches also provides core library components for other languages:
Visit the Apache DataSketches website for algorithm documentation, research background, and project-wide resources.
Questions, bug reports, and feature requests are welcome through GitHub issues and GitHub discussions. The Apache DataSketches community page lists the public mailing lists and other ways to participate.
See CONTRIBUTING.md to build, test, and contribute to the Rust component. All project participation is governed by the Apache Software Foundation Code of Conduct.
To report a security vulnerability, follow the ASF security reporting process instead of opening a public issue.
Licensed under the Apache License, Version 2.0.