You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat: Search compressed data without decompression
Compression saves space, but it usually makes your data opaque: to find anything you first
have to decompress the whole thing. MinLZ changes that. A MinLZ stream can carry a small
per-block search index that lets mz search find a byte sequence while skipping every block
that provably can't contain it — those blocks are never decompressed, and (with a sidecar) never
even read from disk.
The result: you keep your data compressed and searchable.
One number to set the scene: finding a specific string in a 10 GB CockroachDB log
(compressed to 578 MB) takes 0.14 s with mz search, reading about 133 MB. Decompressing
that log and piping it to grep — the usual way to search compressed data — takes ~5 s and
reads the whole file. zstd -dc | grep takes ~5 s, lz4 -dc | grep ~10 s. Scanning the raw
10 GB with rg takes ~8 s.
The search index is stored as skippable chunks: older MinLZ readers ignore them and the compressed
payload is byte-for-byte unchanged, so adding search is always backward-compatible.
Search indexes can be built while compressing or after the fact as sidecar streams,
allowing them to be used separately from the compressed data.
How it works (in 30 seconds)
Every block gets a tiny bloom-filter table: the byte-windows in the block are hashed and their
bits set. To search, mz search hashes the same-length windows of your pattern and checks each
block's table:
any window bit missing → the pattern is definitely not in that block → skip it (no decode);
all bits present → the block might match → decode and scan it.
Longer/rarer patterns produce more independent window checks, so blocks that don't contain them are
rejected with near-certainty. Tables can live inline in the .mz, or in a sidecar .mzs
file you build afterwards — handy for data you can't or don't want to re-compress.
This is why it matches with compression. Compressible data is very likely to produce
search indexes that are also useful, meaning there is a reasonable reduction in the
search table compared to the raw data.
Most Recent Ignore Conditions Applied to This Pull Request
Dependency Name
Ignore Conditions
golang.org/x/net
[>= 0.38.a, < 0.39]
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
@dependabot ignore <dependency name> major version will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
@dependabot ignore <dependency name> minor version will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
@dependabot ignore <dependency name> will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
@dependabot unignore <dependency name> will remove all of the ignore conditions of the specified dependency
@dependabot unignore <dependency name> <ignore condition> will remove the ignore condition of the specified dependency and ignore conditions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dependenciesPull requests that update a dependency filegoPull requests that update Go code
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps the go-dependencies group with 5 updates:
0.53.00.54.01.6.41.6.51.1.11.2.03.19.03.20.00.56.00.57.0Updates
golang.org/x/cryptofrom 0.53.0 to 0.54.0Commits
cdce021go.mod: update golang.org/x dependenciesd9474ccopenpgp: make the deprecation message more explicit7626c50ssh: verify declared key type matches decoded key in authorized_keys0471e79ssh/agent: enforce strict limits on DSA key parameters6435c37ssh: sanitize client disconnect messages7d695dassh/agent: drain channel stderr in agent forwarders5b7f841acme/autocert: fix data race in Manager.createCert0b316e7argon2: update RFC 9106 parameter recommendations55aec0ax509roots/fallback: update bundle5f2de1ainternal: remove wycheproof testsUpdates
github.com/bodgit/sevenzipfrom 1.6.4 to 1.6.5Release notes
Sourced from github.com/bodgit/sevenzip's releases.
Changelog
Sourced from github.com/bodgit/sevenzip's changelog.
Commits
dcfc72achore(main): release 1.6.5 (#460)9dddc71fix(deps): update module github.com/klauspost/compress to v1.19.0 (#474)0ba424cfix(deps): update module github.com/andybalholm/brotli to v1.2.2 (#473)4271f9bfix(deps): update module golang.org/x/text to v0.40.0 (#467)d9d7777fix(deps): update module golang.org/x/sync to v0.22.0 (#466)7f22666chore(deps): update actions/checkout action to v7 (#468)b137c31chore(deps): update marocchino/sticky-pull-request-comment action to v3.0.5 (...db3d12fchore(deps): update pre-commit hook commitizen-tools/commitizen to v4.16.4 (#...d2efd1cchore(deps): update github/codeql-action action to v4.37.0 (#465)b9f4edbfix: return EOF when buffer is empty (#476)Updates
github.com/minio/minlzfrom 1.1.1 to 1.2.0Release notes
Sourced from github.com/minio/minlz's releases.
... (truncated)
Commits
6613957Upgrade compression dependency (#56)6af501edocs: Fix outdated docs and errors. (#55)0051ca9Minor amd64 decoder tweaks (#54)c9c2bdfl3: Do re-matching on borders (#53)05790d8Add github.com/KarpelesLab/minlz-rs to ports (#52)df67e49search: Fix miscounting (#50)8ac7c53Don't skip uncompressed indexing when having a prefix (#51)cfcc325Add prefix only test (#49)eadfacdBump the github-actions group with 6 updates (#48)c8038d8Align assembly encoders with Go (#47)Updates
github.com/coreos/go-oidc/v3from 3.19.0 to 3.20.0Release notes
Sourced from github.com/coreos/go-oidc/v3's releases.
Commits
75dfa5coidc: add constants for "email" and "profile" scopesa89f046oidc: add API for determining when issuer URLs mismatch6a69b6dreadme: update README and docsf9049c9oidc: ignore JWKs with unknown signing algorithms rather than failing2f178e0SECURITY.md: add a security policy and point to project-level reportingb3bc7daoidc: improve documentation for APIs0db9053oidc: modernize with new Go APIsUpdates
golang.org/x/netfrom 0.56.0 to 0.57.0Commits
b8f09f6go.mod: update golang.org/x dependenciesf05f21bidna: reject all-ASCII xn-- labels on all Go versions0f748cfinternal/http3: clean up stream I/O methods usages in tests0bb961einternal/http3: add net/http.ResponseController support0ca694dwebdav: document Dir's lack of defense against filesystem modificationbd5f1dchttp2: initialize Transport on NewClientConn488ff63bpf: add security considerations to package docs93d1f25xsrftoken: avoid token collisions5a3baeeinternal/http3: prevent panic in QPACK decoder due to overflowMost Recent Ignore Conditions Applied to This Pull Request
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore <dependency name> major versionwill close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)@dependabot ignore <dependency name> minor versionwill close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)@dependabot ignore <dependency name>will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)@dependabot unignore <dependency name>will remove all of the ignore conditions of the specified dependency@dependabot unignore <dependency name> <ignore condition>will remove the ignore condition of the specified dependency and ignore conditions