A compression engine that reshapes data before compressing it.
Transform pipeline · Pluggable codecs · Native C++20 codec · Express middleware · Browser decoder
Status: v0.x — the engine is the interesting part; see Honest status.
GitHub · npm · Issues · Design doc
Most compression libraries compete on the entropy coder. That race is over — zstd ships inside Node, brotli ships in every browser, and both have had a decade of tuning. Comprexia competes somewhere else: on the shape of the data before a codec ever sees it.
An array of similar JSON objects stores every field interleaved with unrelated fields, so a compressor never sees two similar values adjacent. Transpose it into columns and the same bytes compress dramatically better — using the same codec. That is the whole idea, and it is worth more than any match-finder change in this repository.
import{pack,unpack}from'comprexia/pack'constbody=Buffer.from(JSON.stringify({data: users}))constpacked=pack(body)// picks the transform and codec for youconstrestored=unpack(packed)// byte-identical to `body`Measured against the same codec with no transform. Reproduce with
npm run bench:engine, which verifies every result byte-exactly before
reporting it:
| Payload | gzip | engine + gzip | brotli | engine + brotli |
|---|---|---|---|---|
| API records, 2000 rows (284 kB) | 37,493 | 20,077 (−46%) | 31,629 | 16,063 (−49%) |
| API records, 200 rows (28 kB) | 4,047 | 2,417 (−40%) | 3,338 | 2,080 (−38%) |
| Multilingual records (387 kB) | 25,815 | 7,597 (−71%) | 18,941 | 5,883 (−69%) |
| Float32 telemetry (195 kB) | 154,441 | 106,095 (−31%) | 135,099 | 106,584 (−21%) |
| English prose (148 kB) | 24,809 | 24,823 (+0.1%) | 28,357 | 28,371 (+0.0%) |
Prose is the honest control: there is no structure to exploit, the engine detects that, applies no transform, and costs you the 14-byte header. It never makes a payload larger than the codec would have on its own, because the untransformed candidate is always in the running and the smallest one wins.
The last transform in this codebase assumed it was reversible and corrupted every non-ASCII payload it touched. This one assumes nothing:
- Every transform is verified at pack time. The engine runs the inverse and compares byte-for-byte before emitting. A transform that fails is silently dropped and the next candidate is used.
- Non-canonical JSON is declined, not rewritten. Pretty-printed input,
1.0,1e2, and\u0041all survive a parse but not a re-stringify, so the columnar transform refuses them rather than quietly changing the bytes. - The container records everything. Transform id, codec id, original length,
and a checksum over the original bytes.
unpacknever guesses, and a transform that fails to invert is caught rather than returned.
| Transform | Applies to | What it does |
|---|---|---|
columnar-json | arrays of ≥4 objects with identical keys | transposes rows into columns, then delta-encodes integer columns where that is smaller |
byte-shuffle | fixed-width numeric series (opt in with shuffleWidth) | groups the Nth byte of every element, collapsing near-identical exponent planes |
none | everything else | passthrough |
zstd when the runtime has it (Node 23.8+), brotli, gzip, and cx — this
project's own native codec. Pick one with { codec: 'brotli' } or let the
engine choose. The transform is the value; the codec stays replaceable.
Packing runs roughly 4× slower than calling the codec directly, because it
builds candidates, verifies the inverse, and compresses more than once.
{ verify: false } trades the correctness guarantee for speed — not
recommended for data you did not produce. Unpacking is a normal decompress plus
one linear pass.
Most compression libraries open with a benchmark that flatters them. This one opens with the benchmark that does not.
The engine is the part worth using. The bundled codec is not yet competitive on ratio. Those are two separate claims and they deserve separate treatment.
The engine beats gzip and brotli by 21–71% on structured data, using those same codecs underneath — that result does not depend on the C++ codec at all, and works on a runtime with no native addon.
The C++ codec compresses 2–6× faster than gzip -1 and decompresses 2–5×
faster, but gzip -1 still produces smaller output on every dataset tested.
Until an entropy stage lands, that is the honest trade, and it is why the engine
defaults to zstd or brotli rather than to cx.
An earlier version of this README quoted 1206 MB/s. That figure came from a benchmark that concatenated one sample file until it reached 1 MB — near-pure repetition, which every LZ codec devours. It was not a lie so much as a measurement of nothing. It has been replaced.
What is genuinely here today:
- A correct codec on every path. Arbitrary bytes — UTF-8, Devanagari, emoji, binary, every value 0x00–0xFF — round-trip byte-exactly through all three encoders, fuzzed under AddressSanitizer and UBSan on every commit.
- A hardened decoder. Malformed streams raise a catchable error instead of reading out of bounds, in both the native and browser decoders.
- Throughput well ahead of gzip: 2–6× faster to compress and 2–5× faster to decompress at every payload size tested. Ratio is the trade — see below.
- Working Express middleware, Node streams, and a browser decoder small enough to inline.
What is not here: a competitive compression ratio or competitive encode speed. See Known limitations — they are documented rather than hidden, and docs/DESIGN.md is the plan for fixing them.
Use the engine if you serve structured JSON and control both ends. Use the C++ codec if you want to read, learn from, or contribute to a compression codec — not yet to serve production traffic.
- The engine
- Honest status
- Benchmarks
- Known limitations
- Installation
- Quick start
- API reference
- Integrations
- Stream format
- Architecture
- When to use what
- Roadmap
- Building from source
- Security
- Contributing
- License
This section measures the bundled C++ codec against other codecs. For the engine's numbers — which are the interesting ones — see The engine.
Node 20, Apple Silicon, deterministic synthetic payloads that resemble real API traffic rather than repeated blobs. Competitors run at the settings servers actually deploy — gzip 6 and brotli 4 for dynamic responses, not brotli 11. Every result is verified byte-exact before it is timed. Reproduce with:
npm run bench:honestAPI list response — 1000 user records, 189 kB
| Codec | Ratio | Saved | Compress MB/s | Decompress MB/s |
|---|---|---|---|---|
| comprexia-fast | 0.197 | 80.3% | 919 | 380 |
| gzip-1 | 0.148 | 85.2% | 465 | 178 |
| gzip-6 | 0.114 | 88.6% | 133 | 170 |
| brotli-4 | 0.122 | 87.8% | 232 | 126 |
| brotli-11 | 0.086 | 91.4% | 1 | 98 |
| lz4 | 0.211 | 78.9% | 1420 | 915 |
Multilingual JSON — Hindi, Bengali, Tamil, emoji, 355 kB
| Codec | Ratio | Saved | Compress MB/s | Decompress MB/s |
|---|---|---|---|---|
| comprexia-fast | 0.095 | 90.5% | 2023 | 332 |
| gzip-1 | 0.047 | 95.3% | 1069 | 113 |
| brotli-4 | 0.038 | 96.2% | 530 | 110 |
| lz4 | 0.064 | 93.6% | 3634 | 625 |
Small API response — 1.5 kB
| Codec | Ratio | Saved | Compress MB/s | Decompress MB/s |
|---|---|---|---|---|
| comprexia-fast | 0.456 | 54.4% | 780 | 546 |
| gzip-1 | 0.328 | 67.2% | 123 | 97 |
| brotli-4 | 0.296 | 70.4% | 65 | 86 |
| lz4 | 0.432 | 56.8% | 868 | 1136 |
Read honestly: comprexia compresses 2–6× faster than gzip level 1 on every dataset and decompresses 2–5× faster, after the encoder rewrite and the wildcopy decoder. Small responses — the worst case at 81 MB/s before — improved roughly 9× to 780 MB/s.
What has not changed is the ratio, and it is still the honest weak spot: gzip level 1 produces smaller output than comprexia on every dataset here, at a fraction of the speed. LZ4 also remains ahead on both throughput axes (about 1.5× on compression, 2.4× on decompression) at a comparable ratio.
Both gaps have known causes rather than mysterious ones. The ratio needs an entropy coding stage, which is what separates LZ4 from gzip and zstd. The remaining decode gap is per-block dispatch overhead, not the copy itself — short matches mean the decoder spends its time branching, which is what the v2 format's token layout is designed to reduce. See docs/DESIGN.md.
Honest boundaries of the current release. Each is a limitation of scope, not a
correctness bug — the correctness bugs that used to live here are fixed and
locked down by test/node/defects.test.js.
No prebuilt binaries.npm install compiles C++ on the consumer's machine,
so anyone without CMake and a C++20 toolchain cannot install the package at all.
This is the single biggest barrier to adoption and the top roadmap item.
No container framing. The stream carries no magic number, version, or checksum, so corruption is undetectable and the format cannot evolve without breaking deployed decoders. Fixed by the v2 container.
Streaming does not match across chunk boundaries.createCompressorStream
restarts its match search for each chunk, so repeated structure between
messages is not exploited. Correct, but it leaves ratio on the table for event
streams — exactly the workload streaming exists for.
The ratio is not competitive. See Benchmarks. This is the honest state of a hand-written LZ with no entropy coding stage — the missing piece is Huffman or FSE over literals and lengths, not more match-finder tuning.
Decompression trails LZ4 by ~2.4×. The wildcopy decoder closed part of the gap; what remains is per-block dispatch, since short matches make the decoder spend its time branching rather than copying.
- The streaming encoder produced output its own decoder rejected. It was a second, independent implementation of the block format and never received the 130-byte fix below, so any stream containing a repeat of that length either threw or silently decoded to wrong bytes. Both encoders now write blocks through one shared definition, and the fuzz harness exercises the streaming path across several chunk sizes.
- Decompression was unbounded. Five bytes of extended match block emit up to
65535, so 100 kB of crafted input decoded to 1.3 GB and could exhaust a
server's memory. All decode entry points now accept
maxOutputLengthand default to 256 MB. level: 'advanced'sent undecodable responses. It labelled its outputContent-Encoding: cx— the same coding as the fast format, which reverses a different transform — so clients silently decoded corrupt JSON. Advanced payloads now usecx-adv, negotiated separately, and the browser decoder gaineddecompressAdvancedToStringto read them.- A failed send left
Content-Encodingon a plain body. The middleware set headers before compressing, so falling back to uncompressed JSON kept the encoding header committed. Accept-Encoding: cx;q=0, *was read as acceptance. Per RFC 9110 §12.5.3 a wildcard only covers codings not explicitly listed, so an explicit refusal now wins regardless of ordering.
- A 130-byte match silently corrupted data. Short match blocks encode
len - 3in seven bits, solen == 130emitted header0xFF— which is the extended-match marker. The decoder misread it and returned wrong bytes, or threw, depending on what followed. Random fuzzing never happened to land on the single length that breaks; a systematic sweep of every match length from 1 to 320 now runs on every commit. Short blocks stop at 129, which keeps0xFFunambiguous and leaves the format readable by older decoders.
- Advanced mode corrupted all non-ASCII data. Three separate bugs — a token range colliding with UTF-8 lead bytes, a delta transform that was not invertible, and a JSON string scanner that mishandled escaped backslashes. The transform was rebuilt to be byte-exact and escape-safe; it no longer parses JSON at all, which is why it can be fuzzed against arbitrary bytes.
- The decoder read out of bounds on crafted streams. A back-reference distance larger than the output produced a buffer underflow — reachable from any untrusted response body. Every field is now validated, and both the native and browser decoders raise a catchable error instead.
- The middleware omitted
Vary: Accept-Encoding, letting a shared cache serve acxbody to a client that cannot decode it. negotiateEncodingsubstring-matchedcx, socxfuturewas a false positive andcx;q=0— an explicit refusal — was treated as support.
npm install comprexiaRequires Node 18+, a C++20 compiler, and CMake 3.20+ — the native addon is compiled during install. Verified in CI on Linux (Node 18, 20, 22), macOS, and Windows.
Prebuilt binaries are the top roadmap item; until they land, treat the toolchain requirement as a hard install dependency.
import{compress,decompress}from'comprexia'constoriginal=Buffer.from(JSON.stringify({id: 1,name: 'संजीव'}))constpacked=compress(original)constrestored=decompress(packed)restored.equals(original)// true — for any input bytes| Function | Description |
|---|---|
compress(input: Buffer): Buffer | Default encoder. Longer match extension, better ratio. |
compressFast(input: Buffer): Buffer | Speed-oriented encoder, shorter match cap. Used by the middleware. |
decompress(input, options?) | Decodes output from either encoder above. options.maxOutputLength caps the result, defaulting to 256 MB. |
compressAdvanced(input: Buffer): Buffer | Applies the substitution transform before compressing. ~5% smaller than compress when common keys appear; neutral otherwise. |
decompressAdvanced(input, options?) | Reverses compressAdvanced. Not interchangeable with decompress. |
createCompressorStream(): Transform | Node Transform stream for chunked responses. |
negotiateEncoding(header?: string): 'cx' | undefined | Parses Accept-Encoding per RFC 9110 — whole tokens, q=0, explicit entries outranking *. |
acceptsCoding(header, coding): boolean | The same negotiation for any coding, e.g. cx-adv. |
createComprexiaMiddleware(opts?) | Express middleware. opts.level is 'fast' (default) or 'advanced'. |
comprexia/web/decoder | Browser decoder — decompressToString, decompressAdvancedToString, decompressBrowser, decompressAdvancedBrowser, ComprexiaDecodeError. |
compress and compressFast emit the same stream format, so a single
decompress reads both. compressAdvanced does not — it applies a transform
that only decompressAdvanced reverses. Because the format carries no version
byte, nothing detects that mismatch for you; pair them correctly.
All decode entry points throw on malformed input rather than returning partial
or garbage data, so wrap them in try/catch when the bytes come from
somewhere you do not control. They also bound their output at 256 MB by
default: five bytes of extended match block expand to 65535, so an unbounded
decoder turns a small hostile body into gigabytes. Pass
{ maxOutputLength: 0 } to lift the cap, and only for input you produced.
importexpressfrom'express'import{createComprexiaMiddleware}from'comprexia'constapp=express()app.use(createComprexiaMiddleware({level: 'fast'}))app.get('/api/posts',(_req,res)=>{res.json({success: true,data: [{id: 1,title: 'hello'}]})})app.listen(3001)When the client sends Accept-Encoding: cx, the response carries:
| Header | Meaning |
|---|---|
Content-Encoding: cx | Body is a comprexia stream |
X-Compression-Ratio | Compressed ÷ original, 3 decimal places |
X-Original-Size | Bytes before compression |
X-Compressed-Size | Bytes on the wire |
Vary: Accept-Encoding is set automatically, so shared caches key the response
correctly. Clients that do not advertise cx fall through to the original
res.json, leaving existing gzip middleware untouched.
importFastifyfrom'fastify'import{compressFast,negotiateEncoding}from'comprexia'constapp=Fastify()app.decorateReply('cxJson',function(payload: unknown){if(negotiateEncoding(this.request.headers['accept-encoding'])!=='cx'){returnthis.send(payload)}returnthis.header('Content-Encoding','cx').header('Content-Type','application/json').header('Vary','Accept-Encoding').send(compressFast(Buffer.from(JSON.stringify(payload))))})app.get('/api/posts',async(_req,reply)=>(replyasany).cxJson({data: []}))app.listen({port: 3002})import{Injectable,NestMiddleware}from'@nestjs/common'import{compressFast,negotiateEncoding}from'comprexia'
@Injectable()exportclassComprexiaMiddlewareimplementsNestMiddleware{use(req: any,res: any,next: ()=>void){constoriginalJson=res.json.bind(res)res.json=(body: unknown)=>{if(negotiateEncoding(req.headers['accept-encoding'])!=='cx'){returnoriginalJson(body)}res.setHeader('Content-Encoding','cx')res.setHeader('Content-Type','application/json')res.setHeader('Vary','Accept-Encoding')returnres.send(compressFast(Buffer.from(JSON.stringify(body))))}next()}}const{ negotiateEncoding, createCompressorStream }=require('comprexia')app.get('/events',(req,res)=>{if(negotiateEncoding(req.headers['accept-encoding'])!=='cx'){returnres.json({error: 'cx encoding required'})}res.setHeader('Content-Encoding','cx')conststream=createCompressorStream()stream.pipe(res)stream.write(Buffer.from(JSON.stringify({type: 'init'})))consttimer=setInterval(()=>{stream.write(Buffer.from(JSON.stringify({type: 'tick',t: Date.now()})))},1000)req.on('close',()=>{clearInterval(timer)stream.end()})})Each chunk is matched independently — repeated structure between messages is
not yet exploited, so streaming trades ratio for incrementality. See
Known limitations. Always clear timers on close; the
example in earlier docs leaked an interval per connection.
importaxiosfrom'axios'import{decompressToString}from'comprexia/web/decoder'constapi=axios.create({baseURL: '/api'})asyncfunctionfetchJson<T>(path: string): Promise<T>{constres=awaitapi.get(path,{responseType: 'arraybuffer',headers: {'Accept-Encoding': 'cx'},})if(res.headers['content-encoding']==='cx'){returnJSON.parse(decompressToString(res.data))}returnJSON.parse(newTextDecoder().decode(newUint8Array(res.data)))}The browser decoder is dependency-free and mirrors the native decoder,
including its validation and its 256 MB output cap. Match the function to the
coding the server sent: Content-Encoding: cx decodes with
decompressToString, and cx-adv with decompressAdvancedToString. The two
formats are not interchangeable and nothing in the bytes distinguishes them —
that is exactly why advanced payloads carry their own coding.
Note: browsers control the real
Accept-Encodingheader onfetch/XHRand will strip a manual override. In practice you negotiatecxwith a custom header or a query parameter, or you use this in a non-browser client.
The v0.1 format is a bare sequence of blocks with no container, no version, and no checksum:
| Block | Header | Payload |
|---|---|---|
| Literal | 0x00–0x7F = byte count | that many literal bytes |
| Match | 0x80 | (len - 3), len ≤ 130 | 2-byte distance, little-endian |
| Extended match | 0xFF | 2-byte length, then 2-byte distance |
Window size is 64 kB (16-bit distances). Minimum match is 4 bytes.
The absence of framing is a real design flaw, not a simplification: corruption is undetectable, and the format cannot evolve without silently breaking deployed decoders. The v2 container fixes this with a magic number, a version byte, a feature-flag byte, and a checksum over the original data.
src/cx_core/
encoder.cpp LZ77 match finder + block emitter
decoder.cpp literal/match replay
preprocessor.cpp JSON tokenizer and UTF-8 transforms (advanced mode)
stream.cpp chunked encoder state
src/cx_bindings/
addon.cc N-API surface
node/ TypeScript wrapper, middleware, browser decoder
test/cpp/ ASan/UBSan roundtrip fuzz harness
The encoder hashes 4-byte sequences into a flat, power-of-two table sized to the
input, keeping one candidate position per slot and extending matches eight bytes
at a time. compress extends up to 258 bytes and uses extended match blocks for
long repeats; compressFast caps at 64 for tighter inner loops. Both write
blocks through block_format.h, the single definition of the wire format — it
exists because the format was once implemented twice and the copies drifted into
a data-corrupting disagreement.
The decoder replays literal runs and back-references with no entropy stage,
which is why the ratio is mediocre — speed and ratio are the same trade here.
Matches are copied in fixed 8-byte steps into a buffer that always keeps a slack
margin past its logical end, so the copy loop never tests a per-byte condition.
That requires a raw buffer rather than std::vector, since writing past size()
is undefined and AddressSanitizer flags it.
Sizing the hash table to the input matters more than it looks: a fixed 64 k-entry table costs a 256 kB clear, which for a 1.5 kB API response is far more work than the compression itself.
Every codec change is compiled with AddressSanitizer and UndefinedBehaviorSanitizer
in CI and run against a deterministic fuzz harness covering random, repetitive,
JSON, and multilingual inputs. That gate has already caught real bugs: an
unaligned uint32_t load in the match finder, and missing standard includes that
made the package fail to compile on Linux entirely.
| Situation | Use |
|---|---|
| Serving production API traffic today | gzip level 1–6, built into Node |
| Maximum ratio for static assets | brotli level 11 |
| Maximum throughput, ratio secondary | lz4 |
| Modern Node with the best all-round balance | zstd — built into node:zlib since Node 23.8 |
| Many small responses sharing a schema | zstd or brotli with a trained dictionary |
| Learning how an LZ codec works end to end | this repository |
Being clear about this costs nothing and is the whole point of publishing real numbers. Comprexia earns a place on that list when the v2 format lands, not before.
The direction is set by where the ecosystem actually has a gap, not by trying to
out-tune LZ4. Zstd and brotli both ship in Node now; competing with them on
general-purpose ratio is a losing race. Dictionary compression is the open
niche — a trained dictionary can take a small JSON payload from roughly 32% of
original size down to under 10%, which is exactly where every general-purpose
codec (including this one, at 0.454) is weakest. The
Compression Dictionary Transport
standard shipped in Chrome 130 with the dcb and dcz encoding tokens, and the
Node ecosystem has no middleware for it.
- M1 — v2 container and LZ core. Framed format with version and checksum, flat-array match finder replacing the hash map, fully bounds-checked decoder.
- M2 — browser decoder parity. Shared test vectors between native and JS.
- M3 — dictionary support. Train a dictionary from sample payloads, ship it to clients, negotiate it over HTTP.
- M4 — prebuilt binaries.
npm installwith no toolchain.
Full rationale, format sketches, and the design rules derived from each v0.1 defect are in docs/DESIGN.md.
npm install # installs deps and compiles the addon
npm run build # TypeScript → dist/
npm run build:release # native addon, Release mode
npm test# roundtrip + stream tests
npm run lint # eslint
npm run typecheck # tsc --noEmit
npm run bench:honest # realistic benchmark suiteSanitizer harness, the gate that matters for codec changes:
g++ -std=c++20 -O1 -g -fsanitize=address,undefined -fno-sanitize-recover=all \
-Iinclude -Isrc/cx_core \
test/cpp/roundtrip_fuzz.cpp src/cx_core/*.cpp -o roundtrip_fuzz
./roundtrip_fuzzThe decoder parses attacker-controllable bytes by design. Report vulnerabilities privately — see SECURITY.md. Never open a public issue for one.
Two operational notes that apply to every compressor, not just this one:
- Do not compress secrets mixed with attacker-controlled input over a channel an attacker can measure. That is the BREACH/CRIME class of attack, and compression is an optimization, never a security boundary.
- Bound your inputs. A decompressor turns small inputs into large outputs by definition; cap the accepted compressed size at your edge.
Contributions are welcome — see CONTRIBUTING.md and the Code of Conduct.
The bar for codec changes: the sanitizer harness passes, roundtrip tests cover non-ASCII input, and any format change updates the native decoder, the browser decoder, and the format table in this README together. Commits follow Conventional Commits — releases are cut automatically from them.
Good first issues live in docs/DESIGN.md: every defect listed there is a well-specified, self-contained fix.
MIT © WebCoderSpeed