You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
node:zlib's zstd functions accept a dictionary option and drop it. On compression
that's silent, and on decompression it surfaces as a corruption error against bytes
that aren't corrupt.
Node grew this option in nodejs/node@201304537e
and workerd's zstd bindings (#6007) predate it, so I assume this is a gap rather than a
regression.
Versions
workerd 1.20260801.1, via wrangler 4.120.0 / miniflare 5.20260801.1-alpha. compatibility_date = "2026-08-01", compatibility_flags = ["nodejs_compat"].
Compared against node v26.7.0 running the identical program.
Repro
import{zstdCompressSync,zstdDecompressSync,constants}from"node:zlib";// A frame produced by node with a dictionary, base64'd in so the worker can try to read it.constFRAME="<node's dictionary-compressed output, see below>";exportdefault{fetch(){constprobe=Buffer.from("the quick brown fox jumps over the lazy dog ".repeat(200));constwrong=Buffer.from("completely unrelated filler bytes ".repeat(200));constlvl={[constants.ZSTD_c_compressionLevel]: 19};constout={};out.compress={none: zstdCompressSync(probe,{params: lvl}).length,good: zstdCompressSync(probe,{dictionary: probe,params: lvl}).length,wrong: zstdCompressSync(probe,{dictionary: wrong,params: lvl}).length,};try{constback=zstdDecompressSync(Buffer.from(FRAME,"base64"),{dictionary: probe});out.decompress={ok: true,bytes: back.length,matches: back.equals(probe)};}catch(e){out.decompress={ok: false,error: String(e?.message??e)};}returnResponse.json(out);},};
Generate FRAME under node:
const{ zstdCompressSync, constants }=require("node:zlib");constprobe=Buffer.from("the quick brown fox jumps over the lazy dog ".repeat(200));zstdCompressSync(probe,{dictionary: probe,params: {[constants.ZSTD_c_compressionLevel]: 19}}).toString("base64");// 19 bytes
Results
node v26.7.0
workerd 1.20260801.1
compress, no dictionary
63
63
compress, correct dictionary
19
63
compress, wrong dictionary
63
63
decompress a dictionary-compressed frame
ok, 8800 bytes, byte-exact
throws
workerd's decompress error:
Zstd decompression failed: Data corruption detected
Why this isn't just a missing-feature note
On compress, nothing tells you. A frame compressed without a dictionary decodes
perfectly with one, so no consumer errors and no log line looks unusual. The only
signal is a byte count that never shrank. The only reliable detection I've found is a
feature-detect that compresses a buffer against itself and asserts the output collapses.
On decompress, the error blames the wrong thing. "Data corruption detected" points
at the payload. The payload is fine. Anyone debugging that will go looking at their
storage or their transport before they suspect that the option they passed was dropped.
The practical consequence is that a Worker can't participate in RFC 9842 Compression Dictionary Transport
in either direction: it can't produce a dcz delta, and it can't read one. That's a
shame specifically for Workers, since computing a delta against a dictionary the client
already holds is an edge-shaped job.
Suggested fix
Wire ZSTD_CCtx_loadDictionary / ZSTD_DCtx_loadDictionary (and the _refPrefix
variants, if you want raw-content dictionaries, which is what RFC 9842 uses) into the
bindings from Add ZSTD bindings to the node:zlib package. #6007, matching node's implementation linked above.
If that's not near-term, please throw on a dictionary option the binding can't
honour, rather than accepting it. ERR_INVALID_ARG_VALUE at the call site is far
cheaper to debug than either symptom above.
I'm more than happy to take a stab via a PR for (2) if that's a useful stopgap.
oven-sh/bun#34427, where bun had the
identical silent-ignore for both brotli and zstd and fixed it by loading the dictionary
into the context. Its test additions may be a useful reference.
node:zlib's zstd functions accept adictionaryoption and drop it. On compressionthat's silent, and on decompression it surfaces as a corruption error against bytes
that aren't corrupt.
Node grew this option in nodejs/node@201304537e
and workerd's zstd bindings (#6007) predate it, so I assume this is a gap rather than a
regression.
Versions
workerd
1.20260801.1, via wrangler4.120.0/ miniflare5.20260801.1-alpha.compatibility_date = "2026-08-01",compatibility_flags = ["nodejs_compat"].Compared against node
v26.7.0running the identical program.Repro
Generate
FRAMEunder node:Results
workerd's decompress error:
Why this isn't just a missing-feature note
On compress, nothing tells you. A frame compressed without a dictionary decodes
perfectly with one, so no consumer errors and no log line looks unusual. The only
signal is a byte count that never shrank. The only reliable detection I've found is a
feature-detect that compresses a buffer against itself and asserts the output collapses.
On decompress, the error blames the wrong thing. "Data corruption detected" points
at the payload. The payload is fine. Anyone debugging that will go looking at their
storage or their transport before they suspect that the option they passed was dropped.
The practical consequence is that a Worker can't participate in
RFC 9842 Compression Dictionary Transport
in either direction: it can't produce a
dczdelta, and it can't read one. That's ashame specifically for Workers, since computing a delta against a dictionary the client
already holds is an edge-shaped job.
Suggested fix
ZSTD_CCtx_loadDictionary/ZSTD_DCtx_loadDictionary(and the_refPrefixvariants, if you want raw-content dictionaries, which is what RFC 9842 uses) into the
bindings from Add ZSTD bindings to the node:zlib package. #6007, matching node's implementation linked above.
dictionaryoption the binding can'thonour, rather than accepting it.
ERR_INVALID_ARG_VALUEat the call site is farcheaper to debug than either symptom above.
I'm more than happy to take a stab via a PR for (2) if that's a useful stopgap.
Related
isn't triaged as a duplicate of it
identical silent-ignore for both brotli and zstd and fixed it by loading the dictionary
into the context. Its test additions may be a useful reference.