Decompress: reuse the caller-supplied buffer for unknown-size frames

[KrisKennawayDD]

Problem

Decompress sizes its destination buffer from decompressSizeHint(src), which reads the frame's content-size field. Frames that don't carry that field make the hint fall back to a pessimistic upper bound (max(50×len(src), decompressSizeBufferLimit) — at least 1 MB). This happens for:

• legacy zstd v0.5 frames (ZSTD_getFrameContentSize reports unknown), and
• streaming frames compressed without a pledged source size.

When a caller passes a smaller-but-adequate buffer, Decompressdiscards it (cap(dst) < bound) and allocates the full bound. So every decode of such a frame allocates at least 1 MB regardless of the real payload size — a significant per-call heap/GC cost for callers decoding many small unknown-size payloads with a pooled/hinted buffer.

Fix

decompressSizeHint now also returns foundHint — whether the decompressed size was actually read from the frame header.

Decompress and ctx.Decompress select the destination buffer with a single switch, make one DecompressInto, and fall back to the streaming reader only if that buffer is too small:

• cap(dst) >= hint — reuse the caller buffer (unchanged from before).
• !foundHint && cap(dst) > 0 — size unknown, so reuse the caller buffer as-is rather than allocating the pessimistic bound.
• otherwise — allocate hint (the exact size when known; the bound when unknown and no caller buffer was supplied).

Consequences:

• Known-size frames (the common case where the same zstd version compressed and decompressed): behavior is unchanged, and a caller passing a too-small buffer does not pay for a failed DecompressInto.
• Unknown-size frames with an adequate caller buffer: no more ≥1 MB allocation — the caller buffer is used.
• Unknown-size frames with a too-small (or no) caller buffer: fall back to the streaming reader instead of allocating the ≥1 MB bound.

The same change is applied to ctx.Decompress; BulkProcessor.Decompress is adapted to the new signature with its behavior preserved.

Regression tests cover caller-buffer reuse (unknown- and known-size), the too-small fallback, nil dst, and the foundHint signal — run as subtests against both Decompress and ctx.Decompress.

🤖 Generated with Claude Code

[Viq111]

we should make it the same logic here to be consistent ?

[KrisKennawayDD]

Added a comment about why this diverges

[Viq111]

✅
(I've disabled circleCI for future runs / PR since we moved to github actions)

[pr-shepherd-6ad11f]

PR Shepherd is now watching this PR: fixing basic CI failures, rebasing when it falls behind, and re-queueing after transient merge-queue failures. To disable it, add the pr-shepherd:ignore label.

If you have any questions, reach the team in #ai-devx-flow.