⚠️ DRAFT / proposal. Filed while the design is fresh. Schedule when a consumer needs incremental / resumable partial decode (lsm-tree #257 currently works around it — see below).
Feature gate: all code would be #[cfg(feature = "lsm")] (default off), same as #173 / #175.
Context
lsm-tree #257 reads only the inner zstd blocks a key range touches from a large cold block (decode [0, end_block) covering the range, skip the trailing blocks).
Finding: decode_blocks_partial is NOT incrementally resumable
Implementing the consumer surfaced this: decode_blocks_partial drains the in-range output from the match window on return. Two consecutive calls on the same decoder therefore cannot resume one another — the second call's first block fails match resolution against the (now-drained) earlier blocks:
decode_blocks_partial(0, 1) // OK: decodes block 0, returns + drains it
decode_blocks_partial(1, k) // ERR: NotEnoughBytesInDictionary { got: 0, need: N }
// block 1 needs block 0 in the window — drained
A single decode_blocks_partial(0, end_block) call is correct (the window is maintained until the final drain). But growing a decoded extent means re-decoding the covering prefix from block 0 — O(extent) per growth, O(N²) for a block-by-block forward walk. This holds whether the decoder is kept live (warm) or recreated (cold); there is no incremental path today.
lsm-tree #257 works around it by decoding once to the query's known upper bound per access (fine for a range scan with a known bound, but no cheap top-up / streaming).
Proposal
Provide a resumable entry point. Either (or both):
(a) Window-priming resume — caller supplies the already-decompressed tail to prime the window, then continues at block N without re-decoding the prefix:
#[cfg(feature = "lsm")]
impl FrameDecoder {
/// Resume partial decoding at inner block `block_index` WITHOUT
/// re-decompressing the preceding blocks. `window_prime` is the caller's
/// already-decompressed tail (>= the frame's `window_size` bytes ending
/// just before `block_index`'s output); it is loaded into the match window.
/// The caller positions `source` at the block's compressed frame offset
/// (`FrameEmitInfo::blocks[block_index].offset_in_frame`, shipped in #173).
/// `decode_blocks_partial(block_index, end)` then decodes only `[N, end)`.
pub fn resume_at(&mut self, block_index: u32, window_prime: &[u8])
-> Result<(), FrameDecoderError>;
}(b) Non-draining decode mode — a flag/variant where in-range output is RETAINED in the window (not drained), so a subsequent decode_blocks_partial(N, ..) on the same live decoder resumes natively, no caller priming. Covers the warm (kept-live decoder) path; (a) then only serves the cold (dropped-decoder) path.
Consumer prerequisite (persisted by the consumer, not zstd)
For (a), to position source at block N the consumer persists each inner block's compressed offset_in_frame alongside the decompressed ends it already stores. offset_in_frame is already on FrameEmitInfo.blocks[*] (#173) — no new write-side field; this issue is decode-side only.
Acceptance criteria
Relationship
Context
lsm-tree #257 reads only the inner zstd blocks a key range touches from a large cold block (decode
[0, end_block)covering the range, skip the trailing blocks).Finding:
decode_blocks_partialis NOT incrementally resumableImplementing the consumer surfaced this:
decode_blocks_partialdrains the in-range output from the match window on return. Two consecutive calls on the same decoder therefore cannot resume one another — the second call's first block fails match resolution against the (now-drained) earlier blocks:A single
decode_blocks_partial(0, end_block)call is correct (the window is maintained until the final drain). But growing a decoded extent means re-decoding the covering prefix from block 0 — O(extent) per growth, O(N²) for a block-by-block forward walk. This holds whether the decoder is kept live (warm) or recreated (cold); there is no incremental path today.lsm-tree #257 works around it by decoding once to the query's known upper bound per access (fine for a range scan with a known bound, but no cheap top-up / streaming).
Proposal
Provide a resumable entry point. Either (or both):
(a) Window-priming resume — caller supplies the already-decompressed tail to prime the window, then continues at block
Nwithout re-decoding the prefix:(b) Non-draining decode mode — a flag/variant where in-range output is RETAINED in the window (not drained), so a subsequent
decode_blocks_partial(N, ..)on the same live decoder resumes natively, no caller priming. Covers the warm (kept-live decoder) path; (a) then only serves the cold (dropped-decoder) path.Consumer prerequisite (persisted by the consumer, not zstd)
For (a), to position
sourceat blockNthe consumer persists each inner block's compressedoffset_in_framealongside the decompressed ends it already stores.offset_in_frameis already onFrameEmitInfo.blocks[*](#173) — no new write-side field; this issue is decode-side only.Acceptance criteria
N(via (a) priming or (b) a prior non-draining call),decode_blocks_partial(N, end)yields bytes byte-identical to a full decode's[ends[N-1]..ends[end-1])slice, forNin{1, mid, last}.< Nare not re-decoded on resume.window_primeshorter than the required window is rejected with a typed error, not silent mis-decode.lsm; default build byte-identical, no new public symbols.Relationship
FrameEmitInfo, incl.offset_in_frame) and perf+feat(decoding): block-subset partial decode (range + recovery) + per-block decompressed byte ranges #175 (decode_blocks_partial).