Skip to content

feat: streaming encoder (io::Write wrapper) #9

Description

@polaz

Summary

The current encoder requires all input data upfront via Read trait. There is no streaming encoder that implements io::Write — callers cannot incrementally feed data and get compressed output. This is required for network streams, large files, and pipe-based workflows.

Current state

  • FrameCompressor::compress() reads all data in a loop until EOF
  • No Write-based wrapper analogous to StreamingDecoder for decompression
  • CLI buffers entire input before compression

C reference implementation

Streaming compression API

ZSTD_compressStream2(cctx, &outBuf, &inBuf, endDirective)
// endDirective: ZSTD_e_continue, ZSTD_e_flush, ZSTD_e_end
  • ZSTD_e_continue — accumulate input, compress when block is full
  • ZSTD_e_flush — flush current data (partial block)
  • ZSTD_e_end — finalize frame with last block flag + optional checksum

Key design patterns

  • Input buffer accumulation until block size reached (128KB max)
  • Partial flush support for latency-sensitive use cases
  • Frame finalization separate from data feeding

What needs to be implemented

  1. StreamingEncoder<W: Write> — wraps a Write drain
  2. impl Write for StreamingEncoder — accumulates input, emits compressed blocks when buffer full
  3. flush() method — flush partial block (non-last)
  4. finish() method — write last block + checksum, return inner writer
  5. Frame header — written on first write() call
  6. Block buffering — accumulate up to 128KB before compressing block

Acceptance criteria

  • StreamingEncoder implements io::Write
  • Can compress data incrementally (multiple write() calls)
  • finish() properly closes the frame
  • Interop: C zstd decompresses streaming-encoded output
  • Roundtrip: StreamingEncoder → StreamingDecoder works
  • CLI uses streaming encoder for pipe support

Time estimate

2d

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1-highHigh priority — core functionalityenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions