Skip to content

feat: Added MessagesBatchStream. - #394

Open
ShogunPanda wants to merge 1 commit into
mainfrom
issue-388
Open

feat: Added MessagesBatchStream.#394
ShogunPanda wants to merge 1 commit into
mainfrom
issue-388

Conversation

@ShogunPanda

Copy link
Copy Markdown
Contributor

Summary

Adds MessagesBatchStream, a first-class way to consume messages in batches, closing #388 (and the earlier #56). It is an object-mode Duplex piped after a MessagesStream which groups messages by topic-partition and emits MessagesBatch objects, playing the same role as the KafkaJS eachBatch handler. It promotes the pattern previously only shown in examples/batching and used by the backpressure load tests, and mirrors the producer-side asStream batching with the same batchSize and batchTime options.

MessagesStream and consume() are unchanged: batching is purely a downstream stage, so the consumer stream typing stays simple and users keep full control over batch size independently of fetch size.

Changes

  • Added MessagesBatchStream with batchSize, batchTime and highWaterMark options (defaults 100, 10 ms, 16). Messages are buffered per topic-partition and a batch is emitted when a partition reaches batchSize or when batchTime elapses since the oldest buffered message; ending the writable side flushes everything so no message is lost when the consumer stream closes.
  • Backpressure is propagated to the source: when the readable side is above highWaterMark the write callback is held, so pipe() pauses the MessagesStream until batches are consumed. Destroying the stream releases the held write with the error.
  • Added the MessagesBatch type carrying topic, partition, messages, firstOffset, lastOffset, leaderEpoch, commit() and toJSON(). The batch commit is the commit of its last message, so it commits the offset following the batch and is a no-op when autocommit is enabled, while individual messages keep their own commit for partial acknowledgement.
  • Options are always validated in the constructor since the stream is not created through a client; the validation error formatting was extracted to formatValidationErrors in utils and Base now delegates to it.
  • Added the messages-batch-stream creation type to the diagnostics channel.
  • Documented the stream and the batch type, added a "Consuming in batches" section to the consumer guide, mapped eachBatch in the KafkaJS migration guide and rewrote examples/batching on top of the new class.

Notes

  • Breaking out of for await over the batch stream destroys both streams and the pipeline rejects with ABORT_ERR, as with any Node.js stream; the documentation recommends closing the consumer stream instead, which flushes the buffered batches.
  • After a rebalance, batches buffered for a partition which is no longer assigned can still be delivered for up to batchTime milliseconds.
  • The MessageBatchStream helper in test/helpers/backpressure.ts intentionally replicates a third-party stream and was left untouched; the backpressure tests could be switched to the shipped class in a follow-up.

Assisted-By: Anthropic:Claude Fable 5.1 <anthropic/claude-fable-5-1>

Fixes #338.
Fixes #56.

Signed-off-by: Paolo Insogna <paolo@cowtech.it>

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure this is usef

@prasadw

prasadw commented Sep 9, 2026

Copy link
Copy Markdown

when will it be available?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Requesting consumer.consumeBatch

3 participants