feat: Added MessagesBatchStream. - #394
Open
ShogunPanda wants to merge 1 commit into
Open
Conversation
Signed-off-by: Paolo Insogna <paolo@cowtech.it>
|
when will it be available? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
MessagesBatchStream, a first-class way to consume messages in batches, closing #388 (and the earlier #56). It is an object-modeDuplexpiped after aMessagesStreamwhich groups messages by topic-partition and emitsMessagesBatchobjects, playing the same role as the KafkaJSeachBatchhandler. It promotes the pattern previously only shown inexamples/batchingand used by the backpressure load tests, and mirrors the producer-sideasStreambatching with the samebatchSizeandbatchTimeoptions.MessagesStreamandconsume()are unchanged: batching is purely a downstream stage, so the consumer stream typing stays simple and users keep full control over batch size independently of fetch size.Changes
MessagesBatchStreamwithbatchSize,batchTimeandhighWaterMarkoptions (defaults 100, 10 ms, 16). Messages are buffered per topic-partition and a batch is emitted when a partition reachesbatchSizeor whenbatchTimeelapses since the oldest buffered message; ending the writable side flushes everything so no message is lost when the consumer stream closes.highWaterMarkthe write callback is held, sopipe()pauses theMessagesStreamuntil batches are consumed. Destroying the stream releases the held write with the error.MessagesBatchtype carryingtopic,partition,messages,firstOffset,lastOffset,leaderEpoch,commit()andtoJSON(). The batchcommitis the commit of its last message, so it commits the offset following the batch and is a no-op when autocommit is enabled, while individual messages keep their owncommitfor partial acknowledgement.formatValidationErrorsinutilsandBasenow delegates to it.messages-batch-streamcreation type to the diagnostics channel.eachBatchin the KafkaJS migration guide and rewroteexamples/batchingon top of the new class.Notes
for awaitover the batch stream destroys both streams and the pipeline rejects withABORT_ERR, as with any Node.js stream; the documentation recommends closing the consumer stream instead, which flushes the buffered batches.batchTimemilliseconds.MessageBatchStreamhelper intest/helpers/backpressure.tsintentionally replicates a third-party stream and was left untouched; the backpressure tests could be switched to the shipped class in a follow-up.Assisted-By: Anthropic:Claude Fable 5.1 <anthropic/claude-fable-5-1>
Fixes #338.
Fixes #56.