Uh oh!
There was an error while loading. Please reload this page.
[GSoC 2026] Kafka Streams runner: separate source poll size from bundle size, expose the session timeout - #39748
Merged
Conversation
…m the bundle size, and expose the session timeout ReadTranslator took the unbounded source's per-poll limit from --maxBundleSize. They are different concerns: a small bundle is how output arrives promptly, while how much a source reads at a time is about throughput. Sharing one setting means a pipeline cannot have both, since lowering the bundle size to get prompt output also throttles the source. --sessionTimeoutMs exposes how long the consumer group waits before deciding an instance has gone, which is the floor on how quickly its work moves elsewhere. Kafka's 45s default is kept. The heartbeat is derived as a third of it rather than exposed separately, because Kafka rejects a heartbeat that is not shorter than the timeout and deriving it keeps the pair consistent.
je-ik
merged commit Aug 14, 2026
511a40e
into
apache:feat/18479-kafka-streams-runner-skeleton
2 checks passed
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Aug 14, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #18479. Two pipeline options that came out of trying to measure how the runner behaves under load, and that are worth having whatever the measurement ends up looking like.
How much a source reads is not how big a bundle is
ReadTranslatortook the unbounded source's per-poll limit from--maxBundleSize. They are different concerns: a small bundle is how output arrives promptly, while how much a source reads at a time is about throughput. Sharing one setting means a pipeline cannot ask for both — lowering the bundle size to get prompt output also throttles the source to a couple of elements per poll.--readMaxElementsPerPoll(default 1000, what the source effectively had before) separates them.The recovery knob was unreachable
How quickly work moves to another instance after one is lost is bounded by how quickly the consumer group notices, which is
session.timeout.ms. The runner never set it, so it was Kafka's default of 45 seconds and a pipeline had no way to ask for anything else.--sessionTimeoutMsexposes it, keeping Kafka's 45s default so nothing changes for an existing pipeline. Lowering it trades tolerance of a slow or briefly paused instance for quicker recovery, and a broker will refuse a value below its owngroup.min.session.timeout.ms, which itself defaults to 6s — so this is not a knob that can be turned arbitrarily far.The heartbeat is derived as a third of the timeout rather than exposed separately. Kafka rejects a heartbeat that is not shorter than the session timeout, so deriving it keeps the pair consistent however the timeout is set; a third is the ratio Kafka's own defaults use.
Testing
KafkaStreamsPipelineRunnerConfigTestcovers the default, the override, the ratio, and that the heartbeat stays shorter than the timeout across a range of values. That last one is not decoration: the derivation first had a fixed 200ms floor, which produces a heartbeat equal to the timeout at 200ms and would be rejected by Kafka. The test fails if that floor is put back.