Uh oh!
There was an error while loading. Please reload this page.
Request: incremental sink reads (forward + blob watermark) [escalated from #122] - #152
Merged
Merged
Conversation
Escalated from issue #122 by the neutral reconciler. A bug-fix attempt found the proposed cursor/watermark to be a cross-cutting design change (extends the kernel storage read API + a persisted per-sink watermark, spanning the central forward sink and the core blob sink) whose exactly-once guarantee depends on a watermark that survives retention prunes and compaction generation swaps. Files it as a live request LLP so it re-enters the pipeline family for design. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Jul 28, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Files a request LLP (
llp/0039-incremental-sink-reads.spec.md, Type: Spec, Status: Active) for incremental reads in the central forward sink and the core blob sink — read only rows added since the last successful export, instead of re-reading and re-streaming the whole partition every tick.Why this is a request, not a fix
The neutral reconciler picked up issue #122 as a
neutral:fixbug. The bug-fix worker investigated and determined a test-provable localized fix is not credible: the proposed watermark is a cross-cutting design change thatstorage.readRowsAPI (consumed by sinks, backfill, query, vector-search, the projector) with asince/continuation, and(sink instance, partition)watermark contract,and whose exactly-once guarantee depends on choosing a watermark that survives retention prunes (rows deleted from the front of a partition) and compaction generation swaps — a real design decision with several viable shapes (snapshot ancestry, monotonic row-sequence, content-addressed continuation). It also revises documented design in LLP 0013/0014.
So per LLP 0009, the work is escalated out of the maintenance family and re-enters the pipeline family as a live request. Once this is merged, the pipeline's Designer will mint a design covering it.
Human decision point
Merging this PR greenlights the requirement (not the implementation — that comes as a later, separately-reviewed design + change-set PR). If you don't want this architecture, close the PR; issue #122 has had
neutral:fixremoved and points here.Closes-as-request: #122
🤖 Generated with Claude Code