Uh oh!
There was an error while loading. Please reload this page.
[core] Add public binary serialization for BucketVectorSearchSplit - #9386
Merged
Merged
Conversation
JunRuiLeeforce-pushed
the
pkvector-native-bucket-split
branch
5 times, most recently
from
August 25, 2026 14:36
0aa04ec to
9c7a531CompareJunRuiLee
marked this pull request as ready for review
August 26, 2026 02:54
JunRuiLee
marked this pull request as draft
August 26, 2026 03:42
JunRuiLeeforce-pushed
the
pkvector-native-bucket-split
branch
from
August 26, 2026 03:58
9c7a531 to
c55eba5CompareJunRuiLee
marked this pull request as ready for review
August 26, 2026 04:19
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Do we need to do so many things? Maybe just serialize rowRangesByFile and remove defaultWriteObject is OK?
A primary-key vector search is planned per bucket: deciding which ANN segments are current needs the bucket's complete active-file set, which an ordinary table-scan split does not carry. The split that carries it can so far only be moved by Java object serialization, which keeps the search inside a JVM. Give it a byte form a reader in another language can consume, so a Java planner can dispatch buckets to native workers. The form follows IndexedSplit, the sibling split for global index: magic and version, the nested DataSplit, then the split's own state. Payloads go through IndexFileMetaSerializer rather than a hand-written field encoding, so IndexFileMeta.SCHEMA stays the one source of truth for what a payload carries. The envelope version pins the layout of what it nests rather than the nested bytes, so a change to that schema bumps it too. Row-range entries are written sorted, so two splits that compare equal serialize to the same bytes. writeObject/readObject delegate to this codec instead of calling defaultWriteObject, leaving the class with one serialized form rather than two. That changes the object-stream bytes, which is free while the class is unreleased: both engine call sites produce and consume them inside a single job.
JunRuiLeeforce-pushed
the
pkvector-native-bucket-split
branch
from
August 26, 2026 08:28
c55eba5 to
e9aca7aCompareJunRuiLee
commented
Aug 26, 2026
ContributorAuthor
Thanks for suggestion. I reworked to follow IndexedSplit (magic + version + nested DataSplit), payloads reuse IndexFileMetaSerializer, and writeObject delegates to that one codec so defaultWriteObject is gone. |
JingsongLi
commented
Aug 26, 2026
Contributor
+1 |
Uh oh!
There was an error while loading. Please reload this page.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
BucketVectorSearchSplitis only Java-serializable today, so a reader outside the JVM cannot read it. Pushing PK-table vector search into a native engine needs a byte form: a Java planner enumerates bucket splits throughPrimaryKeyVectorScan-- the ANN current-segment decision needs the bucket's complete active-file set, which an ordinary table scan split does not carry -- and dispatches each to a native worker. The reader is apache/paimon-rust#746; there is no Java production call site yet.This follows
IndexedSplit: magic and version, the nestedDataSplit, then the split's own state.IndexFileMetaSerializer, soIndexFileMeta.SCHEMAstays the only source of truth for what a payload carries. It has no version of its own, so a change there has to bump this one.writeObject/readObjectdelegate to this codec instead of callingdefaultWriteObject, leaving one serialized form instead of two. That changes the object-stream bytes, which is free while the class is unreleased (added in [core] Search primary-key vector indexes #8579, no tag contains it): both call sites,FlinkPrimaryKeyVectorReadandSparkPrimaryKeyVectorRead, produce and consume those bytes inside one job.Read-side only; no existing write or planning path changes.