Skip to content

[core] Add public binary serialization for BucketVectorSearchSplit - #9386

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JunRuiLee:pkvector-native-bucket-split
Aug 26, 2026
Merged

[core] Add public binary serialization for BucketVectorSearchSplit#9386
JingsongLi merged 1 commit into
apache:masterfrom
JunRuiLee:pkvector-native-bucket-split

Conversation

@JunRuiLee

@JunRuiLeeJunRuiLee commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Purpose

BucketVectorSearchSplit is only Java-serializable today, so a reader outside the JVM cannot read it. Pushing PK-table vector search into a native engine needs a byte form: a Java planner enumerates bucket splits through PrimaryKeyVectorScan -- the ANN current-segment decision needs the bucket's complete active-file set, which an ordinary table scan split does not carry -- and dispatches each to a native worker. The reader is apache/paimon-rust#746; there is no Java production call site yet.

This follows IndexedSplit: magic and version, the nested DataSplit, then the split's own state.

i64 magic = "PKVSPLIT"; i32 version = 1
DataSplit // DataSplit#serialize
i32 payloadCount; IndexFileMeta* // IndexFileMetaSerializer#serializeList
i32 rangeFileCount; rangeFile*
rangeFile = utf fileName, i32 rangeCount, (i64 from, i64 to)*
  • Payloads reuse IndexFileMetaSerializer, so IndexFileMeta.SCHEMA stays the only source of truth for what a payload carries. It has no version of its own, so a change there has to bump this one.
  • Row ranges are written sorted by file name: the map's iteration order is its construction order while its equality is not, so two splits that compare equal would otherwise serialize differently.
  • writeObject/readObject delegate to this codec instead of calling defaultWriteObject, leaving one serialized form instead of two. That changes the object-stream bytes, which is free while the class is unreleased (added in [core] Search primary-key vector indexes #8579, no tag contains it): both call sites, FlinkPrimaryKeyVectorRead and SparkPrimaryKeyVectorRead, produce and consume those bytes inside one job.

Read-side only; no existing write or planning path changes.

@JunRuiLee
JunRuiLeeforce-pushed the pkvector-native-bucket-split branch 5 times, most recently from 0aa04ec to 9c7a531CompareAugust 25, 2026 14:36
@JunRuiLee
JunRuiLee marked this pull request as ready for review August 26, 2026 02:54
@JunRuiLee
JunRuiLee marked this pull request as draft August 26, 2026 03:42
@JunRuiLee
JunRuiLeeforce-pushed the pkvector-native-bucket-split branch from 9c7a531 to c55eba5CompareAugust 26, 2026 03:58
@JunRuiLee
JunRuiLee marked this pull request as ready for review August 26, 2026 04:19

@JingsongLiJingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do so many things? Maybe just serialize rowRangesByFile and remove defaultWriteObject is OK?

A primary-key vector search is planned per bucket: deciding which ANN
segments are current needs the bucket's complete active-file set, which
an ordinary table-scan split does not carry. The split that carries it
can so far only be moved by Java object serialization, which keeps the
search inside a JVM. Give it a byte form a reader in another language can
consume, so a Java planner can dispatch buckets to native workers.
The form follows IndexedSplit, the sibling split for global index: magic
and version, the nested DataSplit, then the split's own state. Payloads
go through IndexFileMetaSerializer rather than a hand-written field
encoding, so IndexFileMeta.SCHEMA stays the one source of truth for what
a payload carries. The envelope version pins the layout of what it nests
rather than the nested bytes, so a change to that schema bumps it too.
Row-range entries are written sorted, so two splits that compare equal
serialize to the same bytes.
writeObject/readObject delegate to this codec instead of calling
defaultWriteObject, leaving the class with one serialized form rather
than two. That changes the object-stream bytes, which is free while the
class is unreleased: both engine call sites produce and consume them
inside a single job.
@JunRuiLee
JunRuiLeeforce-pushed the pkvector-native-bucket-split branch from c55eba5 to e9aca7aCompareAugust 26, 2026 08:28
@JunRuiLee

Copy link
Copy Markdown
ContributorAuthor

Do we need to do so many things? Maybe just serialize rowRangesByFile and remove defaultWriteObject is OK?

Thanks for suggestion. I reworked to follow IndexedSplit (magic + version + nested DataSplit), payloads reuse IndexFileMetaSerializer, and writeObject delegates to that one codec so defaultWriteObject is gone.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 088d488 into apache:masterAug 26, 2026
12 of 13 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@JunRuiLee@JingsongLi