[core] Add primary-key vector index foundation

[JingsongLi]

Summary

Introduce the foundation for bucket-local vector indexes on primary-key tables. This PR defines the table-option and validation contract, ANN payload metadata, and ANN build/search primitives used by later write, compaction, manifest, and query integrations.

This is intentionally a foundation PR: configuring these options does not yet wire end-to-end index maintenance or query routing. Those pieces will be submitted as stacked PRs.

Architecture

The intended data flow is:

complete Level-1-or-higher compact data file(s)
-> projected sequential vector readers
-> bucket-local ANN payload
one or more source data files
ANN ordinal -> (data file, row position)
-> snapshot deletion-vector filtering
-> row lookup / Top-K merge

The payload uses the actual global-index algorithm as IndexFileMeta.indexType; there is no synthetic pk-vector-ann wrapper type. There is also no persisted RAW vector copy or vector sidecar format: later maintenance code reads the configured vector column directly from complete compact data files. Follow-up maintenance will eagerly build one multi-source ANN for every non-empty set of newly eligible compact sources; there is no row-count grouping threshold.

The index stores physical row positions instead of duplicating primary keys. Snapshot visibility is enforced with the deletion vectors of the referenced source files.

Changes

• Add field-scoped primary-key vector options and generated configuration documentation.
• Resolve the single configured field and its normalized algorithm Options directly through CoreOptions; no separate primary-key vector options resolver is retained.
• Validate fixed-bucket primary-key tables, supported merge engines, deletion-vector mode, VECTOR<FLOAT> fields, and supported distance metrics.
• Reject renaming an indexed vector column so field-scoped options remain unambiguous.
• Append nullable _SOURCE_META to GlobalIndexMeta, preserving existing five-field metadata rows as sourceMeta=null.
• Store plugin-produced metadata directly in _INDEX_META and Paimon-owned ordered source file names and row counts in versioned PkVectorSourceMeta under _SOURCE_META.
• Store the actual global-index algorithm in IndexFileMeta.indexType; no PkVectorAnnSegmentMeta wrapper or synthetic payload type is needed.
• Add a generic sequential PkVectorReader contract so ANN construction is independent of the physical data-file reader introduced by follow-up PRs.
• Build ANN payloads through VectorGlobalIndexer, support single-source and multi-source ordinals, validate algorithm/metric compatibility, and clean up failed builds.
• Persist inclusive ANN ordinal ranges as [0, totalSourceRows - 1] and reject source sets with no physical rows.
• Search ANN payloads by loading IndexFileMeta.indexType, apply per-source deletion-vector filtering, and map results back to (dataFileName, rowPosition).

Table-option API

bucket=8
merge-engine=deduplicate
deletion-vectors.enabled=true
deletion-vectors.merge-on-read=false
pk-vector.index.columns=embedding
fields.embedding.pk-vector.index.type=ivf-pq
fields.embedding.pk-vector.distance.metric=cosine
fields.embedding.pk-vector.index.options={"nlist":64,"pq.m":8}

pk-vector.index.columns is deliberately plural so each vector field can own one independent index in the future. This first version validates exactly one configured column.

ANN construction has no minimum-row option. Once complete compact sources become eligible, later maintenance code will attempt to index them immediately; algorithm-specific build failures are surfaced instead of silently deferring those sources.

No compatibility aliases or payload compatibility paths are retained because this feature has not been released.

Testing

• Field-scoped configuration resolution and invalid-option coverage
• Eligibility validation for deletion vectors, merge engine, bucket mode, vector type, and metric
• Indexed vector column rename rejection
• Generic reader handling for null and excluded physical rows
• Single-source and multi-source ANN build/search, ordinal mapping, and deletion-vector filtering
• _SOURCE_META round-trip and legacy five-field Global Index metadata compatibility
• Focused Core run: 50 tests passed

mvn -pl paimon-core -Pfast-build \
-Dmaven.compiler.forceJavacCompilerUse=true \
-DwildcardSuites=none \
-Dtest=IndexFileMetaSerializerTest,IndexManifestEntrySerializerTest,ManifestCommittableSerializerCompatibilityTest,PkVectorAnnSegmentFileTest,PkVectorSourceMetaTest,PrimaryKeyVectorIndexOptionsTest,PrimaryKeyVectorIndexValidationTest test

Notes

Follow-up stacked PRs will add projected data-file readers, synchronous bucket-level ANN maintenance from compact outputs, manifest restore, exact fallback before eligible compact sources exist, query planning, Top-K merge, and physical-row materialization.

[leaves12138]

I found one blocking schema-evolution issue. The five focused vector-index test classes pass locally (47 tests), but a SchemaManager-level column-rename reproduction fails with IllegalArgumentException: Field embedding can not be found in table schema. The PK-vector option keys and column registry need to participate in the real rename path before this can be merged.

[leaves12138]

The top-level rename path is fixed and the focused suite passes locally (48 tests), but one supported field-qualified JSON option case still changes the vector index definition ID after a column rename. I added an inline reproduction below.

[leaves12138]

The rename restriction looks good, but the latest ANN build API simplification drops a source-cardinality invariant and can create metadata that the searcher refuses to read. I left one inline comment.

[leaves12138]

I found one correctness issue in the rebuilt ANN segment metadata. The focused regression assertion reproduces it consistently.

[leaves12138]

Re-reviewed the latest head. The inclusive row-range issue is fixed, empty source rows are rejected, and the focused 50-test suite passes locally. LGTM.