Skip to content

[core][connectors][docs] Complete primary-key sorted-index support#8615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JingsongLi merged 1 commit into from
Jul 14, 2026
Merged

[core][connectors][docs] Complete primary-key sorted-index support #8615

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
380 changes: 380 additions & 0 deletions docs/docs/primary-key-table/global-index.mdx
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,380 @@
---
title: "Primary-Key Indexes"
sidebar_position: 9
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Primary-Key Indexes

Primary-key tables can maintain Vector, BTree, and Bitmap indexes together with compact data
files. These indexes are bucket-local and source-backed: every index group records its source data
files and maps matches back to physical row positions. Deletion vectors are applied when indexed
rows are read, so updates and deletes remain exact.

Primary-key indexes differ from indexes created with `create_global_index`. A regular
[Global Index](../multimodal-table/global-index) addresses table-wide row IDs and is built
independently for a Data Evolution table. A primary-key index follows the write and compaction
lifecycle of a primary-key table and addresses rows inside its compact data files.

## Choose an Index

<Tabs groupId="primary-key-index-family">

<TabItem value="btree" label="BTree">

Use BTree for selective scalar predicates such as equality, `IN`, comparisons, ranges, and null
checks. Normal Flink, Spark, and Java batch scans apply it automatically; no index-specific query
API is required.

See [BTree Index](../multimodal-table/global-index/btree) for the built-in format and supported
data types.

</TabItem>

<TabItem value="bitmap" label="Bitmap">

Use Bitmap for enum-like dimensions, tags, and other columns where compressed bitmaps can evaluate
equality, `IN`, null checks, complement predicates, and supported string predicates efficiently.
Normal batch scans apply it automatically.

See [Bitmap Index](../multimodal-table/global-index/bitmap) for predicate and format details.

</TabItem>

<TabItem value="vector" label="Vector">

Use Vector for approximate nearest-neighbor (ANN) Top-K search on embeddings that are updated with
the primary-key table. The index implementation and distance metric are configured per field, and
search is exposed through Spark SQL, a Flink procedure, and the Java API.

For an append-only or Data Evolution table whose vector index is built independently, see
[Global Vector Index](../multimodal-table/global-index/vector).

</TabItem>

</Tabs>

Different columns in one table can use different index families. One column can occur in at most
one of `pk-vector.index.columns`, `pk-btree.index.columns`, and `pk-bitmap.index.columns`.

## Requirements

All primary-key indexes require:

- A primary-key table in fixed-bucket mode (`bucket > 0`) or postpone-bucket mode
(`bucket = -2`).
- `pk-clustering-override = false`.
- An indexed column supported by the selected index implementation.

Dynamic-bucket tables, append-only tables, compound indexes, and custom index families are not
supported by this configuration.

<Tabs groupId="primary-key-index-requirements">

<TabItem value="scalar" label="BTree and Bitmap">

BTree and Bitmap additionally require:

- `deletion-vectors.enabled = true`.
- `deletion-vectors.merge-on-read = false`.
- A supported scalar column type.

Each entry creates an independent single-column index. Multiple BTree and Bitmap columns are
supported as long as a column is not listed by another primary-key index family.

</TabItem>

<TabItem value="vector" label="Vector">

Vector additionally requires:

- A `VECTOR` column whose element type is `FLOAT`.
- A merge engine of `deduplicate`, `partial-update`, `aggregation`, or `first-row`.
- `deletion-vectors.enabled = true`, except for `first-row`, where it must be `false`.
- The configured ANN implementation on every writer and reader classpath.

Exactly one vector column is currently supported per table.

</TabItem>

</Tabs>

## Create a Table

The following table uses all three families on different columns: Vector for `embedding`, BTree
for `amount`, and Bitmap for `status`.

<Tabs groupId="primary-key-index-create-table">

<TabItem value="flink-sql" label="Flink SQL">

```sql
CREATE TABLE items (
id BIGINT,
status STRING,
amount DECIMAL(12, 2),
embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;3',
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'bucket' = '8',
'deletion-vectors.enabled' = 'true',
'pk-vector.index.columns' = 'embedding',
'fields.embedding.pk-vector.index.type' = 'ivf-flat',
'fields.embedding.pk-vector.distance.metric' = 'cosine',
'fields.embedding.pk-vector.index.options' = '{"nlist":"256"}',
'pk-btree.index.columns' = 'amount',
'fields.amount.pk-btree.index.options' = '{"block-size":"64 kb"}',
'pk-bitmap.index.columns' = 'status',
'fields.status.pk-bitmap.index.options' = '{"dictionary-block-size":"16 kb"}'
);
```

The vector comment directive converts the SQL `ARRAY<FLOAT>` column to Paimon's fixed-length
`VECTOR<FLOAT>` type. Java API users can define it directly with
`DataTypes.VECTOR(3, DataTypes.FLOAT())`.

</TabItem>

<TabItem value="spark-sql" label="Spark SQL">

```sql
CREATE TABLE items (
id BIGINT,
status STRING,
amount DECIMAL(12, 2),
embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;3'
) USING paimon
TBLPROPERTIES (
'primary-key' = 'id',
'bucket' = '8',
'deletion-vectors.enabled' = 'true',
'pk-vector.index.columns' = 'embedding',
'fields.embedding.pk-vector.index.type' = 'ivf-flat',
'fields.embedding.pk-vector.distance.metric' = 'cosine',
'fields.embedding.pk-vector.index.options' = '{"nlist":"256"}',
'pk-btree.index.columns' = 'amount',
'fields.amount.pk-btree.index.options' = '{"block-size":"64 kb"}',
'pk-bitmap.index.columns' = 'status',
'fields.status.pk-bitmap.index.options' = '{"dictionary-block-size":"16 kb"}'
);
```

</TabItem>

</Tabs>

Duplicate, empty, unknown, unsupported, or cross-family duplicate columns are rejected during
schema validation.

### Options

| Option | Default | Description |
|---|---|---|
| `pk-vector.index.columns` | Not set | Vector column to index. Exactly one vector column is currently supported. |
| `fields.<column>.pk-vector.index.type` | Required | ANN implementation, such as `ivf-flat`, `ivf-pq`, `ivf-hnsw-flat`, `ivf-hnsw-sq`, or `lumina`. |
| `fields.<column>.pk-vector.distance.metric` | `inner_product` | Distance metric: `l2`, `cosine`, or `inner_product`. |
| `fields.<column>.pk-vector.index.options` | Not set | JSON object containing build options for the selected ANN implementation. |
| `pk-btree.index.columns` | Not set | Comma-separated columns which own independent BTree indexes. |
| `fields.<column>.pk-btree.index.options` | Not set | JSON object containing BTree build options. Unqualified keys are scoped to `btree-index`. |
| `pk-bitmap.index.columns` | Not set | Comma-separated columns which own independent Bitmap indexes. |
| `fields.<column>.pk-bitmap.index.options` | Not set | JSON object containing Bitmap build options. Unqualified keys are scoped to `bitmap-index`. |
| `fields.<column>.pk-index.compaction.level-fanout` | `5` | Number of similarly sized index groups which triggers a rebuild and maximum row-count ratio within one size tier. Shared by all three families. Must be greater than `1`. |
| `fields.<column>.pk-index.compaction.stale-ratio-threshold` | `0.2` | Ratio of rows from inactive source files which triggers a rebuild. Shared by all three families. Must be in `(0, 1]`. |

For algorithm-specific options, see the corresponding
[BTree](../multimodal-table/global-index/btree),
[Bitmap](../multimodal-table/global-index/bitmap), or
[Vector](../multimodal-table/global-index/vector) index page.

## Maintenance and Coverage

Paimon builds primary-key indexes from complete `COMPACT` data files above Level 0. An index group
stores the source file names and row counts required to map index results back to physical rows.
Data-file and index-file changes are committed in the same snapshot.

New Level-0 `APPEND` files are not index sources. They remain uncovered until physical data
compaction produces an eligible output. A metadata-only level upgrade keeps its `APPEND` source
and therefore remains uncovered as well.

For a postpone-bucket table, foreground writes can land in bucket `-2` or in pending Level-0 files
inside real buckets. These rows become visible after batch compaction publishes them to real
buckets. Indexes are created when that process physically rewrites the rows into eligible compact
output; simply assigning or upgrading a pending file does not make it an index source.

### Index LSM Maintenance

Each indexed column maintains its own immutable index groups. Maintenance uses the shared
field-scoped compaction options:

- When at least `level-fanout` similarly sized groups exist, Paimon rebuilds them into a larger
group. The largest selected group can contain at most `level-fanout` times the rows of the
smallest group.
- When the ratio of rows belonging to inactive source files reaches
`stale-ratio-threshold`, Paimon rebuilds the affected group from its remaining active sources.
- A rebuild atomically replaces its input groups after the new group is complete.

Index construction can execute asynchronously inside the writer. A writer which waits for
compaction also waits for active index maintenance; a non-blocking writer can complete maintenance
in a later commit. Coverage can therefore be temporarily partial.

Partial coverage affects acceleration, not correctness:

- BTree and Bitmap scans read uncovered files through the ordinary data path.
- Vector search evaluates files without an active ANN group exactly.
- The original scalar predicate and deletion vectors are applied after index pruning.

## BTree and Bitmap Queries

BTree and Bitmap indexes are applied automatically to snapshot-scoped batch scans. They can
accelerate equality, `IN`, null checks, comparisons and ranges, complement predicates, and
supported string predicates.

Paimon combines indexed scalar results as follows:

- For `AND`, any usable indexed child can narrow a source file; unindexed children remain residual
filters.
- For `OR`, index positions are used only when every branch can be evaluated safely. Otherwise,
the source file is scanned normally.

Planning uses the data and index manifests from the selected snapshot. If an index group is
missing, incomplete, unreadable, corrupt, or returns an invalid physical position, the affected
source file falls back to the ordinary data path.

## Vector Search

A vector search captures one table snapshot, searches active ANN groups in the selected buckets,
and merges their candidates into a global Top-K. Candidates are materialized from source data
files by physical row position. Deletion vectors remove stale versions and deleted rows.

<Tabs groupId="primary-key-vector-search-api">

<TabItem value="spark-sql" label="Spark SQL">

Use the `vector_search` table-valued function. Spark exposes the ANN score through the
`__paimon_search_score` metadata column.

```sql
SELECT id, status, __paimon_search_score
FROM vector_search(
'items',
'embedding',
array(0.1f, 0.2f, 0.3f),
10,
map('ivf.nprobe', '32')
);
```

The query dimension must match the indexed vector dimension. Partition predicates are applied
before ANN search. When `spark.paimon.vector-search.distribute.enabled` is `true`, Spark can
distribute sufficiently large groups of bucket-local searches and merge task-local Top-K results
on the driver.

</TabItem>

<TabItem value="flink-sql" label="Flink SQL">

Flink exposes vector search as a procedure and returns JSON-serialized rows. Use `projection` to
avoid reading columns which are not needed.

```sql
CALL sys.vector_search(
`table` => 'default.items',
vector_column => 'embedding',
query_vector => '0.1,0.2,0.3',
top_k => 10,
projection => 'id,status',
options => 'ivf.nprobe=32'
);
```

</TabItem>

<TabItem value="java-api" label="Java API">

```java
GlobalIndexResult result = table.newVectorSearchBuilder()
.withVectorColumn("embedding")
.withVector(queryVector)
.withLimit(10)
.withOption("ivf.nprobe", "32")
.executeLocal();

ReadBuilder readBuilder = table.newReadBuilder();
TableScan.Plan plan = readBuilder.newScan().withGlobalIndexResult(result).plan();
try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(plan)) {
reader.forEachRemaining(row -> consume(row));
}
```

</TabItem>

</Tabs>

### Exact Rerank

Primary-key vector search can retrieve additional ANN candidates and rerank them with the original
vectors. For example, this table option retrieves up to four times the requested Top-K before
computing exact distances:

```sql
'fields.embedding.ivf-flat.refine_factor' = '4'
```

`refine_factor`, `refine-factor`, `rerank_factor`, and `rerank-factor` are accepted. Query options
override table options, and field/index prefixes take precedence over less-specific prefixes. The
factor must be a positive integer. A factor of `1` performs exact reranking without retrieving
additional candidates.

Only ANN candidates can win the rerank, so a larger factor can improve recall but does not
guarantee the exact global Top-K. It also increases ANN work and data-file I/O. Uncovered files are
searched exactly and merged separately with the ANN candidates.

## Merge-Engine Behavior

Vector maintenance follows the table's merge engine:

- `deduplicate`: an update indexes the latest row and the deletion vector hides the replaced
physical row. A delete removes the old row from search results through the deletion vector.
- `partial-update`: the vector index is built from the lookup-completed compact-output row.
- `aggregation`: the vector index is built from the aggregated compact-output row.
- `first-row`: the retained first row is indexed. Deletion vectors must be disabled because later
rows with the same primary key are ignored rather than deleting the retained row.

## Schema Evolution

An indexed column cannot be renamed, dropped, or have its type changed while its definition is
present. Treat index column lists, field-scoped implementation settings, and build options as part
of the index definition. To change the family or incompatible build options of an indexed column,
create a table with the desired definition and migrate the data.

## Limitations

- BTree and Bitmap definitions are single-column indexes.
- Exactly one vector index column is currently supported per table.
- Only `FLOAT` vectors are supported.
- Indexes are built from eligible compact output, not directly from Level-0 appends.
- Index acceleration and vector search are snapshot-scoped batch operations; continuous streaming
and lateral vector search are not supported.
- Flink vector search returns rows but does not expose the ANN score as a separate column.
- Online replacement between two definitions on the same column is not supported.
Loading
Loading