Skip to content

branch-4.1: [feature](paimon) Support table writes via JNI and schema evolution - #65868

Merged
yiguolei merged 19 commits into
apache:branch-4.1from
suxiaogang223:codex/backport-paimon-jni-write-4.1
Jul 30, 2026
Merged

branch-4.1: [feature](paimon) Support table writes via JNI and schema evolution#65868
yiguolei merged 19 commits into
apache:branch-4.1from
suxiaogang223:codex/backport-paimon-jni-write-4.1

Conversation

@suxiaogang223

@suxiaogang223suxiaogang223 commented Jul 22, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Related to #65086.

Doris supports reading Paimon external tables, but branch-4.1 lacks a complete write path for inserting query results into Paimon tables.

This PR adds Paimon table writes through the Paimon Java SDK via JNI. It also adds schema evolution support and correctness coverage for merge engines, bucket modes, changelog producers, compaction, schema defaults, partition overwrite, and transaction handling.

What is changed?

Write path and DML

  • Support INSERT INTO ... SELECT ....
  • Support INSERT INTO ... VALUES ....
  • Support INSERT OVERWRITE for:
    • non-partitioned tables;
    • static partitions;
    • partial static partition specifications;
    • dynamic partitions;
    • empty input.
  • Support append-only and primary-key tables.
  • Support reordered and partial target column lists.
  • Apply Paimon schema defaults to omitted columns through the Paimon SDK.

Primary-key merge engines

Support writes to the following Paimon merge engines:

  • deduplicate;
  • partial-update;
  • first-row;
  • aggregation.

Full-row and partial-column writes share the same schema mapping path. Partial-column writes are accepted only when the Paimon merge engine defines safe semantics for omitted fields.

Bucket modes

Support the following Paimon bucket modes:

  • fixed hash bucket;
  • unaware bucket;
  • postpone bucket;
  • hash dynamic bucket;
  • key dynamic bucket.

Dynamic bucket modes currently use a correctness-first single-owner execution model. Paimon SDK assigners determine the bucket; Doris does not duplicate Paimon hashing or bucket assignment logic.

Changelog and compaction

  • Support changelog producers:
    • none;
    • input;
    • lookup;
    • full-compaction.
  • Cover automatic write-time compaction correctness.
  • Preserve Paimon audit-log and snapshot changelog semantics.

Transaction and commit

  • Add FE-coordinated Paimon transactions.
  • Aggregate commit messages from multiple BE writers.
  • Support idempotent commit retry and commit outcome reconciliation.
  • Deduplicate retried commit messages.
  • Abort uncommitted files on write failure.
  • Commit empty INSERT OVERWRITE statements correctly.

Schema and metadata

  • Support Paimon external table schema evolution:
    • add columns;
    • drop columns;
    • rename columns;
    • modify column types, nullability, comments, defaults, and positions;
    • reorder columns.
  • Refresh external-table metadata after schema changes.
  • Load the latest remote Paimon schema when building scan projections.
  • Preserve typed partition identity instead of treating physical partition paths as unique logical partition identifiers.
  • Honor Paimon partition.default-name and partition.legacy-name.
  • Fall back to Paimon scan planning when Doris cannot safely represent all partitions for pruning.

Catalog coverage

  • Filesystem catalog writes.
  • Object storage and HDFS-backed writes.
  • HMS and REST catalog write regression suites.

Architecture

Nereids Paimon Table Sink
-> PaimonTableSinkOperator
-> PaimonTableWriter
-> JniPaimonWriteBackend
-> PaimonJniWriter
-> Paimon Java SDK
-> CommitMessage
-> FE PaimonTransaction

Arrow IPC is used to transfer Doris blocks to the Java writer. Partition routing, bucket assignment, merge-engine behavior, schema validation, default conversion, file writing, and compaction are delegated to the Paimon SDK where possible.

Scope

The following features are intentionally not included in this PR:

  • row-level UPDATE / DELETE SQL;
  • bucket-aware parallel data distribution and exchange optimization;
  • the Rust FFI write backend.

Primary-key upsert through INSERT is supported. Row-level SQL DML requires a separate scan/rewrite and RowKind propagation path.

Dynamic bucket writes are functionally supported, but currently use a correctness-first single-owner execution model. Bucket-aware parallel routing will be implemented separately.

Test coverage

Regression coverage includes:

  • append-only and primary-key writes;
  • primitive and complex types;
  • merge engines;
  • fixed, unaware, postpone, and dynamic bucket modes;
  • static and dynamic partition overwrite;
  • transaction retry, abort, and empty overwrite;
  • changelog producers;
  • automatic compaction;
  • schema defaults and omitted columns;
  • schema changes before and after writes;
  • Paimon CREATE TABLE properties;
  • filesystem, HMS, and REST catalog suites;
  • failure and edge cases.

Doris query results are validated using regression output files and compared with Spark/Paimon results where applicable.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
suxiaogang223 marked this pull request as ready for review July 22, 2026 03:35
@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

run buildall

@suxiaogang223suxiaogang223 changed the title [feature](paimon) Add Paimon write support via JNI (branch-4.1)branch-4.1: [feature](paimon) Paimon write support via JNIJul 22, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 3.62% (16/442) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage72.88% (28598/39238)
Line Coverage56.38% (309604/549162)
Region Coverage53.61% (258791/482765)
Branch Coverage54.57% (113606/208184)

Comment threadbe/src/exec/sink/writer/paimon/paimon_table_writer.cpp
@Gabriel39

Copy link
Copy Markdown
Contributor

这几个TODO放在这个PR里一起完成:

  • Row-level DELETE/UPDATE
  • Dynamic bucket
  • Bucket-aware data distribution
  • Changelog Input、Lookup、FullCompaction
  • Write-time compaction
  • HMS catalog、REST catalog
  • Schema evolution compatibility
  • Paimon CREATE TABLE、ALTER TABLE

@Gabriel39

Copy link
Copy Markdown
Contributor

回归测试验证不够,不能只跑happy path

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

回归测试验证不够,不能只跑happy path

已补充更多case

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

这几个TODO放在这个PR里一起完成:

  • Row-level DELETE/UPDATE
  • Dynamic bucket
  • Bucket-aware data distribution
  • Changelog Input、Lookup、FullCompaction
  • Write-time compaction
  • HMS catalog、REST catalog
  • Schema evolution compatibility
  • Paimon CREATE TABLE、ALTER TABLE

Bucket-aware data distribution比较难搞,目前先不在这个pr实现了,现在是通过单线程写入保证正确性。我准备再下一个pr中和iceberg写入一起优化

@suxiaogang223
suxiaogang223force-pushed the codex/backport-paimon-jni-write-4.1 branch from b72b118 to 93eb91bCompareJuly 27, 2026 11:41
@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

run buildall

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 2nd, 2026 1:27 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30263780057

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 21.38% (170/795) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 1.36% (6/442) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage57.02% (23302/40864)
Line Coverage41.05% (230984/562731)
Region Coverage37.36% (183102/490066)
Branch Coverage38.41% (82180/213929)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 3.62% (16/442) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage72.97% (29117/39903)
Line Coverage56.77% (317836/559832)
Region Coverage53.94% (265951/493068)
Branch Coverage54.73% (117218/214166)

@shuke987

Copy link
Copy Markdown
Contributor

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Your access token could not be refreshed because your refresh token was revoked. Please log out and sign in again.
Workflow run: https://github.com/apache/doris/actions/runs/30324949656

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@shuke987

Copy link
Copy Markdown
Contributor

/review

@Gabriel39

Copy link
Copy Markdown
Contributor

对照 #65086 重新核对了当前 head a129238。Issue checklist 目前不能视为全部完成,至少还有以下项目:

  1. Row-level DELETE / UPDATE:未实现。 当前没有对应执行链路和回归用例。
  2. Bucket-aware data distribution:未实现。HASH_DYNAMIC / KEY_DYNAMIC 当前通过 single writer 保证正确性,并不是真正按 Paimon bucket 分发;该项已确认计划放到后续 PR。
  3. Rust FFI backend:未实现。ffi_paimon_write_backend.cpp 仍直接返回 NotSupported("Paimon Rust FFI writer is not implemented")
  4. Schema evolution compatibility:实现存在但当前验收失败。 最新 External Regression 中,主键表执行 RENAME COLUMN metric metric_value 后立即读取失败:RequiredField metric_value not found in schematest_paimon_write_schema_change line 916)。因此这项目前不能标记完成。
  5. Static partition overwrite:已有用例暴露正确性问题。 自定义 partition.default-name 后执行 PARTITION(region = NULL),会出现 Duplicate Paimon partition nametest_paimon_write_transaction),表元数据无法正常加载。Issue 中该项虽然已标 ✅,但当前实际状态应视为 blocker。

另外还有两个需要明确的交付边界:

  • Paimon schema/default value 在省略目标列时目前仍不生效,回归文件中已有 TODO 并把相关 case 注释掉;建议修复后恢复 case,或者在 PR scope 中明确该限制。
  • HMS/REST catalog write case 已加入,但没有进入本轮 External P0 的执行范围,目前只有代码/用例,尚缺最新 CI 通过证据。

Dynamic partition overwrite、partial-update/aggregation/first-row、dynamic bucket、Input/Lookup/FullCompaction changelog、write-time compaction、CREATE/ALTER DDL 等虽然 Issue 仍显示 ⬜,当前 PR 已有实现和用例,建议同步更新 PR description 与 Issue checklist。合入前请至少修复上述两个 External Regression 正确性失败,并明确 3 个未实现 checklist 项是否移出本 PR scope。

@Gabriel39

Copy link
Copy Markdown
Contributor

检查了当前新增的 13 个 Paimon write regression suites。覆盖相比之前明显增加:基础 INSERT/OVERWRITE、append/PK、merge engine、bucket modes、primitive/complex types、changelog、compaction、schema change、失败后原子性都有 case;最新 External Regression 中 11 个 suite 通过,2 个 suite 失败。

但对照 Issue 中“failure, retry, rollback correctness”的目标,当前覆盖仍不完整,建议补充以下场景:

P0:事务 exactly-once 与真实故障注入

现有 transaction regression 主要验证正常 commit、数据校验失败和 snapshot 数量;commit retry/reconcile 主要是 Mockito unit test。需要端到端覆盖:

  • 多 writer prepareCommit 完成后、FE commit 前失败/取消;
  • Paimon 已成功 commit,但响应丢失或 FE 超时,使用相同 transaction/commit message 重试;
  • FE leader restart/failover、BE fragment retry、某个 fragment 失败而其他 fragment 已产生文件;
  • INSERT OVERWRITE 在上述故障点下的 rollback;
  • 验证结果不能只看行数:还要检查 snapshot 数量、无重复数据、旧 snapshot 不被误删,以及未提交 data/manifest 文件得到 cleanup。

P1:并发与冲突

  • 两个 Doris session 并发 INSERT 同一表/同一分区及不同分区;
  • 并发 INSERT OVERWRITE 同一分区;
  • Doris 与 Spark/Flink 并发写同一 Paimon 表;
  • commit conflict/retry 后验证无丢行、无重复 snapshot、最终结果与 Paimon reader 一致。

P1:partition/overwrite 边界

  • 保留并修复当前失败的 custom default partition + NULL case;
  • 多级分区中 static partial overwrite,组合 NULL、空串、字面量 "null"、特殊字符和不同分区类型;
  • dynamic overwrite 同时覆盖多个分区,并验证未命中的分区、物理 partition metadata 和 snapshot;
  • failed overwrite 后检查 orphan files,而不只是行数和 snapshot count。

P1:HMS/REST catalog 完整语义

当前 HMS/REST case 只有 CREATE + 基础 append/PK happy path。至少补跑并覆盖:

  • INSERT OVERWRITE(全表、static、dynamic)、失败 rollback;
  • schema evolution 后立即读写;
  • dynamic bucket / merge engine;
  • catalog refresh/reopen 后继续写;
  • Doris 写入后由 Spark/Paimon reader 校验,而不只由 Doris 自读。

P1:JNI/BE 单测覆盖

当前 BE 增量行覆盖率只有 3.62% (16/442),C++ test 只覆盖 backend factory 选择;PaimonJniWriterTest 主要覆盖 classloader 恢复,PaimonArrowConverterTest 只有 timestamp 用例。建议补:

  • C++ writer/backend 的 open/write/prepareCommit/abort/close、错误传播、payload framing、多 block 和资源释放;
  • Java Arrow converter 的 ARRAY/MAP/STRUCT/BINARY、递归 NULL、DECIMAL 边界及非法类型;
  • JNI writer 的真实 open→write→prepareCommit→abort/close 生命周期、dynamic bucket/global index、compaction/changelog 分支。

P2:已知边界与规模

  • 恢复 omitted column + Paimon default value 的 case(当前被 TODO 注释);
  • key-dynamic 高基数、多分区、多次 transaction、writer/进程重建后的 global index 恢复;同时记录 Java heap 和文件数;
  • 大数据量、多 bucket、多 partition 下的吞吐和 small-file 数量;
  • compaction/changelog 在失败、重试、跨分区和重启后的正确性。

此外 FE coverage gate 当前还因模块重命名后仍引用旧 paimon-scanner.jar 路径而失败。建议修复 coverage 脚本并让 BE/FE coverage gate 正常通过后再判断单测覆盖是否达标。

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Your access token could not be refreshed because your refresh token was revoked. Please log out and sign in again.
Workflow run: https://github.com/apache/doris/actions/runs/30329307471

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@shuke987

Copy link
Copy Markdown
Contributor

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Your access token could not be refreshed because your refresh token was revoked. Please log out and sign in again.
Workflow run: https://github.com/apache/doris/actions/runs/30331515846

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@shuke987

Copy link
Copy Markdown
Contributor

/review

@suxiaogang223
suxiaogang223force-pushed the codex/backport-paimon-jni-write-4.1 branch from 68a3413 to 7b2a7c6CompareJuly 29, 2026 07:02
@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

/review

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

run buildall

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage77.58% (1900/2449)
Line Coverage64.41% (33974/52745)
Region Coverage64.94% (17522/26980)
Branch Coverage54.07% (9384/17356)

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step.
Workflow run: https://github.com/apache/doris/actions/runs/30430357527

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

/review

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for four correctness issues at reviewed head 96a7fb8bf8c8357f3d05254f3793c2534099573d:

  • FE sink binding still makes Paimon VARIANT targets unwritable before the corrected Java converter can run.
  • Case-colliding top-level Paimon fields can silently receive the same source value.
  • NTZ partition timestamps are serialized in a form Doris rejects, unnecessarily disabling partition metadata and partition-aligned MTMV behavior.
  • The literal-NULL MTMV suffix introduces a new partition-name collision.

Checkpoint conclusions:

  • Commit/report/JNI lifecycle and resource ownership were traced through final reporting, retry/deduplication, commit/abort state, and writer shutdown; no new non-duplicate issue remained beyond existing threads.
  • Write-target/schema/type fidelity produced the two binding blockers above; generation pinning, static overwrite identity, defaults, and supported Arrow/InternalRow conversions were otherwise coherent.
  • Snapshot/partition/MTMV handling produced the two metadata blockers above; LTZ fallback, ambiguous-display fallback, dictionary freshness, and snapshot scoping were otherwise coherent.
  • DDL, time-travel, overwrite, transaction, planner distribution, and the unit/regression coverage were reviewed across all changed files; other concerns were either fixed or already covered by existing threads.

User focus: no additional review focus was provided, so the full PR was reviewed without a narrower focus.

Validation was static only, as required by the review environment; no builds or tests were run.

Comment threadfe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java Outdated
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 73.08% (706/966) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

/review

@suxiaogang223

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage77.58% (1900/2449)
Line Coverage64.42% (33978/52745)
Region Coverage64.97% (17529/26980)
Branch Coverage54.09% (9388/17356)

github-actions[bot]
github-actionsBot previously requested changes Jul 29, 2026

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two distinct failure-path issues that should be fixed before merge: partial multi-bucket
preparation can strand uncommitted files, and an explicitly unknown FE commit outcome loses
its only recovery record. No additional user review focus was provided, so the full PR was
reviewed.

List<CommitMessage> messages = commitIdentifier > 0
? writer.prepareCommit(true, commitIdentifier)
: writer.prepareCommit();
preparedCommitMessages = new ArrayList<>(messages);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve partial committables when prepare fails

Paimon 1.3.1 prepares partition/bucket writers sequentially, and each successful writer drains
its pending file increment before the next writer is attempted. If bucket A succeeds and bucket
B then throws, this assignment is never reached, so preparedCommitMessages stays empty.
abortWriter() retries prepare, but A now returns an empty increment and closeWriter() no
longer owns A's drained data/index/changelog files, leaving orphan files after the INSERT fails.
Please retain each drained increment until the aggregate prepare succeeds (or add an atomic
prepare/abort facility), and cover a second-bucket prepare failure with a cleanup assertion.

transactionId, tableName());
return;
}
if (currentState == CommitState.COMMITTING

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep unknown commit outcomes recoverable

When both commit attempts and the identifier lookup fail, this state means the snapshot may
already have landed, so skipping abort is correct. However, statement failure immediately calls
the transaction manager's rollback(), whose finally removes this transaction from both
registries. That discards the only commit user/id, payloads, pinned table, and state needed to
reconcile later. If the commit landed, the client sees an error and a retry under a new
transaction can duplicate the append; if it did not, the prepared files remain stranded.
Please retain/persist OUTCOME_UNKNOWN transactions and reconcile them before cleanup instead of
letting statement rollback erase the recovery record.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 72.42% (709/979) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit 139cd2c into apache:branch-4.1Jul 30, 2026
34 of 41 checks passed
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 3, 2026
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#65868, apache#66612, apache#66810
Problem Summary: Add the BE Paimon sink, JNI writer backend, Java writer, commit payload transport, and worktree-local spill and memory lifecycle support required to write Paimon tables from master.
### Release note
Support writing Apache Paimon tables through the native Doris execution pipeline.
### Check List (For Author)
- Test: Static validation
- BE clang-format/check-format and FE checkstyle; full build and tests are deferred until all forward-port picks are complete
- Behavior changed: Yes, adds Paimon table writes
- Does this need documentation: Yes, documentation can follow separately
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 3, 2026
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#65868, apache#66498, apache#66685
Problem Summary: Forward-port the connector transaction, commit-message bridge, connector-neutral write distribution contract, and Paimon write-plan selection needed to execute table writes on master. The plan preserves native Paimon fixed-bucket routing and exposes row-change capabilities for subsequent DML planning.
### Release note
Support planning and committing Paimon table writes through the connector framework.
### Check List (For Author)
- Test: Manual test
- FE checkstyle passed; final compilation is deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon writes now use connector transactions and native write distribution planning.
- Does this need documentation: No
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Sep 3, 2026
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#65868, apache#66325, apache#67023, apache#67085
Problem Summary: Forward-port the final branch-4.1 Paimon write regression matrix to master, including append and primary-key modes, bucket routing, transactions, schema evolution, row-level DML, Variant V2, snapshot references, writer lifecycle, cache coherence, and stabilized external paths. Generated golden outputs are preserved byte-for-byte from branch-4.1.
### Release note
None
### Check List (For Author)
- Test: Regression test
- Test cases and generated expected outputs were forward-ported; execution is deferred until the final integrated validation.
- Behavior changed: No
- Does this need documentation: No
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@suxiaogang223@hello-stephen@Gabriel39@shuke987@yiguolei