Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [feature](paimon) Support table writes via JNI and schema evolution - #65868
Conversation
hello-stephen
commented
Jul 22, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
suxiaogang223
commented
Jul 22, 2026
run buildall |
hello-stephen
commented
Jul 22, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Gabriel39
commented
Jul 25, 2026
这几个TODO放在这个PR里一起完成:
|
Gabriel39
commented
Jul 25, 2026
回归测试验证不够,不能只跑happy path |
suxiaogang223
commented
Jul 27, 2026
已补充更多case |
suxiaogang223
commented
Jul 27, 2026
Bucket-aware data distribution比较难搞,目前先不在这个pr实现了,现在是通过单线程写入保证正确性。我准备再下一个pr中和iceberg写入一起优化 |
b72b118 to
93eb91bComparesuxiaogang223
commented
Jul 27, 2026
run buildall |
suxiaogang223
commented
Jul 27, 2026
/review |
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 2nd, 2026 1:27 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
hello-stephen
commented
Jul 27, 2026
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Jul 27, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jul 27, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
shuke987
commented
Jul 28, 2026
/review |
Codex automated review failed and did not complete. Error: Your access token could not be refreshed because your refresh token was revoked. Please log out and sign in again. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
shuke987
commented
Jul 28, 2026
/review |
Gabriel39
commented
Jul 28, 2026
对照 #65086 重新核对了当前 head
另外还有两个需要明确的交付边界:
Dynamic partition overwrite、partial-update/aggregation/first-row、dynamic bucket、Input/Lookup/FullCompaction changelog、write-time compaction、CREATE/ALTER DDL 等虽然 Issue 仍显示 ⬜,当前 PR 已有实现和用例,建议同步更新 PR description 与 Issue checklist。合入前请至少修复上述两个 External Regression 正确性失败,并明确 3 个未实现 checklist 项是否移出本 PR scope。 |
Gabriel39
commented
Jul 28, 2026
检查了当前新增的 13 个 Paimon write regression suites。覆盖相比之前明显增加:基础 INSERT/OVERWRITE、append/PK、merge engine、bucket modes、primitive/complex types、changelog、compaction、schema change、失败后原子性都有 case;最新 External Regression 中 11 个 suite 通过,2 个 suite 失败。 但对照 Issue 中“failure, retry, rollback correctness”的目标,当前覆盖仍不完整,建议补充以下场景: P0:事务 exactly-once 与真实故障注入现有 transaction regression 主要验证正常 commit、数据校验失败和 snapshot 数量;commit retry/reconcile 主要是 Mockito unit test。需要端到端覆盖:
P1:并发与冲突
P1:partition/overwrite 边界
P1:HMS/REST catalog 完整语义当前 HMS/REST case 只有 CREATE + 基础 append/PK happy path。至少补跑并覆盖:
P1:JNI/BE 单测覆盖当前 BE 增量行覆盖率只有
P2:已知边界与规模
此外 FE coverage gate 当前还因模块重命名后仍引用旧 |
Codex automated review failed and did not complete. Error: Your access token could not be refreshed because your refresh token was revoked. Please log out and sign in again. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
shuke987
commented
Jul 28, 2026
/review |
Codex automated review failed and did not complete. Error: Your access token could not be refreshed because your refresh token was revoked. Please log out and sign in again. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
shuke987
commented
Jul 28, 2026
/review |
68a3413 to
7b2a7c6Comparesuxiaogang223
commented
Jul 29, 2026
/review |
suxiaogang223
commented
Jul 29, 2026
run buildall |
suxiaogang223
commented
Jul 29, 2026
run buildall |
hello-stephen
commented
Jul 29, 2026
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Codex automated review failed and did not complete. Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
suxiaogang223
commented
Jul 29, 2026
/review |
There was a problem hiding this comment.
Requesting changes for four correctness issues at reviewed head 96a7fb8bf8c8357f3d05254f3793c2534099573d:
- FE sink binding still makes Paimon VARIANT targets unwritable before the corrected Java converter can run.
- Case-colliding top-level Paimon fields can silently receive the same source value.
- NTZ partition timestamps are serialized in a form Doris rejects, unnecessarily disabling partition metadata and partition-aligned MTMV behavior.
- The literal-NULL MTMV suffix introduces a new partition-name collision.
Checkpoint conclusions:
- Commit/report/JNI lifecycle and resource ownership were traced through final reporting, retry/deduplication, commit/abort state, and writer shutdown; no new non-duplicate issue remained beyond existing threads.
- Write-target/schema/type fidelity produced the two binding blockers above; generation pinning, static overwrite identity, defaults, and supported Arrow/InternalRow conversions were otherwise coherent.
- Snapshot/partition/MTMV handling produced the two metadata blockers above; LTZ fallback, ambiguous-display fallback, dictionary freshness, and snapshot scoping were otherwise coherent.
- DDL, time-travel, overwrite, transaction, planner distribution, and the unit/regression coverage were reviewed across all changed files; other concerns were either fixed or already covered by existing threads.
User focus: no additional review focus was provided, so the full PR was reviewed without a narrower focus.
Validation was static only, as required by the review environment; no builds or tests were run.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Jul 29, 2026
FE Regression Coverage ReportIncrement line coverage |
suxiaogang223
commented
Jul 29, 2026
/review |
suxiaogang223
commented
Jul 29, 2026
run buildall |
hello-stephen
commented
Jul 29, 2026
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
There was a problem hiding this comment.
Found two distinct failure-path issues that should be fixed before merge: partial multi-bucket
preparation can strand uncommitted files, and an explicitly unknown FE commit outcome loses
its only recovery record. No additional user review focus was provided, so the full PR was
reviewed.
| List<CommitMessage> messages = commitIdentifier > 0 | ||
| ? writer.prepareCommit(true, commitIdentifier) | ||
| : writer.prepareCommit(); | ||
| preparedCommitMessages = new ArrayList<>(messages); |
There was a problem hiding this comment.
[P2] Preserve partial committables when prepare fails
Paimon 1.3.1 prepares partition/bucket writers sequentially, and each successful writer drains
its pending file increment before the next writer is attempted. If bucket A succeeds and bucket
B then throws, this assignment is never reached, so preparedCommitMessages stays empty.abortWriter() retries prepare, but A now returns an empty increment and closeWriter() no
longer owns A's drained data/index/changelog files, leaving orphan files after the INSERT fails.
Please retain each drained increment until the aggregate prepare succeeds (or add an atomic
prepare/abort facility), and cover a second-bucket prepare failure with a cleanup assertion.
| transactionId, tableName()); | ||
| return; | ||
| } | ||
| if (currentState == CommitState.COMMITTING |
There was a problem hiding this comment.
[P1] Keep unknown commit outcomes recoverable
When both commit attempts and the identifier lookup fail, this state means the snapshot may
already have landed, so skipping abort is correct. However, statement failure immediately calls
the transaction manager's rollback(), whose finally removes this transaction from both
registries. That discards the only commit user/id, payloads, pinned table, and state needed to
reconcile later. If the commit landed, the client sees an error and a retry under a new
transaction can duplicate the append; if it did not, the prepared files remain stranded.
Please retain/persist OUTCOME_UNKNOWN transactions and reconcile them before cleanup instead of
letting statement rollback erase the recovery record.
hello-stephen
commented
Jul 29, 2026
FE Regression Coverage ReportIncrement line coverage |
Uh oh!
There was an error while loading. Please reload this page.
### What problem does this PR solve? Issue Number: closeapache#65086 Related PR: apache#65868, apache#66612, apache#66810 Problem Summary: Add the BE Paimon sink, JNI writer backend, Java writer, commit payload transport, and worktree-local spill and memory lifecycle support required to write Paimon tables from master. ### Release note Support writing Apache Paimon tables through the native Doris execution pipeline. ### Check List (For Author) - Test: Static validation - BE clang-format/check-format and FE checkstyle; full build and tests are deferred until all forward-port picks are complete - Behavior changed: Yes, adds Paimon table writes - Does this need documentation: Yes, documentation can follow separately
### What problem does this PR solve? Issue Number: closeapache#65086 Related PR: apache#65868, apache#66498, apache#66685 Problem Summary: Forward-port the connector transaction, commit-message bridge, connector-neutral write distribution contract, and Paimon write-plan selection needed to execute table writes on master. The plan preserves native Paimon fixed-bucket routing and exposes row-change capabilities for subsequent DML planning. ### Release note Support planning and committing Paimon table writes through the connector framework. ### Check List (For Author) - Test: Manual test - FE checkstyle passed; final compilation is deferred until all forward-port picks are complete. - Behavior changed: Yes, Paimon writes now use connector transactions and native write distribution planning. - Does this need documentation: No
### What problem does this PR solve? Issue Number: closeapache#65086 Related PR: apache#65868, apache#66325, apache#67023, apache#67085 Problem Summary: Forward-port the final branch-4.1 Paimon write regression matrix to master, including append and primary-key modes, bucket routing, transactions, schema evolution, row-level DML, Variant V2, snapshot references, writer lifecycle, cache coherence, and stabilized external paths. Generated golden outputs are preserved byte-for-byte from branch-4.1. ### Release note None ### Check List (For Author) - Test: Regression test - Test cases and generated expected outputs were forward-ported; execution is deferred until the final integrated validation. - Behavior changed: No - Does this need documentation: No
What problem does this PR solve?
Related to #65086.
Doris supports reading Paimon external tables, but branch-4.1 lacks a complete write path for inserting query results into Paimon tables.
This PR adds Paimon table writes through the Paimon Java SDK via JNI. It also adds schema evolution support and correctness coverage for merge engines, bucket modes, changelog producers, compaction, schema defaults, partition overwrite, and transaction handling.
What is changed?
Write path and DML
INSERT INTO ... SELECT ....INSERT INTO ... VALUES ....INSERT OVERWRITEfor:Primary-key merge engines
Support writes to the following Paimon merge engines:
deduplicate;partial-update;first-row;aggregation.Full-row and partial-column writes share the same schema mapping path. Partial-column writes are accepted only when the Paimon merge engine defines safe semantics for omitted fields.
Bucket modes
Support the following Paimon bucket modes:
Dynamic bucket modes currently use a correctness-first single-owner execution model. Paimon SDK assigners determine the bucket; Doris does not duplicate Paimon hashing or bucket assignment logic.
Changelog and compaction
none;input;lookup;full-compaction.Transaction and commit
INSERT OVERWRITEstatements correctly.Schema and metadata
partition.default-nameandpartition.legacy-name.Catalog coverage
Architecture
Arrow IPC is used to transfer Doris blocks to the Java writer. Partition routing, bucket assignment, merge-engine behavior, schema validation, default conversion, file writing, and compaction are delegated to the Paimon SDK where possible.
Scope
The following features are intentionally not included in this PR:
UPDATE/DELETESQL;Primary-key upsert through
INSERTis supported. Row-level SQL DML requires a separate scan/rewrite and RowKind propagation path.Dynamic bucket writes are functionally supported, but currently use a correctness-first single-owner execution model. Bucket-aware parallel routing will be implemented separately.
Test coverage
Regression coverage includes:
Doris query results are validated using regression output files and compared with Spark/Paimon results where applicable.