branch-4.1: [feature](paimon) Support row-level UPDATE, DELETE, and MERGE by suxiaogang223 · Pull Request #66498 · apache/doris

suxiaogang223 · 2026-08-05T11:27:35Z

What changed

This PR adds row-level DML support for Apache Paimon tables through the Doris Paimon catalog:

UPDATE ... SET ... WHERE ...
- Supports expressions in assignments and predicates.
- Supports primary-key tables using the deduplicate and partial-update merge engines.
- Rejects updates to primary-key columns.
DELETE FROM ... WHERE ...
- Supports primary-key tables using deduplicate.
- Supports partial-update and aggregate tables when their Paimon delete/removal options are enabled.
MERGE INTO ... USING ... ON ...
- Supports conditional WHEN MATCHED THEN UPDATE.
- Supports conditional WHEN MATCHED THEN DELETE.
- Supports conditional WHEN NOT MATCHED THEN INSERT.
- Supports combining update, delete, and insert branches in one statement.

Append-only tables are rejected for row-level DML. Unsupported merge engines and table options fail during analysis with explicit error messages.

Implementation

Adds dedicated Nereids commands for Paimon UPDATE, DELETE, and MERGE.
Carries a per-row operation value through the logical and physical Paimon sink.
Converts the operation value into Paimon row kinds in the JNI writer.
Binds changelog sink outputs at the logical sink level, preserving the operation column while coercing data columns to the target schema.
Adds target-table collection and schema-change retry integration for bound Paimon row-level sinks.
Adds regression coverage for successful UPDATE, DELETE, and mixed MERGE operations, plus unsupported append-only and first-row cases.

User impact

Users can modify existing Paimon primary-key tables directly with standard Doris SQL instead of rewriting the table through INSERT or an external compute engine.

Validation

BUILD_TYPE=Debug DISABLE_BUILD_UI=ON ./build.sh --fe -j12
Debug BE build using the Doris toolchain
test_paimon_write_row_level_dml: UPDATE, DELETE, MERGE, and negative cases all passed
FE and BE deployment health checks passed

Related to #65086

hello-stephen · 2026-08-05T11:27:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2026-08-05T11:33:49Z

run buildall

hello-stephen · 2026-08-05T12:26:00Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.62% (1904/2453)
Line Coverage	64.46% (34018/52774)
Region Coverage	64.46% (17208/26694)
Branch Coverage	53.92% (9201/17064)

hello-stephen · 2026-08-05T15:30:02Z

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.27% (24674/42347)
Line Coverage	42.51% (248497/584589)
Region Coverage	38.57% (196304/508963)
Branch Coverage	39.94% (89813/224850)

hello-stephen · 2026-08-05T17:09:01Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.75% (30440/41275)
Line Coverage	57.70% (334767/580214)
Region Coverage	54.60% (278665/510329)
Branch Coverage	55.52% (124692/224608)

hello-stephen · 2026-08-05T17:25:46Z

FE Regression Coverage Report

Increment line coverage 2.38% (9/378) 🎉
Increment coverage report
Complete coverage report

Gabriel39 · 2026-08-06T02:04:41Z

Thanks for adding Paimon row-level DML support. I reviewed the current head ac1681ac and found the following correctness issues and test gaps that should be addressed before merge.

Correctness issues

[P1] DELETE can silently succeed without deleting rows when `ignore-delete=true`

PaimonDmlCommandUtils.checkDelete() accepts every DEDUPLICATE table unconditionally, without checking CoreOptions.IGNORE_DELETE.

In the Paimon 1.3.1 dependency used by this branch, DeduplicateMergeFunction deliberately discards retract records when ignore-delete=true. This PR emits RowKind.DELETE, so both a standalone DELETE and a MERGE DELETE branch can commit successfully while leaving the existing row unchanged.

Please reject DELETE/MERGE DELETE during analysis when delete records are configured to be ignored, or otherwise provide semantics that guarantee the row is removed.

[P1] UPDATE cannot set a nullable column to NULL on `merge-engine=partial-update`

checkUpdate() explicitly permits PARTIAL_UPDATE, and the new write path sends a complete UPDATE_AFTER row containing the explicit NULL value.

However, Paimon's partial-update merge function only updates non-null fields. As a result:

UPDATE t SET nullable_col =NULLWHERE id =1;

can report success while retaining the old value. The same issue applies to a MERGE UPDATE assignment.

Please either reject SQL UPDATE/MERGE UPDATE for partial-update tables until explicit NULL can be represented correctly, or add a connector-specific encoding/write path that distinguishes "field not supplied" from "set field to NULL".

[P2] Schema-change retry reuses the old immutable write target

The UPDATE/DELETE/MERGE commands create and pin a PaimonWriteTarget before constructing InsertIntoTableCommand. InsertIntoTableCommand retries by replanning the same already-bound LogicalPaimonTableSink; it does not reload the Paimon table or rebuild the write target.

The new CollectRelation handling can help detect a target schema change, but it does not refresh the pinned columns/table serialized to the writer. A concurrent remote schema change can therefore cause repeated retry failures or a mismatch between the analyzed output schema and the writer table generation.

Please rebuild the row-level DML plan/write target on retry, and add a concurrent schema-change test.

[P2] MERGE does not detect multiple source rows matching one target row

The implementation directly joins source and target and emits one changelog record per joined row. There is no cardinality check for multiple source rows matching the same target primary key. For deduplicate tables this can produce last-write-wins behavior whose result depends on execution/write order, instead of rejecting an ambiguous MERGE.

Please define and enforce the expected cardinality semantics and add a duplicate-source-key test.

Existing regression/coverage failures

external_table_p0/paimon/test_paimon_write_boundary still asserts that UPDATE, DELETE, and MERGE are unsupported. It now fails in External Regression and must be updated to validate the new behavior.
BE incremental coverage is currently 0/2 for the changed C++ lines.
FE incremental coverage is only 9/378 (2.38%).
The new regression verifies only a default deduplicate table with fixed buckets plus one UPDATE, one DELETE, and one mixed MERGE.

Missing test scenarios

Please add coverage for at least:

ignore-delete=true: standalone DELETE and MERGE DELETE must not silently succeed.
merge-engine=partial-update:
- update a non-null value;
- explicitly set a nullable column to NULL;
- MERGE UPDATE with an explicit NULL;
- delete behavior with partial-update.remove-record-on-delete;
- sequence-group removal options.
merge-engine=aggregate with and without aggregation.remove-record-on-delete.
MERGE with duplicate source keys matching one target key.
All unsupported-engine negative paths for UPDATE, DELETE, and MERGE, not only UPDATE on first-row.
Schema changes between initial analysis, retry, writer binding, and commit.
Multiple conditional MATCHED/NOT MATCHED clauses and branch-priority behavior.
Dynamic bucket modes, sequence fields, partitioned tables, and reordered/mixed-case columns.
UPDATE/DELETE with CTEs, joins/USING, aliases, ORDER BY/LIMIT where supported, and zero matched rows.
Failure atomicity: writer/commit failure after mixed INSERT/UPDATE/DELETE records must not partially apply the MERGE.
Large MERGE/UPDATE/DELETE workloads covering join spill, Paimon writer spill/backpressure, memory limits, cancellation, and OOM protection.
JNI/Thrift compatibility for the changed Java open signature and the new CHANGELOG write mode, including a clear mixed-version/rolling-upgrade behavior.

Given the two silent data-correctness failures above and the directly related red regression, I do not think this is ready to merge yet.

suxiaogang223 · 2026-08-06T07:40:24Z

run buildall

hello-stephen · 2026-08-06T09:18:24Z

FE UT Coverage Report

Increment line coverage 3.49% (17/487) 🎉
Increment coverage report
Complete coverage report

suxiaogang223 · 2026-08-06T12:27:03Z

run buildall

Gabriel39

I found two remaining row-level DML correctness gaps on the current head. Both can let a statement succeed while Paimon silently chooses or retains a different row version.

suxiaogang223 · 2026-08-07T06:40:16Z

run buildall

Gabriel39 · 2026-08-07T09:27:42Z

I completed a follow-up review of the current head ef5b271d. The latest commit addresses the previous inline findings about modifying sequence.field and duplicate primary keys among the rows emitted by the same NOT MATCHED INSERT branch. I found the following remaining issues.

[P1] `rowkind.field` overrides the RowKind emitted by Doris

PaimonWriteSchema.tableRow() sets the InternalRow RowKind from the Doris operation column, but Paimon 1.4.2 uses RowKindGenerator.getRowKind() when rowkind.field is configured. That generator takes precedence over row.getRowKind().

PaimonRowChangeCapabilities currently does not reject or otherwise neutralize this table option. Consequently:

DELETE can be converted back into INSERT/UPDATE when the stored rowkind field contains +I or +U, so the statement succeeds without deleting the row.
A MERGE NOT MATCHED INSERT carrying -D in that field can become a physical DELETE.
Updating the rowkind field can turn an UPDATE into a DELETE or another operation.

Please either reject row-level UPDATE/DELETE/MERGE on tables configured with rowkind.field, or construct a pinned writer table where the generator cannot override the operation selected by Doris. Please also add UPDATE, DELETE, and MERGE coverage for this option.

[P1] NOT MATCHED INSERT does not detect a primary key that already exists in the target

checkInsertPrimaryKeyUniqueness() only checks for duplicate projected keys among the INSERT rows emitted by this MERGE statement. It does not check those keys against existing target rows.

For example, given an existing target row (id=1, status='old'), a source row (id=1, status='new'), and:

ONt.id=s.idANDt.status=s.status

the source row is classified as NOT MATCHED and can emit an INSERT with id=1. A Paimon deduplicate table does not raise a primary-key violation; it applies upsert/sequence semantics and can silently replace or retain a row even though the SQL MERGE branch classified it as an insert.

Please either validate projected INSERT keys against the target primary keys, or restrict the MERGE ON condition so that it guarantees complete primary-key matching. A regression test should cover an additional non-key ON predicate producing this collision.

[P1] Nondeterministic INSERT key expressions are evaluated separately for validation and writing

The uniqueness projection calls generateFinalExpression() to materialize INSERT keys, and the final sink projection calls it again for the actual output row. Expressions such as rand() or uuid() can therefore produce one key during validation and a different key during writing. The uniqueness assertion can pass while the actual sink rows collide.

Please materialize the selected operation and row values once, then use the same slots for both uniqueness validation and the final sink output.

[P2] Unmatched rows form one unbounded NULL window partition

generateTargetMatchCount() partitions by target primary-key slots before branch filtering. In a LEFT JOIN, every NOT MATCHED row has NULL target-key slots, so all unmatched source rows enter the same window partition whenever the MERGE also contains a MATCHED clause.

A normal mixed MERGE with a small number of updates and millions of inserts can therefore create a single shuffle/buffer hotspot and cause excessive spill, memory growth, or OOM. Please run the target cardinality check only for matched rows, without placing all unmatched rows into one NULL partition, and add a mostly-unmatched large-input test.

Additional planner performance concern

Analyzer.buildAnalyzerJobs() now runs a second full bottom-up BindExpression pass for every analyzed statement, although the new expressions are introduced only by sink binding. The applied-aware conditions avoid most transformations, but every plan still pays for another traversal and rule matching. Please scope the rebind to newly created sink nodes if possible, or provide planner benchmark evidence showing that the global pass does not cause a meaningful analysis-time regression.

The current regression suite does not cover the four cases above. Given the silent row-operation/key correctness failures, I do not think this is ready to merge yet.

hello-stephen · 2026-08-07T10:16:35Z

BE UT Coverage Report

Increment line coverage 50.00% (4/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.38% (24785/42451)
Line Coverage	42.69% (250203/586129)
Region Coverage	38.72% (197663/510520)
Branch Coverage	40.15% (90601/225674)

hello-stephen · 2026-08-07T11:47:32Z

BE Regression && UT Coverage Report

Increment line coverage 50.00% (4/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.66% (30477/41374)
Line Coverage	57.65% (335354/581742)
Region Coverage	54.44% (278657/511883)
Branch Coverage	55.45% (124996/225436)

hello-stephen · 2026-08-07T12:00:36Z

FE Regression Coverage Report

Increment line coverage 1.85% (10/542) 🎉
Increment coverage report
Complete coverage report

suxiaogang223 · 2026-08-08T17:27:24Z

run buildall

Gabriel39 · 2026-08-09T14:21:38Z

/review

github-actions · 2026-08-09T15:23:40Z

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/31318240749

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

github-actions

Requesting changes for two silent row-level DML correctness failures.

Critical checkpoints:

Policy and authorization: LOAD/SELECT checks and row-policy filtering remain in place, but data-mask projections are reused as physical UPDATE/DELETE/MERGE values and keys. This can persist masked values or mutate the masked key instead of the stored row.
Paimon capability semantics: partial-update.remove-record-on-sequence-group is conditional, yet the gate treats option presence as an unconditional DELETE contract. An all-NULL designated group can leave the existing row while DELETE or MERGE DELETE reports success.
MERGE correctness: branch priority, PK proof, and matched/inserted cardinality checks are coherent for the admitted subset. Constant/source-free key mappings fail closed and are not a correctness defect.
ABI, routing, and lifecycle: the operation column, Thrift/JNI/Arrow/Paimon RowKind contract, bucket routing, CTE/EXPLAIN/schema retry, and prepare/abort/commit paths were traced without another issue.
Tests and documentation: add data-mask UPDATE/DELETE/MERGE regressions and an all-NULL sequence-group DELETE/MERGE regression. The PR description's partial-update UPDATE claim is stale because the implementation correctly rejects that engine.
Existing review context: the current head addresses the prior global sequence.field assignment and duplicate NOT MATCHED key threads; those were not duplicated here.
User focus and completion: no additional review focus was provided, so the full 35-file PR was reviewed. The maximum third round converged with no unresolved candidate at head 7256326d2dae2b1bf005ee7a4ea75f68470f53df.

No local builds or tests were run, as required by the review bundle.

hello-stephen · 2026-08-10T02:30:44Z

run buildall

hello-stephen · 2026-08-10T03:56:05Z

FE UT Coverage Report

Increment line coverage 2.16% (13/603) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-10T05:34:21Z

BE UT Coverage Report

Increment line coverage 50.00% (4/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.44% (24826/42483)
Line Coverage	42.77% (250922/586728)
Region Coverage	38.81% (198304/510934)
Branch Coverage	40.24% (90948/225995)

hello-stephen · 2026-08-10T07:09:59Z

BE Regression && UT Coverage Report

Increment line coverage 50.00% (4/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.78% (30543/41400)
Line Coverage	57.78% (336463/582333)
Region Coverage	54.66% (280023/512279)
Branch Coverage	55.61% (125539/225741)

suxiaogang223 · 2026-08-16T14:47:37Z

run buildall

hello-stephen · 2026-08-16T15:08:24Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.48% (1909/2464)
Line Coverage	64.29% (34164/53141)
Region Coverage	64.43% (17296/26845)
Branch Coverage	53.94% (9259/17166)

hello-stephen · 2026-08-16T18:14:43Z

FE UT Coverage Report

Increment line coverage 2.94% (20/681) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-16T20:03:17Z

BE Regression && UT Coverage Report

Increment line coverage 50.00% (4/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.90% (30819/41705)
Line Coverage	57.96% (340330/587135)
Region Coverage	54.83% (283582/517233)
Branch Coverage	55.73% (127480/228744)

hello-stephen · 2026-08-16T20:18:38Z

FE Regression Coverage Report

Increment line coverage 78.27% (533/681) 🎉
Increment coverage report
Complete coverage report

suxiaogang223 · 2026-08-17T02:22:19Z

run buildall

suxiaogang223 · 2026-08-17T02:26:06Z

run buildall

hello-stephen · 2026-08-17T18:44:22Z

FE Regression Coverage Report

Increment line coverage 78.33% (535/683) 🎉
Increment coverage report
Complete coverage report

1

### What problem does this PR solve? Issue Number: closeapache#65086 Related PR: apache#65868, apache#66498, apache#66685 Problem Summary: Forward-port the connector transaction, commit-message bridge, connector-neutral write distribution contract, and Paimon write-plan selection needed to execute table writes on master. The plan preserves native Paimon fixed-bucket routing and exposes row-change capabilities for subsequent DML planning. ### Release note Support planning and committing Paimon table writes through the connector framework. ### Check List (For Author) - Test: Manual test - FE checkstyle passed; final compilation is deferred until all forward-port picks are complete. - Behavior changed: Yes, Paimon writes now use connector transactions and native write distribution planning. - Does this need documentation: No

suxiaogang223 marked this pull request as ready for review August 5, 2026 11:33

suxiaogang223 requested a review from yiguolei as a code owner August 5, 2026 11:33

suxiaogang223 changed the title ~~[feature](paimon) Support row-level UPDATE, DELETE, and MERGE~~branch-4.1: [feature](paimon) Support row-level UPDATE, DELETE, and MERGEAug 5, 2026

suxiaogang223 force-pushed the codex/paimon-row-level-dml branch from ac1681a to d594db9Compare August 6, 2026 07:39

Gabriel39 reviewed Aug 6, 2026
View reviewed changes

Comment thread...-core/src/main/java/org/apache/doris/nereids/rules/analysis/PaimonRowChangeCapabilities.java
Comment thread...e-core/src/main/java/org/apache/doris/nereids/rules/analysis/PaimonRowChangePlanBuilder.java Outdated

suxiaogang223 force-pushed the codex/paimon-row-level-dml branch from f666857 to ef5b271Compare August 7, 2026 06:31

Gabriel39 added the dev/5.0.x label Aug 7, 2026

github-actionsBot requested changes Aug 9, 2026
View reviewed changes

Comment thread...e-core/src/main/java/org/apache/doris/nereids/rules/analysis/PaimonRowChangePlanBuilder.java
Comment thread...-core/src/main/java/org/apache/doris/nereids/rules/analysis/PaimonRowChangeCapabilities.java Outdated

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 16, 2026

suxiaogang223 force-pushed the codex/paimon-row-level-dml branch from fbeee92 to d3ed2fdCompare August 17, 2026 02:22

suxiaogang223 added 13 commits August 17, 2026 10:23

feat: support Paimon update delete merge
f488dd5

fix: bind Paimon row-level DML sink outputs
1427a1c

fix: reanalyze Paimon row-level DML sinks
39b76eb

fix: address BE build and formatting failures
68fdd8d

fix: enforce Paimon MERGE row-change semantics
547a7aa

refactor: simplify Paimon row-change planning
cd3f3a3

fix: reject unsafe Paimon row-level DML
fa1bfd0

fix: guard unsafe Paimon row changes
865f58e

fix: harden Paimon row-level DML planning
407ad89

fix: address Paimon row-level DML review findings
b027c1f

fix: preserve short-circuit semantics in expression rewrites
736a41a

test: strengthen short-circuit rewrite coverage
ef0eec1

fix: preserve short-circuit boundaries during CSE
34a1807

suxiaogang223 force-pushed the codex/paimon-row-level-dml branch from d3ed2fd to 34a1807Compare August 17, 2026 02:25

yiguolei approved these changes Aug 18, 2026
View reviewed changes

yiguolei merged commit 612dec7 into apache:branch-4.1Aug 18, 2026
29 of 33 checks passed

This was referenced Aug 31, 2026
Tracking: Paimon table write support #65086
Open
[feature](paimon) Forward-port Paimon table writes to master #67395
Open

Uh oh!

Conversation

suxiaogang223 commented Aug 5, 2026

What changed

Implementation

User impact

Validation

Uh oh!

hello-stephen commented Aug 5, 2026

Uh oh!

suxiaogang223 commented Aug 5, 2026

Uh oh!

hello-stephen commented Aug 5, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Aug 5, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 5, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 5, 2026

FE Regression Coverage Report

Uh oh!

Gabriel39 commented Aug 6, 2026

Correctness issues

[P1] DELETE can silently succeed without deleting rows when ignore-delete=true

[P1] UPDATE cannot set a nullable column to NULL on merge-engine=partial-update

[P2] Schema-change retry reuses the old immutable write target

[P2] MERGE does not detect multiple source rows matching one target row

Existing regression/coverage failures

Missing test scenarios

Uh oh!

suxiaogang223 commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

FE UT Coverage Report

Uh oh!

suxiaogang223 commented Aug 6, 2026

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

suxiaogang223 commented Aug 7, 2026

Uh oh!

Gabriel39 commented Aug 7, 2026

[P1] rowkind.field overrides the RowKind emitted by Doris

[P1] NOT MATCHED INSERT does not detect a primary key that already exists in the target

[P1] Nondeterministic INSERT key expressions are evaluated separately for validation and writing

[P2] Unmatched rows form one unbounded NULL window partition

Additional planner performance concern

Uh oh!

hello-stephen commented Aug 7, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 7, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 7, 2026

FE Regression Coverage Report

Uh oh!

suxiaogang223 commented Aug 8, 2026

Uh oh!

Gabriel39 commented Aug 9, 2026

Uh oh!

github-actionsBot commented Aug 9, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 10, 2026

[P1] DELETE can silently succeed without deleting rows when `ignore-delete=true`

[P1] UPDATE cannot set a nullable column to NULL on `merge-engine=partial-update`

[P1] `rowkind.field` overrides the RowKind emitted by Doris