Skip to content

[feature](paimon) Forward-port Paimon table writes to master - #67395

Open
suxiaogang223 wants to merge 12 commits into
apache:masterfrom
suxiaogang223:codex/forward-pick-paimon-write-master
Open

[feature](paimon) Forward-port Paimon table writes to master#67395
suxiaogang223 wants to merge 12 commits into
apache:masterfrom
suxiaogang223:codex/forward-pick-paimon-write-master

Conversation

@suxiaogang223

@suxiaogang223suxiaogang223 commented Sep 1, 2026•
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close#65086

Related PR: #65868, #65955, #66321, #66325, #66334, #66346, #66498, #66547, #66612, #66626, #66685, #66810, #66990, #67023, #67085

Problem Summary:

Forward-port the complete Paimon table-write path from branch-4.1 to master in one integrated change, avoiding repeated rewrites of shared FE, BE, JNI, and connector interfaces.

This PR:

  • Upgrades the Paimon dependency baseline from 1.3.1 to 1.4.2. A Paimon 2.0 upgrade is intentionally deferred.
  • Adds the BE Paimon sink, JNI/Java writer, commit payload transport, memory lifecycle, and spill handling.
  • Adds native Paimon bucket routing and writer assignment.
  • Adds connector transactions, write planning, and commit handling.
  • Adds INSERT plus row-level DELETE, UPDATE, and MERGE planning.
  • Adds Variant V2 write serialization and JNI read support.
  • Keeps the Doris and Paimon table caches coherent for writes.
  • Forward-ports the final branch-4.1 Paimon write regression matrix and generated expected outputs.
  • Adapts branch-4.1 code to master APIs and compile checks.

PR #66790 is intentionally excluded from this forward-port.

Release note

Support writing Apache Paimon tables, including INSERT, DELETE, UPDATE, MERGE, native bucket routing, schema evolution, and Variant V2 values. The bundled Paimon runtime is upgraded to 1.4.2.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
      • build-support/clang-format.sh
      • build-support/check-format.sh
      • build-support/check-build-hygiene.sh
      • Maven checkstyle validation: 0 violations
      • BE compilation completed successfully through ./build.sh --be --fe
      • Final FE and Java extensions compilation completed successfully through ./build.sh --fe
    • No need to test or manual test. Explain why:

  • Behavior changed:

    • No.
    • Yes. Paimon tables can be written through the Doris execution pipeline and support row-level DML.

  • Does this need documentation?

    • No.
    • Yes. Documentation can follow in a separate PR.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
[chore](paimon) Upgrade Paimon to 1.4.2
551f4c1
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#66346
Problem Summary: Upgrade the master Paimon runtime from 1.3.1 to 1.4.2 and adapt the JNI scanner and Connector SPI read path to the updated fallback-table, catalog, partition, scan-statistics, and split-count APIs. This establishes the dependency baseline required by the branch-4.1 table-write forward-port without importing unrelated write implementations.
### Release note
Upgrade the Paimon connector runtime to 1.4.2.
### Check List (For Author)
- Test: Unit Test
- PaimonJniScannerTest: 33 tests passed
- Connector Paimon targeted tests: 182 tests passed
- Maven validate passed
- Behavior changed: Yes (Paimon runtime is upgraded from 1.3.1 to 1.4.2)
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Add Paimon table write runtime
03181cb
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#65868, apache#66612, apache#66810
Problem Summary: Add the BE Paimon sink, JNI writer backend, Java writer, commit payload transport, and worktree-local spill and memory lifecycle support required to write Paimon tables from master.
### Release note
Support writing Apache Paimon tables through the native Doris execution pipeline.
### Check List (For Author)
- Test: Static validation
- BE clang-format/check-format and FE checkstyle; full build and tests are deferred until all forward-port picks are complete
- Behavior changed: Yes, adds Paimon table writes
- Does this need documentation: Yes, documentation can follow separately
@suxiaogang223
[feature](paimon) Route Paimon writes by native bucket semantics
23dd8fc
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#66685
Problem Summary: Forward-port external sink hash partitioning, Paimon fixed-bucket routing, native row hashing, and writer assignment so distributed Doris writers match Paimon bucket ownership.
### Release note
Paimon writes now use native Paimon-compatible routing for fixed bucket tables.
### Check List (For Author)
- Test: Static validation
- BE clang-format/check-format and FE checkstyle; full build and tests are deferred until all forward-port picks are complete
- Behavior changed: Yes, changes Paimon writer distribution
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Add connector transaction and write planning
21c96da
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#65868, apache#66498, apache#66685
Problem Summary: Forward-port the connector transaction, commit-message bridge, connector-neutral write distribution contract, and Paimon write-plan selection needed to execute table writes on master. The plan preserves native Paimon fixed-bucket routing and exposes row-change capabilities for subsequent DML planning.
### Release note
Support planning and committing Paimon table writes through the connector framework.
### Check List (For Author)
- Test: Manual test
- FE checkstyle passed; final compilation is deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon writes now use connector transactions and native write distribution planning.
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Add row-level Paimon DML
f966736
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#66990
Problem Summary: Forward-port changelog planning for Paimon DELETE, UPDATE, and MERGE statements. The planner emits row-kind plus full-row records, validates deterministic primary-key behavior for MERGE, and preserves branch evaluation with an explicit short-circuit expression.
### Release note
Support DELETE, UPDATE, and MERGE writes for Paimon tables.
### Check List (For Author)
- Test: Manual test
- FE checkstyle passed; final compilation is deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon tables support connector row-level DML planning.
- Does this need documentation: No
@suxiaogang223
[feature](paimon) Support Variant V2 Paimon writes
6cd62fa
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#66321
Problem Summary: Forward-port Variant V2 serialization for Paimon Arrow batches, including encoded value/type byte pairs, outer-null preservation, supported nested constructors, and array-to-Variant conversion behavior.
### Release note
Support writing Variant V2 values to Paimon tables.
### Check List (For Author)
- Test: Unit Test
- Added serialization and cast coverage; BE format/check and FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete.
- Behavior changed: Yes, Variant V2 values can be serialized for Paimon writes.
- Does this need documentation: No
@suxiaogang223
[improvement](paimon) Read Variant through Paimon JNI
590e4f8
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#66334, apache#66547, apache#66626
Problem Summary: Forward-port Variant V2 decoding across the Paimon Java scanner and native JNI bridge. The reader propagates nested access paths, materializes encoded Variant bytes into ColumnVariantV2, and accepts Paimon required-value leaves for fallback-only Variant arrays.
### Release note
Improve Paimon Variant reads with nested projection and native Variant V2 materialization.
### Check List (For Author)
- Test: Unit Test
- Added Java vector/projection and BE Parquet schema/reader coverage; BE format/check, header hygiene, and FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon Variant reads support nested projection and fallback-array schemas.
- Does this need documentation: No
@suxiaogang223
[fix](paimon) Keep write table cache coherent
f137671
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#67023
Problem Summary: Doris and Paimon maintain separate table caches. After a Doris metadata-cache miss, Paimon CachingCatalog could return the same stale table handle, leaving writes planned with outdated bucket, path, or table options. Invalidate the inner Paimon entry before every outer-cache reload and verify the invalidate-before-get ordering.
### Release note
Ensure refreshed Paimon writes use current table metadata.
### Check List (For Author)
- Test: Unit Test
- Added cache reload ordering coverage; FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete.
- Behavior changed: Yes, Paimon table reloads no longer reuse stale inner-cache entries.
- Does this need documentation: No
@suxiaogang223
[test](paimon) Forward-port table write coverage
a9d096e
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#65868, apache#66325, apache#67023, apache#67085
Problem Summary: Forward-port the final branch-4.1 Paimon write regression matrix to master, including append and primary-key modes, bucket routing, transactions, schema evolution, row-level DML, Variant V2, snapshot references, writer lifecycle, cache coherence, and stabilized external paths. Generated golden outputs are preserved byte-for-byte from branch-4.1.
### Release note
None
### Check List (For Author)
- Test: Regression test
- Test cases and generated expected outputs were forward-ported; execution is deferred until the final integrated validation.
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
[fix](be) Align Paimon partitioners with master compile checks
a99f294
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#66685
Problem Summary: branch-4.1 wrapped new partitioner code with compile_check_begin/end headers that no longer exist on master because conversion warnings are enforced globally. Remove the obsolete wrappers so the forward-ported Paimon routing code builds under the master warning policy.
### Release note
None
### Check List (For Author)
- Test: Manual test
- BE full build exposed the missing headers; clang-format/check and build hygiene passed. The incremental full build continues after this commit.
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
[fix](paimon) Fix table options test compilation
a61e01e
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#67023
Problem Summary: The forward-ported Paimon table cache coherence test uses List to record catalog calls but did not import java.util.List, causing FE test compilation to fail. Add the missing import.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Maven checkstyle validation for the Paimon connector
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
[fix](fe) Use expression arguments in array legality check
b6bf67d
### What problem does this PR solve?
Issue Number: closeapache#65086
Related PR: apache#66321
Problem Summary: The Variant V2 forward-port referenced a branch-specific arguments field while validating ARRAY children. Master exposes legality-check inputs through getArguments(), so FE compilation failed. Use the master expression API while preserving validation of every array argument.
### Release note
None
### Check List (For Author)
- Test: Manual test
- FE compilation identified the issue; final incremental build follows this commit
- Behavior changed: No
- Does this need documentation: No
@suxiaogang223
suxiaogang223force-pushed the codex/forward-pick-paimon-write-master branch from 21ae767 to b6bf67dCompareSeptember 3, 2026 16:00
@suxiaogang223

suxiaogang223 commented Sep 3, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 3.48% (25/719) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 3.20% (23/719) 🎉
Increment coverage report
Complete coverage report

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929924060929Awaiting requested review from 924060929924060929 is a code owner
@CalvinKirsCalvinKirsAwaiting requested review from CalvinKirsCalvinKirs is a code owner
@morrySnowmorrySnowAwaiting requested review from morrySnowmorrySnow is a code owner
@engleflyengleflyAwaiting requested review from engleflyenglefly is a code owner
@starocean999starocean999Awaiting requested review from starocean999starocean999 is a code owner
@Gabriel39Gabriel39Awaiting requested review from Gabriel39Gabriel39 is a code owner
@yiguoleiyiguoleiAwaiting requested review from yiguoleiyiguolei is a code owner
@morningmanmorningmanAwaiting requested review from morningmanmorningman is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Tracking: Paimon table write support

2 participants

@suxiaogang223@hello-stephen