[feature](paimon) Add Paimon 2.0 transactional writes

[damokelis]

What problem does this PR solve?

Issue Number: close#66784

Related PR: #65868

Problem Summary:

The Connector SPI path on master can read Paimon tables, but it still uses Paimon 1.3.1 and does not provide a complete external-table write path. The implementation from branch-4.1 cannot be applied directly because catalog, transaction, planning, scan, and write contracts have moved to the connector modules on master.

This draft:

• upgrades the FE connector, BE JNI scanner/writer, and shaded Hive runtime to Apache Paimon 2.0.0;
• adds Connector SPI write planning and a Paimon table sink backed by the Paimon Java SDK;
• aggregates BE commit messages in the FE transaction and provides acknowledgement, retry reconciliation, abort cleanup, and statement-level commit;
• supports external catalog CREATE TABLE, DROP TABLE, INSERT INTO, INSERT OVERWRITE (full-table and static-partition), ALTER TABLE ... DROP PARTITION, and CTAS paths;
• supports column schema evolution (ALTER TABLE ADD/DROP/RENAME/MODIFY COLUMN) for both flat and nested schemas;
• supports row-level DELETE, UPDATE, and MERGE INTO on primary-key tables, and DELETE, UPDATE, and MERGE INTO on append-only tables with deletion vectors enabled;
• auto-heals every FE's table cache after an out-of-band schema change from an external engine (Spark/Flink/…) sharing the same warehouse, via an opt-in background poller;
• maps primitive and complex values through Arrow, including Paimon 2.0 BLOB, VARIANT, and VECTOR representations;
• supports Parquet and Vortex reads, with automatic JNI fallback for schemas that the native reader cannot decode;
• rejects invalid Vortex schemas before the remote catalog create call.

Release note

Support transactional writes to Apache Paimon 2.0 external catalogs, including CREATE/DROP, INSERT, INSERT OVERWRITE (full-table and static-partition), DROP PARTITION, CTAS, column schema evolution, row-level DELETE/UPDATE/MERGE INTO on both primary-key and append-only (deletion-vector) tables, and opt-in automatic healing after an out-of-band external schema change.

Check List (For Author)

• Test
  • Regression test
  • Unit Test
  • Manual test
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

Current validation:

• FE Paimon connector: 552 tests, 0 failures, 1 skipped.
• C++ focused unit tests: 13/13 passed.
• Linux release BE incremental build: passed.
• Changed-line clang-tidy for all 5 modified C++ files: passed.
• clang-format, checkstyle, and git diff --check: passed.
• TKE live deployment:
  • backend ID 1786733167633 remained stable and is alive on 172.21.48.181;
  • deployed image digest: sha256:8f82d5cf69147101a15e6c46f90523391c1a07a0ea28226901fd36bf677f40b6;
  • doris_be SHA256: 055313b210249c0f595f97211855590d0e95336655bbdd03de93b5bc9d9f4887;
  • Paimon scanner SHA256: 0a9b83bb92f376275ef9a428fdc333d414e60c0813a3cad4c175c4b9e44b4b8e;
  • COS Parquet+VARIANT query returned the expected rows, and EXPLAIN VERBOSE reported SplitStat [type=JNI] with force_jni_scanner=false;
  • Vortex+VARIANT create was rejected before the REST catalog call, with no residual table;
  • 28/31 continuously tiered COS tables completed exact COUNT(*) scans; the remaining three were blocked by external COS getFileStatus API timeouts, including after raising the S3A request timeout from 15s to 120s. Their Paimon snapshot/file metadata remained readable.
• Continuous CDC remained healthy during validation:
  • both DTS-to-Fluss tasks kept advancing with queues draining to zero;
  • Fluss-to-Paimon tiering checkpoints advanced through checkpoint 2347 with no failed table epochs.

Production rollout on a disaggregated-storage cluster (2026-08-20):

The branch was built as a disaggregated-storage image set (build.sh --be --fe --cloud, docker/runtime/{be,fe,ms} entrypoints) and rolled out to a production Doris cluster (3 FE + 4 BE, shared FoundationDB metastore) that already serves other lakehouse workloads. This tracked three prior rollout attempts and their fixes:

• Attempt 1/2 hit a be_exec_version protocol mismatch during the mixed-version rolling window and a transient brpc exchange-unavailability window (5-15s) caused by simultaneous BE pod restarts under the StatefulSet controller — both were operational/rollout-sequencing issues, not code defects. Root-caused via canary reproduction with isolated pod-kill/rolling experiments.
• Attempt 3 hit a real code defect: static-partition INSERT OVERWRITE aborted the BE with DORIS_CHECK_EQ(column_names.size(), block.columns()). Root cause: BindSink materializes the partition literal into the projected output (full schema including the partition column), but PaimonWritePlanProvider still built TPaimonTableSink.column_names from handle.getColumns(), which excludes the partition column in the static-partition case. Fixed in 2147f234 by switching to handle.getBoundTargetColumns() (the full bound schema) when a static partition is present. Covered by a new unit test asserting column_names size matches the bound schema, not the INSERT column list.
• Attempt 4 (this build, 2147f234) succeeded: all 7 components rolled node-by-node (BE first, then FE follower → follower → master) with the operator scaled to zero to avoid partition resets, verified Alive/zero-restart at each step, and confirmed static-partition INSERT OVERWRITE, idempotent partition re-overwrite, and full-table INSERT OVERWRITE against real COS-backed Paimon tables post-rollout. Ingestion lag stayed ≤14s throughout with no interruption.

Post-rollout stress test (2026-08-20, 913M rows, isolated catalog against real COS): insert-select throughput peaked at 730K rows/s (780M-row backfill tranche in 1069s), with 64 output files averaging 245MB (near the 256MB target, no fragmentation). A 913M-row count(*) returned in 6.6s and a GROUP BY + count(distinct) aggregation in 9.2s. The full DML/DDL matrix (INSERT/UPDATE/DELETE on primary-key tables, MERGE INTO, DELETE on deletion-vector tables, ALTER ADD/RENAME/MODIFY/DROP COLUMN, CTAS, time travel, concurrent writes to the same table, and transaction atomicity under a server-side KILL QUERY) passed. Production stayed at zero restarts and ≤5s ingestion lag throughout.

The stress test surfaced one caching defect, since root-caused and fixed in this PR ([fix](paimon) evict the CachingCatalog's frozen Table on invalidateTable/Db/All): on a multi-FE cluster, ALTER TABLE ... ADD COLUMN left every FE other than the master serving a frozen pre-ALTER Table object out of the paimon SDK's CachingCatalog (column DDL forwards to the master, and only the mutating catalog instance self-invalidates; the default meta.cache.paimon.table.ttl-second is 24h). The metadata path already reads the latest schema live (schemaManager().latest()), so DESC/binding saw the new column — but the scan path serializes the frozen Table to the BE, and once a post-ALTER commit creates an overlapping sorted run (a JNI merged-read split), PaimonJniScanner failed every read on those FEs with The jni reader fields' size {N} is not matched with paimon fields' size {M}. REFRESH TABLE could not heal it despite correct routing on every FE, because PaimonConnector.invalidateTable dropped the three Doris-side caches but never touched the CachingCatalog. The fix routes all three invalidation scopes through the paimon catalog's own eviction API; verified on a 3-FE disaggregated cluster where the follower-read repro (create → write → ALTER → write → SELECT * on a follower) failed deterministically before the fix and passes after, plus a unit test that reproduces the frozen-Table state against a real CachingCatalog with an out-of-band ALTER. Until a deployment carries this fix, the workaround is ALTER CATALOG <name> SET PROPERTIES ("meta.cache.paimon.table.ttl-second"="0"). Re-verified the full DML/DDL matrix against a fresh 920M-row backfill after deploying this fix to production (FE only, BE unchanged): tier-4 backfill throughput held at 1075s vs. 1069s pre-fix (no regression), and the follower-ALTER repro plus a 3-round consecutive ALTER ADD/ADD/DROP-with-immediate-write-and-read stress passed cleanly.

Three follow-up gaps identified during that validation, all closed in this PR:

• ALTER TABLE ... DROP PARTITION was rejected on every external table (not Paimon-specific — the allow-list in AlterTableCommand never included it). Paimon's own InnerTableCommit (the same commit handle INSERT OVERWRITE already uses) already exposes truncatePartitions(List<Map<String,String>>), so this was pure plumbing: an SPI seam (ConnectorTableDdlOps#dropPartitions), a PaimonConnectorMetadata implementation that resolves Doris's displayed partition name back to Paimon's native spec (reusing the same rendering SHOW PARTITIONS already shares, so IF EXISTS gets existence-checking for free), and a one-shot newCommit(user).truncatePartitions() call — no schema change, so no PaimonConnectorTransaction involvement.
• UPDATE / MERGE INTO on append-only (no primary key) tables with deletion-vectors.enabled were rejected with "Only DELETE is supported", even though DELETE already worked there. The missing piece: an UPDATE/MERGE needs deletion-vector marks for the matched rows' old positions AND the new row values appended as fresh data, in the SAME commit. PaimonJniWriter.prepareCommitMessages() already merged the plain writer's data-file CommitMessages with the deletion-vector collector's index-file CommitMessages into one list every write — the write-side machinery for "both halves in one commit" already existed; the row-level-DML gate just refused to route non-DELETE operations there. Extending the same RowKind-tagged dispatch primary-key UPDATE/MERGE already uses (a matched-update/delete row records its locator via the existing PaimonDeletionVectorCollector, an insert/replacement row writes through the normal append path) closed it.
• An out-of-band ALTER from an external engine (Spark/Flink/… on the same warehouse) produced no Doris event, so every FE's CachingCatalog — and the separate fe-core ExtMetaCache — stayed stale up to the 24h TTL with no path to a manual REFRESH. Paimon exposes no schema-change listener, so this needed an opt-in background poller (meta.cache.paimon.external-change-poll-interval-second, <= 0 disables it — unchanged prior behavior by default) that compares each cached table's held vs. latest schema id (one small metadata-file read, no data scan) and evicts on drift. The first wiring only reached the connector's own cache (PaimonConnector#invalidateTable) — a cluster test then showed the query-facing symptom was still only half-fixed: the connector healed internally, but a query still bound against the stale column list, because fe-core's ExtMetaCache is a separate cache the connector module architecturally cannot reach. Closed with a new SPI seam, ConnectorContext#notifyExternalTableChanged (default no-op), that DefaultConnectorContext implements by resolving the remote names to the catalog's loaded local table and calling RefreshManager.refreshTableInternal directly — the same fe-core-side refresh a manual REFRESH TABLE triggers. Re-ran the identical out-of-band-ALTER repro after this half landed: the new column appeared with zero manual REFRESH, confirmed via the FE log showing PaimonExternalChangePoller → RefreshManager.refreshTableInternal in sequence.

fe-connector-spi + fe-connector-paimon: 600 tests, 0 failures (up from the 599 at the CachingCatalog-fix commit); be-java-extensions/paimon-scanner: 85 tests, 0 failures; fe-core AlterTableCommandTest (17) + DropPartitionTest (3): 0 failures. Checkstyle clean across all four touched modules. All three closed on a 3-FE + 4-BE disaggregated cluster (doris-verify): DROP PARTITION leaves sibling partitions untouched and IF EXISTS is silent on a missing one; UPDATE/MERGE INTO on an append-only deletion-vector table update in place with no row-count drift; the external-change poller genuinely self-heals a follower's stale schema with no manual REFRESH.

Requesting a committer to trigger /review: this PR is ready_for_review and CI-green on the checks that don't require repo permissions (Check large file, Sync review status), but code-review has been stuck PENDING since — per the bot's own message — "Trigger /review to start automated review" requires a MEMBER/OWNER/COLLABORATOR to post the comment. I've posted /review twice (most recently for e9ba7bf5, the CachingCatalog fix) with no effect from this account. Would appreciate anyone with the right permissions triggering it so the full buildall pipeline and automated review can run.

• Behavior changed:

  • No.
  • Yes. Paimon external catalogs gain transactional DDL/DML writes and use Paimon 2.0.0.

• Does this need documentation?

  • No.
  • Yes. A follow-up documentation PR will describe supported catalog properties, statements, and type mappings.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[damokelis]

Pushed 4 new commits extending this PR from read/insert support to the full write surface for Paimon 2.0:

• [fix](build) hive-shade artifact via explicit outputFile (clean builds failed with main artifact does not exist)
• [fix](filesystem) accept the dotted s3.path.style.access alias every existing regression catalog uses (silently defaulting to false breaks hostname-addressed S3-compatible endpoints; CI never saw it because an IP endpoint force-switches the SDK to path style)
• [feature](paimon) column schema evolution — ALTER TABLE ADD/DROP/RENAME/MODIFY COLUMN, flat and nested dotted-path, with nullability/comment change gates and a statement-fence pin that no longer time-travels the schema generation
• [feature](paimon) row-level DELETE / UPDATE / MERGE INTO — keyed writes on primary-key tables (operation-tagged stream), deletion-vector DELETE on unaware-bucket append tables (merging with existing vectors), actionable rejections elsewhere

Capability matrix as implemented and verified end-to-end (minio + FE + BE from this branch, both new regression suites green):

table shape	ALTER	DELETE	UPDATE	MERGE
primary-key	✅ flat+nested	✅ keyed	✅	✅ incl. matched-delete
append-only + deletion vectors (unaware bucket)	✅	✅ DV write + vector merge	rejected with hint	rejected
append-only w/o vectors, bucketed append	✅	rejected with hint	rejected	rejected

Two pre-existing issues surfaced while validating and are worth separate attention: (1) several assembly descriptors reference reactor modules their pom does not declare (e.g. fe-core's make-fe-lib packaging hadoop-deps), so a clean mvn package hits isn't a file unless retried — build.sh's retry masks it; (2) ColumnPath.fromDotName has no callers — nested column DDL only enters through the parser's multipart identifier path.

[damokelis]

/review

[damokelis]

/review

[damokelis]

Found while smoke-testing the DROP PARTITION / RENAME·TRUNCATE TABLE / append-only UPDATE-MERGE features on a production disaggregated-storage Doris cluster running this branch (commit base 2147f234, images fe/be-master-paimon2-2147f234-r9/r6).

Symptom: UPDATE against a Paimon primary-key table fails when the assignment is an un-cast decimal literal targeting a DOUBLE/FLOAT column:

CREATETABLEpk_orders (id BIGINT, region STRING, amount DOUBLE, dt DATE)
ENGINE=paimon PARTITION BY LIST(dt)()
PROPERTIES ('primary-key'='id,dt', 'bucket'='4');
INSERT INTO pk_orders VALUES (1,'east',100.0,DATE'2026-08-01'); -- succeedsUPDATE pk_orders SET amount =999.0WHERE id =1; -- fails

[JNI_ERROR]JNI exception in JniPaimonWriter::write: RuntimeException: PaimonJniWriter write failed: bytes=1272 | CAUSED BY: ClassCastException: class org.apache.paimon.data.Decimal cannot be cast to class java.lang.Double (org.apache.paimon.data.Decimal is in unnamed module of loader org.apache.doris.common.classloader.JniScannerClassLoader; java.lang.Double is in module java.base of loader 'boot...

Workaround confirmed: explicit cast avoids it —

UPDATE pk_orders SET amount = CAST(999.0AS DOUBLE) WHERE id =1; -- succeeds

Failure is atomic — the target row/table is left untouched after the failed UPDATE, no partial write observed.

Scope check: this looks distinct from the schema-drift issue 9c5c6ea9 already fixed (that one was about PaimonWritePlanProvider#validateBoundColumns rejecting writes due to a DECIMALV3 type-name mismatch between bind-time and execute-time schemas). Here the write is accepted and reaches the JNI writer, but the literal's inferred DECIMALV3 runtime value is never converted to the target column's declared type before being handed to JniPaimonWriter::write — INSERT ... VALUES (..., 999.0, ...) into the same DOUBLE column works fine, so the gap seems specific to the UPDATE row-level-DML assignment-expression path (likely somewhere around PaimonWritePlanProvider / the JNI writer's value materialization, haven't pinned an exact file:line yet).

Happy to help narrow this down further if useful — let me know if a minimal repro project/branch would help.

[damokelis]

Rebased onto current master (f054492cbb9). The only conflict was in PaimonConnector.java, from #66633 (refactor Unify external metadata cache framework) collapsing the three connector-owned caches into metaCache — resolved by keeping the new metaCache.invalidateTable/Database/Catalog() calls and re-attaching the paimon-SDK-side CachingCatalog eviction on top of them.

I have also split four self-contained fixes out of this PR so they can be reviewed independently of the Paimon 2.0 upgrade. None of them depend on any Paimon 2.0 code, and each fixes a defect that is present on master today:

PR	Scope	Size
#67459	[fix](be) duplicate default argument on the clear_blocks forward declaration added by #66400 — a hard -Werror failure under Unity Build	+1/-1
#67460	[fix](filesystem) accept the dotted s3.path.style.access alias, the spelling 42 existing regression catalogs already use	+3/-2
#67461	[fix](build)fe-connector-hms-hive-shade clean build fails with main artifact does not exist	+9
#67462	[fix](nereids) gate IcebergRowLevelDmlTransform on catalog identity, so RowLevelDmlRegistry stays correct once a second connector declares row-level capabilities	+17/-1

Those four are still included here as well, so this PR remains self-contained; whichever land first, I will rebase this one on top and drop the duplicates.

On scope: this PR is large because Paimon 2.0 is a breaking SDK upgrade — the connector runtime bump, the write path, and the read path cannot land separately without leaving the build broken in between. If it would help review, I am happy to split the remainder further along whatever seam you prefer (e.g. read/upgrade first, then transactional writes, then schema evolution and row-level DML). Guidance on the preferred sequencing would be very welcome.