[Feature](paimon) Support Branch/Tag lifecycle and branch writes

[suxiaogang223]

Background

Apache Doris can read Apache Paimon branches and tags, including branch-specific schemas and retained tag snapshots. However, Doris cannot currently create or drop Paimon branches/tags, and Paimon branches cannot be used as Doris write targets.

This issue tracks the Branch/Tag lifecycle and branch-write follow-up to #65086.

Upstream reference: https://paimon.apache.org/docs/1.4/maintenance/manage-branches/

Current Status

• Read from table@branch(name): supported.
• Read from table@tag(name): supported.
• Query $branches and $tags system tables: supported.
• Create/drop Branch or Tag from Doris SQL: not supported.
• Write to a Branch from Doris SQL: not supported.
• Tag writes: must remain unsupported because tags are immutable references.
• Fast Forward / Merge / Replace operations: not supported from Doris.

Goals

Branch and Tag Lifecycle

• Create a branch from the current table state, a snapshot, or a tag.
• Drop and rename branches.
• Create a tag from the latest or a specified snapshot.
• Support tag retention when the Paimon version and catalog provide it.
• Drop and rename tags.
• Support IF EXISTS, IF NOT EXISTS, and idempotent retry semantics where applicable.
• Invalidate Doris metadata, schema, snapshot, and partition caches after reference changes.

The canonical Doris interface should reuse the existing external-table Branch/Tag DDL model. Procedure aliases may be added by the separate maintenance-procedure issue, but must call the same underlying implementation.

Branch Writes

• Support INSERT INTO and INSERT OVERWRITE on a Paimon branch.
• Support row-level UPDATE, DELETE, and MERGE on a Paimon branch where the target table supports them.
• Bind analysis, distribution, writer initialization, commit, abort, and reconciliation to the selected branch table and its schema generation.
• Keep transaction identity, metadata refresh, and cache keys scoped to the branch.
• Reject every write to a tag during analysis.
• Never fall back to main when branch resolution or branch commit fails.

Promotion and Conflict Semantics

• Support Fast Forward from a branch to main.
• Define Merge / Replace behavior according to the Paimon version used by Doris.
• Detect incompatible branch ancestry, schema history, and concurrent changes before promotion.
• Define behavior when main or the branch advances during a Doris statement.
• Preserve Paimon commit/abort and retry semantics on branch-targeted transactions.

Interoperability

• Read and write branches created by Flink or Spark.
• Make Doris-created branches and tags visible and usable from Flink and Spark.
• Cover filesystem, HMS, and REST catalogs.

Acceptance Criteria

• Branch INSERT, overwrite, and row-level DML mutate only the selected branch.
• Main, other branches, and tags remain unchanged after a successful or failed branch write.
• Tag writes fail during analysis with a clear error.
• Branch schema evolution is resolved from the branch rather than the latest main schema.
• Fast Forward succeeds only for a valid ancestry and leaves Doris caches consistent with Paimon metadata.
• Commit timeout, retry, abort, FE-side reconciliation, and concurrent promotion cases have regression coverage.
• Cross-engine tests cover branches/tags created and consumed by Doris, Spark, and Flink.

Suggested Delivery Phases

1. Branch/Tag create, drop, read interoperability, and immutable Tag enforcement.
2. Branch INSERT INTO / INSERT OVERWRITE with transaction isolation.
3. Branch UPDATE / DELETE / MERGE and Fast Forward.
4. Merge / Replace, retention, rename, and extended concurrency coverage.

Non-goals

• A generic Catalog procedure framework.
• Snapshot expiration, orphan-file cleanup, compaction scheduling, or rollback procedures.
• Rust FFI write backend work.

Related

• Paimon write tracking: Tracking: Paimon table write support #65086
• Paimon maintenance procedures and actions: [Feature](catalog) Support Paimon maintenance procedures and actions #67337