branch-4.1: [feature](paimon) Support Variant V2 writes

[suxiaogang223]

What problem does this PR solve?

Issue Number: Part of #65086

Problem Summary:

The Paimon JNI writer already supports primitive and complex Doris columns, but a Doris Variant V2 value could not be written losslessly because the generic Arrow path exposed Variant as JSON text. That loses the native Variant value and metadata representation and does not support Variant nested in ARRAY, MAP, or STRUCT.

This PR adds a Paimon-specific Variant V2 write protocol:

• maps Paimon VARIANT targets, including nested VARIANT nodes, to Doris compute Variant V2 during analysis;
• requires enable_variant_v2=true and rejects legacy Variant V1 inputs during analysis with an actionable error;
• transports Variant V2 through Arrow as struct<value: binary, metadata: binary> while preserving SQL NULL separately from a Variant null value;
• recursively applies the binary representation inside ARRAY, MAP, and STRUCT;
• converts the Arrow pair to Paimon GenericVariant in the Java JNI backend and uses Paimon SDK accessors to validate compatibility with the bundled Paimon version;
• keeps the existing JSON Arrow representation as the default for non-Paimon consumers.

Supported write scenarios

• INSERT INTO ... VALUES and INSERT INTO ... SELECT, including UNION inputs and reordered/omitted columns.
• INSERT OVERWRITE and static partition writes.
• Append-only and primary-key tables, fixed and dynamic bucket modes, and schema evolution involving VARIANT columns.
• Top-level VARIANT and VARIANT nested in ARRAY, MAP, STRUCT, and deeper combinations.
• Scalar/object/array values, SQL NULL, Variant null, typed primitives, long strings, and residual object fields.
• Paimon Parquet Variant shredding. Regression coverage verifies the physical typed_value layout and values, Paimon unshredding, type-mismatch fallback to residual bytes, and reading a table containing both unshredded and shredded files.

Current limitations

• Variant writes are V2-only. When enable_variant_v2=false, or when a legacy Variant V1 expression is supplied to a Paimon VARIANT target, analysis fails intentionally.
• The write path uses the Java Paimon JNI backend.
• Doris-side querying of Paimon Variant V2 is outside this PR. End-to-end logical readback is therefore verified with the Spark/Paimon reader; raw Parquet shredding is independently verified through Doris's S3 TVF.
• The added Paimon integration coverage uses Parquet, which is also the format used to validate Variant shredding.

Release note

Support writing Variant V2 values, including nested Variant values, from Doris into Apache Paimon tables through the Java JNI writer. Set enable_variant_v2=true before writing a Paimon table containing VARIANT.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test
    • Debug FE and BE incremental builds.
    • ./run-regression-test.sh --run -s test_paimon_write_variant_shredding
    • ./run-regression-test.sh --run -s test_paimon_write_variant_table_modes

• Behavior changed:

  • No.
  • Yes. Paimon VARIANT targets now accept Variant V2 writes when enabled and reject disabled/V1 writes during analysis.

• Does this need documentation?

  • No.
  • Yes. User-facing documentation can follow separately.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[suxiaogang223]

run buildall

[suxiaogang223]

/review

[github-actions]

Request changes: three P1 defects and one P2 repository-contract issue.

Correctness and analysis:

• Heterogeneous multi-row VALUES are coerced to one common primitive type before the Variant sink cast, silently changing per-row Variant types.
• The Paimon analyzer accepts Variant V2 source shapes and scalar types that the BE kernel cannot execute, including legacy leaves hidden by shape changes and FE-admitted unsupported sources.
• The new BE Arrow test constructs an explicitly unsupported typed TIMEV2 Variant and deterministically aborts before serialization.

Transport and compatibility:

• I traced encoded, typed, const, nullable, sliced, and nested Variant values through the C++ Arrow builders, IPC ownership, Java conversion, and Paimon compatibility walk. Aside from the failing test input above, byte ownership, null/child alignment, and the private Doris-produced compatibility boundary did not reveal another defect.
• The table-wide enable_variant_v2 gate matches the PR's documented opt-in policy.

Repository guidance:

• All four touched checkLegalityBeforeTypeCoercion overrides still bypass the required getArgument/getArguments accessors.

Tests and validation:

• This was a static-only review as required by the runner instructions; no local builds or tests were run. The added coverage is broad, but it lacks regressions for the accepted analysis/coercion cases and contains the deterministic test-construction failure called out inline.

User focus and completion:

• No additional user-provided review focus was supplied, so the complete 29-file PR diff was reviewed.
• Review completed after three bounded convergence rounds, final changed-file and unresolved-candidate sweeps, and a live head/duplicate check.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 49.09% (54/110) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 32.73% (36/110) 🎉
Increment coverage report
Complete coverage report

[suxiaogang223]

run buildall

[hello-stephen]

BE UT Coverage Report

Increment line coverage 80.95% (51/63) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.04% (24314/41891)
Line Coverage	42.14% (242705/575975)
Region Coverage	38.21% (191652/501528)
Branch Coverage	39.53% (87220/220663)

[suxiaogang223]

/review

[github-actions]

Static review of the full 49-file change at c5bacf78662f73a05ac5ddf2df92eb92f3da6f85 found six blocking issues: the new BinaryBuilder transport can overflow on a batch of individually valid Variants; the Paimon 1.4.2 bump breaks mixed-version serialized table/split payloads; the Docker fixture still lacks the newly mounted 1.4.2 jars; the Java-UDF assembly leaves an orphan standard-JDK chronology provider descriptor; FE admits non-empty ARRAY although the BE encoder rejects it; and the dynamic-bucket regression cannot detect key/payload permutation.

I also traced the required legality-check accessors, nested Variant/null/ownership paths, Paimon wrapper precedence, overwrite binding/commit lifecycle, SDK artifacts, and the remaining regression surface. Those paths did not yield additional non-duplicate findings. There was no additional user-provided review focus, so the whole PR was reviewed. The repository code-review skill was absent from this checkout; the authoritative review bundle and required AGENTS policy were followed directly. This was a static-only review; no builds or tests were run.

[suxiaogang223]

run buildall

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 81.69% (116/142) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.62% (30318/41184)
Line Coverage	57.49% (332462/578256)
Region Coverage	54.36% (276552/508788)
Branch Coverage	55.33% (123754/223673)

[suxiaogang223]

run external

[suxiaogang223]

run buildall

[suxiaogang223]

run buildall

[suxiaogang223]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31160789496

Please trigger /review again after that time.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 53.41% (94/176) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 74.43% (131/176) 🎉
Increment coverage report
Complete coverage report

[suxiaogang223]

run buildall

[suxiaogang223]

run buildall

[Gabriel39]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/31318357740

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[github-actions]

Review result

Requesting changes for four distinct pre-sink Variant V2 type-loss mechanisms, grouped into two inline comments at their shared changed-line anchors:

1. Container unification can scalarize a V2 leaf both inside a heterogeneous Array/CreateMap and later when indexed-Any array functions combine independently homogeneous arrays.
2. Result coercion can scalarize V2 before Paimon validation through heterogeneous set/conditional/selector expressions, while one-argument LEAD/LAG can select a trivial scalar signature and turn object/array roots into NULL.

I did not duplicate the current-head threads covering the non-Parquet Variant format fence, aggregate Arrow/IPC sizing, repeated shared-metadata validation/copying, mixed Paimon serialization versions, or the ServiceLoader descriptor. The final static sweep also confirmed the changed legality hooks use getArgument(s), recursive no-op layout checks retain exact container/field structure, writer and committer share the configured overwrite binding, fallback order matches the adjacent factory precedence, and the changed tests reach their intended existing fixes.

This review is capped and incomplete rather than converged: the third and final permitted round produced two new valuable findings, and the review contract permits no fourth round. All four substantiated mechanisms are reported here, but I cannot claim a clean convergence round for this head.

No additional user-provided focus was specified. No builds or tests were run, as required by the review contract; this was a full static review of the authoritative bundle at 027e11f67c1296d78118a87d5a31be3bd6b9935a.

[github-actions]

PR approved by at least one committer and no changes requested.

[yiguolei]

skip buildall

[yiguolei]

skip buildall

np