Uh oh!
There was an error while loading. Please reload this page.
[feature](variant) Support ColumnVariantV2 segment reads and writes - #66204
Conversation
hello-stephen
commented
Jul 29, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
eldenmoon
commented
Jul 29, 2026
run buildall |
5f99d8c to
06f2ec2Compareeldenmoon
commented
Jul 29, 2026
run buildall |
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Jul 29, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jul 29, 2026
TPC-H: Total hot run time: 29804 ms |
hello-stephen
commented
Jul 29, 2026
TPC-DS: Total hot run time: 177543 ms |
hello-stephen
commented
Jul 29, 2026
ClickBench: Total hot run time: 25.03 s |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Jul 29, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: None Related PR: apache#66204 Problem Summary: ColumnVariantV2 encoded Variant scalar bytes and unpacked owning VariantField values itself, leaking Variant V2 framing knowledge into the column layer. Reuse the existing typed-column-to-VariantScalarRef conversion as the single scalar mapping, let VariantField own only framing and byte ownership, and expose read-only encoded slices for insertion. NULL follows the same scalar ownership entry while the existing lazy legacy-map behavior remains unchanged. ### Release note None ### Check List (For Author) - Test: Unit Test - ASAN focused Variant field and column tests: 16 passed - ASAN Variant wildcard: 633 passed, 5 skipped - Behavior changed: No. This refactor preserves Variant value encoding and V1 lazy legacy-map semantics while simplifying V2 ownership boundaries. - Does this need documentation: No
eldenmoon
commented
Aug 10, 2026
run buildall |
hello-stephen
commented
Aug 10, 2026
TPC-H: Total hot run time: 29389 ms |
hello-stephen
commented
Aug 10, 2026
TPC-DS: Total hot run time: 158160 ms |
hello-stephen
commented
Aug 10, 2026
ClickBench: Total hot run time: 23.87 s |
hello-stephen
commented
Aug 10, 2026
FE Regression Coverage ReportIncrement line coverage |
eldenmoon
commented
Aug 10, 2026
/review ignore the previous already fixed issue and expected issue, if no new issue, approve |
eldenmoon
commented
Aug 10, 2026
run check_coverage |
eldenmoon
commented
Aug 10, 2026
run buildall |
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
hello-stephen
commented
Aug 10, 2026
TPC-H: Total hot run time: 28395 ms |
hello-stephen
commented
Aug 10, 2026
TPC-DS: Total hot run time: 158507 ms |
hello-stephen
commented
Aug 10, 2026
ClickBench: Total hot run time: 23.91 s |
hello-stephen
commented
Aug 10, 2026
FE Regression Coverage ReportIncrement line coverage |
eldenmoon
commented
Aug 11, 2026
run check_coverage |
PR approved by at least one committer and no changes requested. |
eldenmoon
commented
Aug 11, 2026
skip check_coverage |
What problem does this PR solve?
Issue Number: None
Related PR: #65561
Problem Summary:
This PR adds the ColumnVariantV2 segment read and write paths while preserving the existing V1 behavior.
For reads, eligible scans assemble legacy Variant segment data into ColumnVariantV2 through plan/type metadata. Typed, sparse, document, and root values retain their native representation, and the reader covers RowID, hierarchical multi-batch, cache, malformed-input, and legacy scalar cases.
For writes,
Config.enable_variant_v2is propagated throughVariantType.toThrift()to select the V2 storage type. V1 keeps the existing STRING-to-Variant sink path, while V2 inputs are explicitly parsed into ColumnVariantV2. Nereids load planning injectstry_parse_to_variantonly when V2 is enabled. The change deliberately removes the V2-to-V1 cast bridge because a statement is not expected to mix the two physical representations.The segment layer keeps version-native SQL NULL and JSON null behavior without adding conversion state or per-row normalization. Complex and nested arrays that cannot use the same typed layout as V1 fall back to JSONB, which also makes V1 and V2
DESCoutput consistent.V2-only parsing and compute coverage is isolated under
variant_p0/v2and dynamically enables the FE config. It covers casts, aggregation, DISTINCT, UNION/INTERSECT/EXCEPT, conditionals, arrays, explode, INSERT SELECT, and segment round trips. Shared suites keep V1 string literals and use explicitparse_to_variantfor V2.Release note
Support ColumnVariantV2 segment reads and writes when
enable_variant_v2is enabled. The legacy V1 path remains available when the config is disabled.Check List (For Author)
variant_p0with V1: 146/147 suites passed.variant_p0with V2: 146/147 suites passed.InvalidAccessKeyId.DESCcoverage passed.git diff --checkpassed.enable_variant_v2selects the Variant physical read/write path and enables V2-only compute behavior.Scope notes
BaseViewInfo,AggStateType, andBindSinkremain aligned with master.