Uh oh!
There was an error while loading. Please reload this page.
[feature](variant) Enable Variant V2 by default - #66858
Draft
eldenmoon wants to merge 12 commits into
Draft
Conversation
hello-stephen
commented
Aug 18, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
eldenmoon
commented
Aug 18, 2026
MemberAuthor
run buildall |
1 similar comment
eldenmoon
commented
Aug 18, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Aug 18, 2026
Contributor
TPC-H: Total hot run time: 16987 ms |
hello-stephen
commented
Aug 18, 2026
Contributor
TPC-DS: Total hot run time: 81251 ms |
hello-stephen
commented
Aug 18, 2026
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
hello-stephen
commented
Aug 18, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Aug 18, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
eldenmoonforce-pushed
the
branch-variant-v2-default-on
branch
from
August 18, 2026 10:44
3e2ed7f to
488ebeaCompareeldenmoon
commented
Aug 18, 2026
MemberAuthor
run buildall |
eldenmoonforce-pushed
the
branch-variant-v2-default-on
branch
from
August 18, 2026 12:44
488ebea to
2ab8c68Comparehello-stephen
commented
Aug 18, 2026
Contributor
TPC-H: Total hot run time: 17509 ms |
eldenmoon
commented
Aug 18, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Aug 18, 2026
Contributor
TPC-DS: Total hot run time: 83404 ms |
hello-stephen
commented
Aug 18, 2026
Contributor
ClickBench: Total hot run time: 14.58 s |
hello-stephen
commented
Aug 18, 2026
Contributor
TPC-H: Total hot run time: 17370 ms |
hello-stephen
commented
Aug 18, 2026
Contributor
TPC-DS: Total hot run time: 83633 ms |
hello-stephen
commented
Aug 18, 2026
Contributor
ClickBench: Total hot run time: 14.68 s |
hello-stephen
commented
Aug 18, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Aug 18, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
eldenmoon
commented
Aug 19, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Aug 19, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
eldenmoon
commented
Aug 20, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Aug 20, 2026
Contributor
TPC-H: Total hot run time: 17287 ms |
hello-stephen
commented
Aug 20, 2026
Contributor
TPC-DS: Total hot run time: 83206 ms |
hello-stephen
commented
Aug 20, 2026
Contributor
ClickBench: Total hot run time: 14.8 s |
eldenmoonforce-pushed
the
branch-variant-v2-default-on
branch
from
September 1, 2026 02:05
b4d155b to
e9e58c2Compareeldenmoon added a commit
to eldenmoon/incubator-doris
that referenced
this pull request
Sep 1, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: Variant has moved to V2-only execution and storage. The old ColumnVariant V1 implementation kept duplicate factories, cast/load routing, writer/readers, and regression toggles behind enable_variant_v2. This refactor removes the V1 column/data-type/serde implementation, makes Variant factories and storage boundaries produce and accept ColumnVariantV2 only, lets non-V2 compute carriers fail at V2-only boundaries, and updates BE/FE/regression tests to validate V2-only behavior. String-like inputs to Variant are parsed directly into V2 encoded JSON so storage and element access no longer depend on the legacy ColumnVariant carrier. ### Release note Variant now uses V2 only; the legacy enable_variant_v2 FE config and ColumnVariant V1 implementation are removed. ### Check List (For Author) - Test: Regression test / Unit Test - Unit Test: BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh --run --filter=ColumnVariantV2Test.*:VariantPathBuilderTest.*:VariantShredderTest.*:VariantColumnWriterReaderTest.*:HierarchicalDataIteratorTest.*:VariantUtilTest.*:VariantDocModeCompactionTest.*:NestedGroupProviderTest.*:DefaultNestedGroupReadProviderTest.*:VariantRowStoreTest.*:DataTypeVariantV2SerDe*:*VariantV2*; BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh --run --filter=BinaryColumnExtractIteratorV2Test.*; BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh --run --filter=CastVariantV2ToTest.*:VariantElementV2EncodedTest.*:VariantElementV2TypedTest.*; BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF DISABLE_BUILD_UI=ON ./run-fe-ut.sh --run org.apache.doris.nereids.trees.expressions.functions.agg.CountTest,org.apache.doris.nereids.load.VariantLoadParseInjectionTest,org.apache.doris.datasource.scan.PluginDrivenScanNodeCompatibilityTest - Regression test: ./run-regression-test.sh --run -d variant_p1 passed; ./run-regression-test.sh --run -d variant_p0 passed all runnable suites except regression-test/suites/variant_p0/doc_mode/test_outfile_csv_variant_type.groovy, which failed on external OSS InvalidAccessKeyId 403; affected variant_p0 cases were rerun individually after output regeneration. - Build: BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF DISABLE_BUILD_UI=ON ./build.sh --be --fe passed; default GLIBC compatibility link failed on duplicate getrandom from the local toolchain, and FE UI build failed with Node 16.3.0 missing node:util.styleText. - Behavior changed: Yes. Variant no longer supports the legacy V1 ColumnVariant implementation or enable_variant_v2 fallback. - Does this need documentation: No
eldenmoon
commented
Sep 1, 2026
MemberAuthor
run buildall |
eldenmoon
commented
Sep 1, 2026
| // be deleted without changing the V2 representation. | ||
| // Transitional field-map representation kept only for Field API callers. | ||
| VariantField(VariantMap legacy); | ||
| bool is_legacy() const noexcept; |
hello-stephen
commented
Sep 3, 2026
Contributor
TPC-DS: Total hot run time: 82358 ms |
hello-stephen
commented
Sep 3, 2026
Contributor
ClickBench: Total hot run time: 14.46 s |
eldenmoon
commented
Sep 4, 2026
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
Sep 4, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
eldenmoon added a commit
to eldenmoon/incubator-doris
that referenced
this pull request
Sep 4, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: Variant V2 rejected an entire row when a nested array element did not match the target dimension, although non-strict casts should null only the offending element. Nested Variant subpaths rebuilt for constant UNION and one-row relations were also reversed. In addition, several regression cases skipped supported Variant V2 behavior or weakened their assertions. This change uses the existing element cast semantics, preserves root-to-leaf path order, and restores direct coverage for nested arrays, full Variant reads, partial updates, compaction, sparse strings, materialized views, and SQL/JSON null distinctions. ### Release note Fix Variant V2 nested array cast and nested subpath pruning semantics. ### Check List (For Author) - Test: Regression test / Unit Test - VariantPruningLogicTest - CastVariantV2FromTest - Focused affected regression suites - variant_p0 (169/170; one OSS credential failure) - variant_p1 (4/4) - Release and ASAN BE builds - Behavior changed: Yes. Nested array cast mismatches now null the offending element instead of the whole row, and constant nested paths retain root-to-leaf order. - Does this need documentation: No
eldenmoon
commented
Sep 4, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Sep 4, 2026
Contributor
TPC-H: Total hot run time: 17069 ms |
hello-stephen
commented
Sep 4, 2026
Contributor
TPC-DS: Total hot run time: 82137 ms |
hello-stephen
commented
Sep 4, 2026
Contributor
ClickBench: Total hot run time: 14.65 s |
hello-stephen
commented
Sep 4, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Sep 4, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Sep 4, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Sep 4, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
Issue Number: None Related PR: apache#66204 Problem Summary: ColumnVariantV2 is implemented in master but remains opt-in through Config.enable_variant_v2, so normal Variant planning and storage continue to select the legacy path. Enable the existing V2 path by default while preserving enable_variant_v2=false as an explicit fallback. Adapt affected regression inputs to use parse_to_variant in V2 mode, keep shared V1/V2 result oracles representation-independent where serialization differs, and retain legacy-only cast coverage without changing production cast behavior. Enable ColumnVariantV2 by default for Variant execution and storage. The legacy path remains available with enable_variant_v2=false. - Test: - Full ASAN BE+FE build without -j - FE ConfigTest (7/7) - FE VariantEqualityContextTest (3/3) - BE focused unit tests (78/78) - Original affected regression suites (24/24, including two export suites against local MinIO) - Additional CI regression suites (8/8 with Variant V2 enabled, plus V1 fallback matrix) - Regression Groovy compilation - clang-format, check-format, and git diff --check - Behavior changed: Yes (ColumnVariantV2 is enabled by default; explicit false remains supported) - Does this need documentation: No
Issue Number: None Related PR: apache#66858 Problem Summary: Variant has moved to V2-only execution and storage. The old ColumnVariant V1 implementation kept duplicate factories, cast/load routing, writer/readers, and regression toggles behind enable_variant_v2. This refactor removes the V1 column/data-type/serde implementation, makes Variant factories and storage boundaries produce and accept ColumnVariantV2 only, lets non-V2 compute carriers fail at V2-only boundaries, and updates BE/FE/regression tests to validate V2-only behavior. String-like inputs to Variant are parsed directly into V2 encoded JSON so storage and element access no longer depend on the legacy ColumnVariant carrier. Variant now uses V2 only; the legacy enable_variant_v2 FE config and ColumnVariant V1 implementation are removed. - Test: Regression test / Unit Test - Unit Test: BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh --run --filter=ColumnVariantV2Test.*:VariantPathBuilderTest.*:VariantShredderTest.*:VariantColumnWriterReaderTest.*:HierarchicalDataIteratorTest.*:VariantUtilTest.*:VariantDocModeCompactionTest.*:NestedGroupProviderTest.*:DefaultNestedGroupReadProviderTest.*:VariantRowStoreTest.*:DataTypeVariantV2SerDe*:*VariantV2*; BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh --run --filter=BinaryColumnExtractIteratorV2Test.*; BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh --run --filter=CastVariantV2ToTest.*:VariantElementV2EncodedTest.*:VariantElementV2TypedTest.*; BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF DISABLE_BUILD_UI=ON ./run-fe-ut.sh --run org.apache.doris.nereids.trees.expressions.functions.agg.CountTest,org.apache.doris.nereids.load.VariantLoadParseInjectionTest,org.apache.doris.datasource.scan.PluginDrivenScanNodeCompatibilityTest - Regression test: ./run-regression-test.sh --run -d variant_p1 passed; ./run-regression-test.sh --run -d variant_p0 passed all runnable suites except regression-test/suites/variant_p0/doc_mode/test_outfile_csv_variant_type.groovy, which failed on external OSS InvalidAccessKeyId 403; affected variant_p0 cases were rerun individually after output regeneration. - Build: BUILD_TYPE=ASAN GLIBC_COMPATIBILITY=OFF DISABLE_BUILD_UI=ON ./build.sh --be --fe passed; default GLIBC compatibility link failed on duplicate getrandom from the local toolchain, and FE UI build failed with Node 16.3.0 missing node:util.styleText. - Behavior changed: Yes. Variant no longer supports the legacy V1 ColumnVariant implementation or enable_variant_v2 fallback. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: Casting STRING, CHAR, or VARCHAR to Variant V2 incorrectly parsed the input as a JSON document. Route string types through the scalar conversion path so the Variant retains its string type and value. Add a focused unit test that distinguishes a JSON-looking string from an object. ### Release note Casting string values to VARIANT preserves them as strings. Use parse_to_variant for JSON document parsing. ### Check List (For Author) - Test: Unit Test - CastVariantV2ToTest: 14 passed - Behavior changed: Yes, string-to-VARIANT casts no longer parse JSON text. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: Variant element access rejected an empty object key in both FE analysis and BE path resolution, even though the empty string is a valid JSON object key and existing Variant behavior exposes it. Remove those rejections and cover direct, sparse-bucket, rename, and compaction reads. ### Release note VARIANT object fields with empty-string keys remain addressable. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - BE focused unit tests: 151 passed, 2 skipped, 0 failed - FE ElementAtTest: 1 passed - Regression: test_variant_empty_key_sparse_bucket and test_variant_compaction_empty_path_bug passed - Full ASAN BE and FE build passed - Behavior changed: Yes, empty-string Variant object keys are accepted and readable - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: The V2-only implementation still carried duplicate V1-era state and branches in TabletColumn, storage readers, VariantField, block transforms, array helpers, and hash joins. Remove those unused switches and no-op paths, reuse the existing V2 representation throughout, and restore the predefine regression input so its other_1 values remain JSON strings before and after compaction. ### Release note None ### Check List (For Author) - Test: Regression test / Unit Test - BE focused unit tests: 151 passed, 2 skipped because NestedGroup write support is unavailable, 0 failed - Regression: test_all_prdefine_type_to_sparse passed - Full ASAN BE and FE build passed - clang-format and build-hygiene checks passed - Behavior changed: No, this removes obsolete V1-only internal state and no-op work - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: Variant V2 pre-created exact typed paths even when the input path was missing or entirely JSON null, and flat-leaf compaction reconstructed materialized and sparse children only to discard them in the root-only writer. Public Variant aliases also lacked the BIGINT element_at overload after the V2-only migration, while Decimal256 storage read failures were not covered at their real materialized and sparse boundaries. Remove typed-path pre-initialization, keep the converter flow driven by observed non-null leaves, add a root-only mode to the existing hierarchical iterator, restore the integer selector overload, and add focused storage error coverage without exception allowlists. ### Release note Variant V2 no longer creates physical typed columns for missing or all-JSON-null paths. Flat-leaf compaction reads only the persisted root sidecar, element_at(VARIANT, BIGINT) is supported, and unsupported Decimal256 full-Variant reads return explicit storage-boundary errors. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - ASAN BE build with GLIBC_COMPATIBILITY=OFF - Variant BE UT: 642 tests, 637 passed and 5 skipped - variant_p0: 169/169 included suites passed; one OSS credential-dependent suite excluded after confirmed 403 - variant_p1: 4/4 suites passed - Focused Variant CAST, element_at, Decimal256 storage, RQG, segment golden, and compaction tests passed - Two array order-by suites remain blocked in FE analysis because multi_distinct_count does not support ARRAY - Behavior changed: Yes, as described in the release note - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: The main VARIANT compaction regression returned before executing any storage logic, and the current V2 writer cannot produce a real legacy V1 segment for compatibility testing. Add an immutable segment generated by the removed V1 RowsetWriter, read it through the current rowset reader, compact it together with a fresh V2 rowset, and compare every complete VARIANT root after vertical cumulative compaction. Reactivate the storage regressions, remove the deprecated flatten-nested property, and use a currently supported nested-array case. ### Release note None ### Check List (For Author) - Test: - Unit Test: ASAN legacy V1 fixture compaction test and focused VARIANT compaction tests - Regression test: test_compaction_variant and test_compaction_nokey_variant - Build: ASAN BE build - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: The V2-only Variant migration made multi_distinct_count reuse COUNT DISTINCT legality checks, which rejected ARRAY inputs. Variant join-key casts could also remain above a cross join and be evaluated once per Cartesian-product row, causing q09_trans to time out. Remove the obsolete legality override and push side-local, non-volatile expressions below cross-join inputs while preserving output identities and volatile-expression evaluation semantics. ### Release note Restore multi_distinct_count on arrays and avoid repeated Variant join-key casts above cross joins. ### Check List (For Author) - Test: Regression test / Unit Test - variant_p0: 169/170 passed; one OSS InvalidAccessKeyId environment failure - variant_p1: 4/4 passed - Variant BE ASAN UT: 638 passed, 5 capability skips, 0 failed - PushDownExpressionsInHashConditionTest: 8/8 passed - q09_trans, test_array_order_by, test_array_string_order_by passed - Behavior changed: Yes. Restore array multi_distinct_count and evaluate side-local non-volatile expressions before cross joins without moving volatile expressions. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: Empty ColumnVariantV2 destinations encoded typed sources during generic bulk copies. Cross joins therefore converted a materialized Variant child once per Cartesian-product row, making q09_trans spend most of its time in the cross operator. Preserve the homogeneous typed representation for range, repeated, and indexed copies, and convert explicitly only at storage writer boundaries that require encoded rows. Revert the broader FE cross-join expression pushdown because the representation fix addresses the root cause without changing optimizer scope. ### Release note Preserve typed Variant values across bulk copies to avoid repeated encoding during joins. ### Check List (For Author) - Test: Regression test / Unit Test - variant_p0 and variant_p1: 173 suites passed, 0 failed; external OSS credential case excluded - Variant BE ASAN UT: 620 passed, 5 capability skips, 0 failed - q09_trans, test_array_order_by, and test_array_string_order_by passed - PushDownExpressionsInHashConditionTest: 6/6 passed - Release BE build passed - Behavior changed: Yes. Empty Variant destinations retain homogeneous typed storage during bulk copies; storage writers still encode explicitly. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: MultiDistinctCount intentionally allows ARRAY arguments, but a stale FE unit test still expected the temporary V2-migration rejection. Restore the test expectation to match the SQL behavior and the legality-check removal. ### Release note None ### Check List (For Author) - Test: Unit Test - CountTest#testMultiDistinctCountAllowsArray - CountTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#66858 Problem Summary: Variant V2 uses a canonical empty-object default, preserves raw string CAST input as a string, and has its own native and Segment storage encoding. Tests and checked-in fixtures that still assumed Variant V1 semantics therefore failed after the V2 migration. Update the expectations and explicit JSON parsing, and regenerate only the affected V2 fixtures after validating the complete Segment golden set. ### Release note None ### Check List (For Author) - Test: - Unit Test - Regression test - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66858 Problem Summary: Variant V2 rejected an entire row when a nested array element did not match the target dimension, although non-strict casts should null only the offending element. Nested Variant subpaths rebuilt for constant UNION and one-row relations were also reversed. In addition, several regression cases skipped supported Variant V2 behavior or weakened their assertions. This change uses the existing element cast semantics, preserves root-to-leaf path order, and restores direct coverage for nested arrays, full Variant reads, partial updates, compaction, sparse strings, materialized views, and SQL/JSON null distinctions. ### Release note Fix Variant V2 nested array cast and nested subpath pruning semantics. ### Check List (For Author) - Test: Regression test / Unit Test - VariantPruningLogicTest - CastVariantV2FromTest - Focused affected regression suites - variant_p0 (169/170; one OSS credential failure) - variant_p1 (4/4) - Release and ASAN BE builds - Behavior changed: Yes. Nested array cast mismatches now null the offending element instead of the whole row, and constant nested paths retain root-to-leaf order. - Does this need documentation: No
eldenmoonforce-pushed
the
branch-variant-v2-default-on
branch
from
September 6, 2026 03:15
55e1a38 to
8be9b8fCompareeldenmoon
commented
Sep 6, 2026
MemberAuthor
run buildall |
hello-stephen
commented
Sep 6, 2026
Contributor
TPC-H: Total hot run time: 16884 ms |
hello-stephen
commented
Sep 6, 2026
Contributor
TPC-DS: Total hot run time: 82062 ms |
hello-stephen
commented
Sep 6, 2026
Contributor
ClickBench: Total hot run time: 14.56 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #66204
Problem Summary: ColumnVariantV2 is implemented in master but remains opt-in through Config.enable_variant_v2, so normal Variant planning and storage continue to select the legacy path. Make the existing V2 path the default while preserving enable_variant_v2=false as an explicit fallback.
Enabling V2 by default exposed regression cases that assumed legacy implicit string-to-Variant assignment or legacy output representation. Update those cases following variant_p0 conventions: construct V2 values with parse_to_variant and normalize only representation-dependent expectations. Production Cast behavior is unchanged. Keep the hierarchical iterator test portable across standard library implementations.
Release note
Enable ColumnVariantV2 by default for Variant execution and storage. The legacy path remains available with enable_variant_v2=false.
Check List (For Author)