Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [fix](variant) Avoid mutating shared variant columns - #64198
Merged
Conversation
Issue Number: None Related PR: None Problem Summary: Queries that evaluate VARIANT expressions after local exchange can share input blocks across downstream pipeline tasks. Variant casts and Variant serialization finalized source columns in-place, so one consumer could mutate a shared input column while another consumer still expected the original column shape and row count. This made local-shuffle query results unstable and could trigger later operators to observe changed Variant column contents or sizes. This change confines the fix to Variant handling by using private finalized Variant copies for cast and serialization paths instead of mutating the source column. None - Test: - PATH=/tmp/doris-clang-format-bin:$PATH build-support/clang-format.sh - git diff --check HEAD^ - ./build.sh --be - ./run-be-ut.sh --run --filter='ColumnVariantTest.serialize_does_not_finalize_source_column:ColumnVariantTest.block_serialize_does_not_finalize_source_column:FunctionVariantCast.CastFromVariantDoesNotFinalizeSourceColumn:FunctionVariantCast.CastFromVariant' - Manual test: with column_nullable.cpp and column_nullable_test.cpp reverted from this PR, local 1FE+4BE first local-shuffle repro passed 16x100 concurrent executions - Manual test: with column_nullable.cpp and column_nullable_test.cpp reverted from this PR, local 1FE+4BE second local-shuffle repro matched local-off baseline for 100 iterations - Behavior changed: No - Does this need documentation: No (cherry picked from commit 5078f25)
hello-stephen
commented
Jun 8, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
eldenmoon
commented
Jun 8, 2026
MemberAuthor
run buildall |
Contributor
There was a problem hiding this comment.
Pull request overview
This cherry-pick addresses a BE correctness issue where shared input ColumnVariant instances could be finalized (and therefore mutated) in-place during VARIANT casts or serialization, causing downstream pipeline tasks to observe inconsistent column structure/row counts after local exchange.
Changes:
- Update VARIANT cast-from logic to finalize a private cloned copy of the input variant column instead of mutating the source column.
- Update
DataTypeVariantserialization sizing/serialization to operate on a private finalized clone when the input is not finalized. - Add unit tests to ensure casts and serialization do not finalize (mutate) the source variant column, and that
clone_finalized()deep-copies internal columns.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| be/test/exprs/function/cast/function_variant_cast_test.cpp | Adds coverage ensuring VARIANT CAST does not finalize/mutate the source column. |
| be/test/core/column/column_variant_test.cpp | Adds tests for clone_finalized() deep-copy behavior and for non-mutating variant serialization paths. |
| be/src/exprs/function/cast/cast_to_variant.h | Changes cast-from-variant implementation to finalize a private clone for string/JSONB paths (avoiding shared-input mutation). |
| be/src/core/data_type/data_type_variant.cpp | Changes VARIANT serialized-size and serialization to use a private finalized clone when needed (avoiding shared-input mutation). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
yiguolei
approved these changes
Jun 8, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-pick #64092