[Improvement](serialize) use streamvbyte_encode in DataTypeFixedLengthObject::serialize by BiteTheDDDDt · Pull Request #60526 · apache/doris

BiteTheDDDDt · 2026-02-05T05:35:15Z

What problem does this PR solve?

This pull request introduces a new serialization format for DataTypeFixedLengthObject columns, leveraging streamvbyte encoding for efficient storage and transmission of large data blocks. The new format is activated for BE exec version 10 and above, which is now set as the maximum supported version. Additionally, the AggregateFunctionCount and AggregateFunctionCountNotNullUnary functions are marked as trivial, likely for optimization purposes. Below are the most important changes:

Serialization and Deserialization Improvements

Introduced a new serialization/deserialization format for DataTypeFixedLengthObject that uses streamvbyte encoding for large data, improving efficiency for big data columns. The new logic is gated behind BE exec version 10 and includes fallback to the previous format for older versions (be/src/vec/data_types/data_type_fixed_length_object.cpp). [1][2][3][4]
Updated the calculation of uncompressed serialized bytes to account for the new serialization format and potential streamvbyte compression (be/src/vec/data_types/data_type_fixed_length_object.cpp).
Added the streamvbyte library include to support the new encoding/decoding logic (be/src/vec/data_types/data_type_fixed_length_object.cpp).

Version Management

Increased BeExecVersionManager::max_be_exec_version from 8 to 10, with detailed documentation and warnings about the sensitivity of this field. The new version enables the updated serialization logic (be/src/agent/be_exec_version_manager.cpp).
Defined a new constant USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10 to clearly mark the threshold for the new serialization format (be/src/agent/be_exec_version_manager.h).

Aggregate Function Optimization

Marked AggregateFunctionCount and AggregateFunctionCountNotNullUnary as trivial by overriding the is_trivial() method to return true, which may allow for performance optimizations in the aggregation engine (be/src/vec/aggregate_functions/aggregate_function_count.h). [1][2]

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-02-05T05:35:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

BiteTheDDDDt · 2026-02-05T05:36:36Z

run buildall

BiteTheDDDDt · 2026-02-05T05:37:36Z

run buildall

Copilot

Pull request overview

This pull request introduces a new serialization format for DataTypeFixedLengthObject using streamvbyte encoding to improve efficiency for large data columns. The BE exec version is bumped from 8 to 10 to enable this new format. Additionally, AggregateFunctionCount and AggregateFunctionCountNotNullUnary are marked as trivial for potential optimization benefits.

Changes:

Introduced streamvbyte-based serialization/deserialization for DataTypeFixedLengthObject when data exceeds 256 bytes, gated behind BE exec version 10
Bumped max_be_exec_version from 8 to 10 with comprehensive documentation and warnings
Marked count aggregate functions as trivial, consistent with sum and avg functions

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

File	Description
be/src/vec/data_types/data_type_fixed_length_object.cpp	Implements new streamvbyte-based serialization format for large data with version gating
be/src/agent/be_exec_version_manager.h	Defines new version constant and removes unused old version constants
be/src/agent/be_exec_version_manager.cpp	Updates max version to 10 with detailed documentation and compatibility warnings
be/src/vec/aggregate_functions/aggregate_function_count.h	Marks count aggregate functions as trivial for optimization

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-05T05:41:12Z

+ memcpy(buf, origin_data, mem_size);
+ return buf + mem_size;
+ } else {
+ // Throw exception if mem_size is large than UINT32_MAX


The comment has a grammatical error: "large than" should be "larger than". Additionally, the comment is misleading because streamvbyte_encode will not throw an exception if mem_size is larger than UINT32_MAX; instead, cast_set will throw an exception during the conversion at line 61. Consider revising the comment to accurately reflect this behavior.
Suggested change
//Throw exception if mem_size is large than UINT32_MAX
//cast_set will throw an exception if mem_size exceeds UINT32_MAX during conversion

hello-stephen · 2026-02-05T06:48:53Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

BiteTheDDDDt · 2026-02-05T09:03:56Z

run buildall

hello-stephen · 2026-02-05T10:08:23Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

BiteTheDDDDt · 2026-02-06T05:21:09Z

run buildall

hello-stephen · 2026-02-06T06:24:43Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-02-06T08:34:23Z

BE UT Coverage Report

Increment line coverage 94.92% (56/59) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.60% (19409/36896)
Line Coverage	36.12% (180588/500028)
Region Coverage	32.44% (139904/431303)
Branch Coverage	33.46% (60607/181109)

hello-stephen · 2026-02-06T10:03:00Z

BE Regression && UT Coverage Report

Increment line coverage 96.61% (57/59) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.63% (25900/36157)
Line Coverage	54.27% (270722/498804)
Region Coverage	51.63% (224950/435687)
Branch Coverage	53.14% (96610/181817)

hello-stephen · 2026-02-06T11:50:40Z

BE Regression && UT Coverage Report

Increment line coverage 96.61% (57/59) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.64% (25902/36157)
Line Coverage	54.28% (270766/498804)
Region Coverage	51.67% (225106/435687)
Branch Coverage	53.15% (96643/181817)

HappenLee

LGTM

github-actions · 2026-02-09T06:40:08Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-02-09T06:40:10Z

PR approved by anyone and no changes requested.

…hObject::serialize (apache#60526) This pull request introduces a new serialization format for `DataTypeFixedLengthObject` columns, leveraging streamvbyte encoding for efficient storage and transmission of large data blocks. The new format is activated for BE exec version 10 and above, which is now set as the maximum supported version. Additionally, the `AggregateFunctionCount` and `AggregateFunctionCountNotNullUnary` functions are marked as trivial, likely for optimization purposes. Below are the most important changes: * Introduced a new serialization/deserialization format for `DataTypeFixedLengthObject` that uses streamvbyte encoding for large data, improving efficiency for big data columns. The new logic is gated behind BE exec version 10 and includes fallback to the previous format for older versions (`be/src/vec/data_types/data_type_fixed_length_object.cpp`). [[1]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48L36-L41) [[2]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48L56-R130) [[3]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48L84-R144) [[4]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48R156-R172) * Updated the calculation of uncompressed serialized bytes to account for the new serialization format and potential streamvbyte compression (`be/src/vec/data_types/data_type_fixed_length_object.cpp`). * Added the `streamvbyte` library include to support the new encoding/decoding logic (`be/src/vec/data_types/data_type_fixed_length_object.cpp`). * Increased `BeExecVersionManager::max_be_exec_version` from 8 to 10, with detailed documentation and warnings about the sensitivity of this field. The new version enables the updated serialization logic (`be/src/agent/be_exec_version_manager.cpp`). * Defined a new constant `USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10` to clearly mark the threshold for the new serialization format (`be/src/agent/be_exec_version_manager.h`). * Marked `AggregateFunctionCount` and `AggregateFunctionCountNotNullUnary` as trivial by overriding the `is_trivial()` method to return `true`, which may allow for performance optimizations in the aggregation engine (`be/src/vec/aggregate_functions/aggregate_function_count.h`). [[1]](diffhunk://#diff-a5dbb09237f197bffdcbd3bec4fdd089913ec143d96806618c8eeb4c5dbb8cfeR64-R65) [[2]](diffhunk://#diff-a5dbb09237f197bffdcbd3bec4fdd089913ec143d96806618c8eeb4c5dbb8cfeR212-R213)

BiteTheDDDDt requested a review from zclllyybb as a code owner February 5, 2026 05:35

CopilotAI review requested due to automatic review settings February 5, 2026 05:35

Copilotstarted reviewing on behalf of BiteTheDDDDt February 5, 2026 05:35 View session

CopilotAI reviewed Feb 5, 2026
View reviewed changes

BiteTheDDDDt added 6 commits February 6, 2026 11:53

use streamvbyte_encode in DataTypeFixedLengthObject::serialize
b020e0b

update be exec version
873b871

update
6cd0b59

format
588e22d

fix ut
85f5dcb

fix
430d529

BiteTheDDDDt force-pushed the dev_0205_2 branch from 8224ee7 to 430d529Compare February 6, 2026 05:21

HappenLee approved these changes Feb 9, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 9, 2026

github-actionsBot added the reviewed label Feb 9, 2026

zclllyybb approved these changes Feb 9, 2026
View reviewed changes

zclllyybb merged commit d95d647 into apache:masterFeb 9, 2026
27 of 30 checks passed

BiteTheDDDDt added the dev/4.0.x label Feb 9, 2026

github-actionsBot added the dev/4.0.x-conflict label Feb 9, 2026

BiteTheDDDDt mentioned this pull request Mar 25, 2026
[Chore](pick) pick #58666 #60526 #60079 #60228 #60735 #61503
Merged

BiteTheDDDDt added dev/4.1.0-merged and removed dev/4.0.x dev/4.0.x-conflict labels Mar 25, 2026

	//Throw exception if mem_size is large than UINT32_MAX
	//cast_set will throw an exception if mem_size exceeds UINT32_MAX during conversion

Uh oh!

Conversation

BiteTheDDDDt commented Feb 5, 2026

What problem does this PR solve?

Serialization and Deserialization Improvements

Version Management

Aggregate Function Optimization

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 5, 2026

Uh oh!

BiteTheDDDDt commented Feb 5, 2026

Uh oh!

BiteTheDDDDt commented Feb 5, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

CopilotAIFeb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Feb 5, 2026

FE UT Coverage Report

Uh oh!

BiteTheDDDDt commented Feb 5, 2026

Uh oh!

hello-stephen commented Feb 5, 2026

FE UT Coverage Report

Uh oh!

BiteTheDDDDt commented Feb 6, 2026

Uh oh!

hello-stephen commented Feb 6, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Feb 6, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Feb 6, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Feb 6, 2026

BE Regression && UT Coverage Report

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 9, 2026

Uh oh!

github-actionsBot commented Feb 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants