branch-4.0: [improvement](rowset) Aggregate non-MOW segment key bounds by liaoxin01 · Pull Request #64305 · apache/doris

liaoxin01 · 2026-06-09T09:51:30Z

Pick #62604\n\nNote: keep enable_aggregate_non_mow_key_bounds disabled by default for upgrade/downgrade safety.

### What problem does this PR solve? Issue Number: None Related PR: apache#62604 Problem Summary: Non-MOW duplicate and aggregate rowsets do not consume per-segment key bounds on the read path, but persisting all per-segment bounds can make cloud rowset metadata too large for FDB values. This change adds an aggregated key-bounds layout for non-MOW rowsets, preserves the layout flag through rowset meta conversion, snapshot restore, compaction, and index rebuild paths, and keeps MOW rowsets on per-segment bounds for delete bitmap lookup correctness. The feature is controlled by enable_aggregate_non_mow_key_bounds and is disabled by default for upgrade and downgrade safety. ### Release note Add an opt-in BE config enable_aggregate_non_mow_key_bounds to aggregate non-MOW segment key bounds and reduce rowset metadata size. ### Check List (For Author) - Test: Static Check - Static check: git diff --check - Behavior changed: No. The new aggregation behavior is disabled by default and only takes effect when enable_aggregate_non_mow_key_bounds is set to true - Does this need documentation: No

hello-stephen · 2026-06-09T09:51:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

liaoxin01 · 2026-06-09T09:59:43Z

run buildall

Copilot

Pull request overview

This PR introduces an opt-in mechanism to aggregate non-MOW rowset per-segment key bounds into a single [rowset_min, rowset_max] entry, reducing rowset metadata size (notably for cloud/FDB), while ensuring MOW rowsets continue to preserve per-segment bounds for correctness. It adds a new proto field to persist the layout choice, wires the behavior through rowset writing/compaction/snapshot/index-rewrite paths, and adds regression + unit tests.

Changes:

Add segments_key_bounds_aggregated to rowset meta protos and propagate it in cloud PB conversion/copy paths.
Implement aggregated key-bounds storage in RowsetMeta and gate it behind enable_aggregate_non_mow_key_bounds for non-MOW writers (plus preserve/propagate the layout through index-rewrite and snapshot restore).
Add regression and BE unit tests validating the aggregated behavior and guarding against MOW regressions.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
regression-test/suites/data_model_p0/duplicate/test_non_mow_key_bounds_aggregation.groovy	Regression coverage for aggregated non-MOW bounds, MOW non-aggregation, config on/off behavior, and index-rewrite preservation.
gensrc/proto/olap_file.proto	Adds `segments_key_bounds_aggregated` field to rowset meta protos.
be/test/olap/segments_key_bounds_truncation_test.cpp	Forces aggregation feature off to keep truncation test expectations stable.
be/test/olap/rowset/rowset_meta_test.cpp	Unit tests for aggregation behavior, flag semantics, and truncation interaction.
be/src/olap/task/index_builder.cpp	Preserves aggregated layout/flag when rebuilding rowset meta during index rewrite.
be/src/olap/snapshot_manager.h	Extends `_rename_rowset_id` signature to pass MOW flag into rowset writer context.
be/src/olap/snapshot_manager.cpp	Propagates MOW flag so aggregation isn’t incorrectly applied when restoring MOW tablets’ rowsets.
be/src/olap/rowset/rowset.h	Exposes `is_segments_key_bounds_aggregated()` on `Rowset`.
be/src/olap/rowset/rowset_meta.h	Adds aggregated-flag accessors and extends `set_segments_key_bounds` API.
be/src/olap/rowset/rowset_meta.cpp	Implements aggregation into a single min/max entry and asserts aggregation isn’t used in MOW merge path.
be/src/olap/rowset/beta_rowset_writer.cpp	Applies aggregation for non-MOW rowset meta build (config-gated) and preserves source layout when cloning meta.
be/src/olap/compaction.cpp	Ensures output layout remains aggregated if any input is aggregated; otherwise follows config for non-MOW.
be/src/olap/base_tablet.cpp	Adds runtime guard to prevent MOW lookup from using aggregated/inconsistent bounds.
be/src/common/config.h	Declares `enable_aggregate_non_mow_key_bounds`.
be/src/common/config.cpp	Defines `enable_aggregate_non_mow_key_bounds` (default false).
be/src/cloud/pb_convert.cpp	Copies truncated/aggregated flags only when present to avoid forcing default fields into cloud PB.
be/src/cloud/cloud_snapshot_mgr.cpp	Preserves truncated/aggregated flags when creating rowset meta for cloud snapshots.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

hello-stephen · 2026-06-09T11:59:18Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-09T13:48:32Z

BE UT Coverage Report

Increment line coverage 79.38% (77/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.29% (19439/36475)
Line Coverage	36.44% (181744/498702)
Region Coverage	33.06% (141280/427375)
Branch Coverage	33.95% (61179/180221)

hello-stephen · 2026-06-09T15:08:54Z

BE Regression && UT Coverage Report

Increment line coverage 80.41% (78/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.10% (25031/35707)
Line Coverage	52.70% (262370/497818)
Region Coverage	50.39% (217513/431700)
Branch Coverage	51.60% (93321/180854)

liaoxin01 · 2026-06-17T07:09:50Z

run buildall

hello-stephen · 2026-06-17T08:20:47Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-17T10:57:46Z

BE Regression && UT Coverage Report

Increment line coverage 80.41% (78/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.50% (25533/35712)
Line Coverage	54.37% (270772/498044)
Region Coverage	52.08% (224954/431903)
Branch Coverage	53.47% (96784/181023)

liaoxin01 · 2026-06-23T15:00:49Z

run buildall

hello-stephen · 2026-06-23T16:11:21Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T17:54:44Z

BE UT Coverage Report

Increment line coverage 79.38% (77/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.35% (19464/36486)
Line Coverage	36.48% (182036/499015)
Region Coverage	33.08% (141460/427666)
Branch Coverage	33.94% (61226/180420)

hello-stephen · 2026-06-23T18:05:00Z

BE Regression && UT Coverage Report

Increment line coverage 80.41% (78/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.50% (25538/35717)
Line Coverage	54.36% (270800/498141)
Region Coverage	52.07% (224940/431984)
Branch Coverage	53.45% (96777/181053)

hello-stephen · 2026-06-23T18:45:44Z

BE Regression && UT Coverage Report

Increment line coverage 80.41% (78/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.50% (25538/35717)
Line Coverage	54.36% (270800/498141)
Region Coverage	52.07% (224940/431984)
Branch Coverage	53.45% (96777/181053)

hello-stephen · 2026-06-23T18:48:52Z

BE Regression && UT Coverage Report

Increment line coverage 80.41% (78/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.50% (25538/35717)
Line Coverage	54.36% (270771/498141)
Region Coverage	52.06% (224912/431984)
Branch Coverage	53.44% (96761/181053)

hello-stephen · 2026-06-23T19:05:37Z

BE Regression && UT Coverage Report

Increment line coverage 80.41% (78/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.50% (25538/35717)
Line Coverage	54.37% (270833/498141)
Region Coverage	52.08% (224982/431984)
Branch Coverage	53.47% (96805/181053)

hello-stephen · 2026-06-23T19:08:43Z

BE Regression && UT Coverage Report

Increment line coverage 80.41% (78/97) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.50% (25538/35717)
Line Coverage	54.37% (270833/498141)
Region Coverage	52.08% (224982/431984)
Branch Coverage	53.47% (96805/181053)

CopilotAI review requested due to automatic review settings June 9, 2026 09:51

liaoxin01 requested a review from morningman as a code owner June 9, 2026 09:51

Copilotstarted reviewing on behalf of liaoxin01 June 9, 2026 09:51 View session

liaoxin01 mentioned this pull request Jun 9, 2026
branch-4.0: [opt](rowset) Aggregate non-MOW segment key bounds to reduce rowset meta size #62604 #62799
Closed

CopilotAI reviewed Jun 9, 2026
View reviewed changes

Comment threadbe/src/olap/rowset/rowset_meta.h

morningman approved these changes Jun 24, 2026
View reviewed changes

morningman merged commit 6e4fd99 into apache:branch-4.0Jun 24, 2026
29 of 34 checks passed

morningman mentioned this pull request Jul 9, 2026
4.0.7 Release Notes #65399
Closed

Uh oh!

Conversation

liaoxin01 commented Jun 9, 2026

Uh oh!

hello-stephen commented Jun 9, 2026

Uh oh!

liaoxin01 commented Jun 9, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

hello-stephen commented Jun 9, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 9, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 9, 2026

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 commented Jun 17, 2026

Uh oh!

hello-stephen commented Jun 17, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 17, 2026

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 commented Jun 23, 2026

Uh oh!

hello-stephen commented Jun 23, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants