[improvement](be) Persist rowset key bounds for non-MoW rowsets by liaoxin01 · Pull Request #62457 · apache/doris

liaoxin01 · 2026-04-13T15:49:14Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Persist rowset-level key bounds for non-MoW rowsets to reduce rowset meta memory usage and shrink the meta value stored in FoundationDB, avoiding value-size limit issues caused by full per-segment key bounds. The change keeps full segment key bounds for MoW rowsets and preserves the related compaction, snapshot, cloud conversion, and recycle flows.

Release note

None

Check List (For Author)

Test: Attempted BE unit test and targeted object compilation
- Unit Test / Manual test
Behavior changed: Yes (non-MoW rowsets can persist rowset-level key bounds instead of full per-segment key bounds; this branch temporarily defaults enable_rowset_key_bounds_for_non_mow to true for CI, while the path remains config-gated)
Does this need documentation: No

Thearas · 2026-04-13T15:49:22Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for persisting rowset-level key bounds for non-MoW rowsets and threads the new field through BE + cloud meta paths, with the “persist only rowset-level bounds (omit per-segment bounds)” behavior gated by a disabled-by-default BE config.

Changes:

Extend rowset meta protos with rowset_key_bounds and propagate it through cloud/doris PB conversion and snapshot creation.
Teach BE writers/compaction/index builder to persist/carry forward rowset_key_bounds when per-segment bounds are omitted.
Update truncation logic and unit tests around segment/rowset key-bounds truncation and the new config gate.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
gensrc/proto/olap_file.proto	Adds `rowset_key_bounds` to rowset meta protos.
cloud/src/meta-service/meta_service_job.cpp	Strips `rowset_key_bounds` when recycle-log stripping is enabled.
cloud/src/meta-service/meta_service.cpp	Accounts for `rowset_key_bounds` size when computing key-bounds bytes.
be/test/storage/segment/segments_key_bounds_truncation_test.cpp	Expands tests to cover config-gated rowset-level bounds and MoW behavior.
be/src/storage/task/index_builder.cpp	Copies either per-segment bounds or rowset-level bounds into output rowset meta.
be/src/storage/rowset/rowset_meta.h	Adds rowset-level bounds accessors + changes first/last key bound getters.
be/src/storage/rowset/rowset_meta.cpp	Adds rowset-level bounds building/truncation + merge behavior updates.
be/src/storage/rowset/beta_rowset_writer.h	Tracks optional rowset-level bounds during rowset writer aggregation.
be/src/storage/rowset/beta_rowset_writer.cpp	Persists rowset-level bounds based on config / availability.
be/src/storage/compaction/compaction.cpp	Propagates rowset-level bounds when segment bounds are absent.
be/src/common/config.h / be/src/common/config.cpp	Introduces `enable_rowset_key_bounds_for_non_mow` config (default false).
be/src/cloud/pb_convert.cpp	Copies/swaps `rowset_key_bounds` between doris/cloud PBs.
be/src/cloud/cloud_snapshot_mgr.cpp	Copies `rowset_key_bounds` when creating snapshot rowset meta.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-13T15:56:43Z

+ bool get_first_segment_key_bound(KeyBoundsPB* key_bounds) const {
+ if (_rowset_meta_pb.has_rowset_key_bounds()) {
+ *key_bounds = _rowset_meta_pb.rowset_key_bounds();
+ return true;
+ }


get_first_segment_key_bound / get_last_segment_key_bound now return the full rowset_key_bounds when present. That changes the meaning of the returned KeyBoundsPB: the “first segment bound” will carry the rowset max key (not the first segment’s max), and the “last segment bound” will carry the rowset min key (not the last segment’s min). If any caller relies on both min/max being the segment’s range (not just one side), this will yield incorrect results. Consider either (a) keeping these APIs strictly “segment” by only setting the relevant side (first: min + first-segment max if available; last: last-segment min + max), or (b) introducing a separate get_rowset_key_bounds() and leaving the segment APIs to consult segments_key_bounds only.

Copilot · 2026-04-13T15:56:44Z

+ bool get_last_segment_key_bound(KeyBoundsPB* key_bounds) const {
+ if (_rowset_meta_pb.has_rowset_key_bounds()) {
+ *key_bounds = _rowset_meta_pb.rowset_key_bounds();
+ return true;
+ }


get_first_segment_key_bound / get_last_segment_key_bound now return the full rowset_key_bounds when present. That changes the meaning of the returned KeyBoundsPB: the “first segment bound” will carry the rowset max key (not the first segment’s max), and the “last segment bound” will carry the rowset min key (not the last segment’s min). If any caller relies on both min/max being the segment’s range (not just one side), this will yield incorrect results. Consider either (a) keeping these APIs strictly “segment” by only setting the relevant side (first: min + first-segment max if available; last: last-segment min + max), or (b) introducing a separate get_rowset_key_bounds() and leaving the segment APIs to consult segments_key_bounds only.

Copilot · 2026-04-13T15:56:44Z

 void RowsetMeta::set_segments_key_bounds(const std::vector<KeyBoundsPB>& segments_key_bounds) {
+ if (!segments_key_bounds.empty()) {
+ *_rowset_meta_pb.mutable_rowset_key_bounds() = build_rowset_key_bounds(segments_key_bounds);
+ }


This unconditionally persists rowset_key_bounds whenever set_segments_key_bounds() is called, including when config::enable_rowset_key_bounds_for_non_mow is false. The PR description states “non-MoW rowsets can persist rowset-level key bounds only when enable_rowset_key_bounds_for_non_mow is set to true”. If the intent is truly “only persist rowset_key_bounds behind the gate”, then this needs to be guarded (or moved to the gated write path). If the intent is “only omit per-segment bounds behind the gate (but always persist rowset-level bounds when available)”, then the PR description (and any related comments) should be updated to match the actual behavior.

Copilot · 2026-04-13T15:56:44Z

+ std::string first_key;
+ bool has_first_key = rowset->first_key(&first_key);
+ std::string last_key;
+ bool has_last_key = rowset->last_key(&last_key);
+ DCHECK_EQ(has_first_key, has_last_key);
+ if (has_first_key) {
+ if (!_rowset_key_bounds.has_value()) {
+ _rowset_key_bounds.emplace();
+ _rowset_key_bounds->set_min_key(first_key);
+ }
+ _rowset_key_bounds->set_max_key(last_key);
+ }


rowset->first_key() / rowset->last_key() may involve extra work (depending on implementation) even though the metadata paths now carry rowset_key_bounds / segments_key_bounds. Since you only need rowset-level min/max here, consider reading from rowset->rowset_meta() (e.g., rowset_meta->rowset_key_bounds() when present, or get_first_segment_key_bound/get_last_segment_key_bound) to avoid any potential segment/index reads during add_rowset().

Issue Number: None Related PR: None Problem Summary: Persist rowset-level key bounds for non-MoW rowsets so active rowset metadata no longer depends on per-segment key bounds for non-MoW read paths, while keeping full segment key bounds for MoW rowsets and preserving compatibility across compaction, snapshot, cloud conversion, and recycle flows. None - Test: Attempted BE unit test and targeted object compilation - Unit Test / Manual test - Behavior changed: Yes (non-MoW rowsets now persist rowset-level key bounds instead of full per-segment key bounds; MoW remains unchanged) - Does this need documentation: No

liaoxin01 · 2026-04-13T16:01:35Z

run buildall

liaoxin01 · 2026-04-14T03:43:43Z

run buildall

hello-stephen · 2026-04-14T04:23:40Z

Cloud UT Coverage Report

Increment line coverage 60.00% (3/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.12% (32286/50352)
Region Coverage	65.05% (16243/24969)
Branch Coverage	55.58% (8682/15622)

hello-stephen · 2026-04-14T07:56:21Z

BE UT Coverage Report

Increment line coverage 88.65% (125/141) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.04% (20141/37975)
Line Coverage	36.60% (189518/517799)
Region Coverage	32.87% (147196/447821)
Branch Coverage	34.00% (64431/189491)

hello-stephen · 2026-04-14T08:01:44Z

BE Regression && UT Coverage Report

Increment line coverage 90.78% (128/141) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.85% (26721/37189)
Line Coverage	54.86% (283205/516215)
Region Coverage	51.97% (234892/451953)
Branch Coverage	53.35% (101404/190073)

liaoxin01 · 2026-04-19T15:42:31Z

rewrited by：#62604

liaoxin01 requested review from dataroaring, gavinchou, w41ter and yiguolei as code owners April 13, 2026 15:49

CopilotAI review requested due to automatic review settings April 13, 2026 15:49

liaoxin01 force-pushed the liaoxin/non-mow-rowset-key-bounds-compat branch from aaac837 to b91d64cCompare April 13, 2026 15:50

liaoxin01 changed the title ~~[improvement](be) Persist non-MoW rowset key bounds behind config~~[improvement](be) Persist rowset key bounds for non-MoW rowsetsApr 13, 2026

Copilotstarted reviewing on behalf of liaoxin01 April 13, 2026 15:56 View session

CopilotAI reviewed Apr 13, 2026
View reviewed changes

liaoxin01 force-pushed the liaoxin/non-mow-rowset-key-bounds-compat branch from b91d64c to 6c101beCompare April 13, 2026 15:59

fix
715e28d

liaoxin01 closed this Apr 19, 2026

Navigation Menu

Uh oh!

[improvement](be) Persist rowset key bounds for non-MoW rowsets - #62457

[improvement](be) Persist rowset key bounds for non-MoW rowsets#62457
liaoxin01 wants to merge 2 commits into
apache:masterfrom
liaoxin01:liaoxin/non-mow-rowset-key-bounds-compat

liaoxin01 commented Apr 13, 2026•
edited
Loading

Uh oh!

Thearas commented Apr 13, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

CopilotAIApr 13, 2026

Uh oh!

CopilotAIApr 13, 2026

Uh oh!

CopilotAIApr 13, 2026

Uh oh!

CopilotAIApr 13, 2026

Uh oh!

liaoxin01 commented Apr 13, 2026

Uh oh!

liaoxin01 commented Apr 14, 2026

Uh oh!

hello-stephen commented Apr 14, 2026

Uh oh!

hello-stephen commented Apr 14, 2026

Uh oh!

hello-stephen commented Apr 14, 2026

Uh oh!

liaoxin01 commented Apr 19, 2026•
edited
Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

liaoxin01 commented Apr 13, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

Thearas commented Apr 13, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

CopilotAIApr 13, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIApr 13, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIApr 13, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIApr 13, 2026

Choose a reason for hiding this comment

Uh oh!

liaoxin01 commented Apr 13, 2026

Uh oh!

liaoxin01 commented Apr 14, 2026

Uh oh!

hello-stephen commented Apr 14, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 14, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 14, 2026

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 commented Apr 19, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liaoxin01 commented Apr 13, 2026•
edited
Loading

liaoxin01 commented Apr 19, 2026•
edited
Loading