[improvement](compaction) Disable MOW version-count promotion by default by Yukang-Lian · Pull Request #67178 · apache/doris

Yukang-Lian · 2026-08-26T14:16:54Z

What problem does this PR solve?

Issue Number: None

Related PR: #49383

Problem Summary:

Size-based cumulative compaction currently promotes a merge-on-write output rowset to the base layer once its version span exceeds compaction_promotion_version_count, even when the output rowset is only a few hundred KB. For high-frequency tiny loads, this moves small rowsets into the base layer prematurely and causes frequent base compactions.

PR #49383 makes cumulative compaction aggregate historical delete bitmaps into the output rowset version and remove the pre-rowset delete bitmap versions. With that optimization enabled, raw rowset version span no longer needs to trigger promotion by default.

This PR changes the default compaction_promotion_version_count from 1,000 to the maximum int64 value. This effectively disables version-count promotion by default in both local and cloud modes, while retaining the existing logic and configuration as an operational fallback. Size-based and delete-version promotion remain unchanged.

Release note

MOW cumulative compaction no longer promotes output rowsets by version count under the default configuration. The existing configuration remains available as a fallback.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Focused ASAN BE UT: 68/68 passed for TestSizeBasedCumulativeCompactionPolicy.* and TestCloudSizeBasedCumulativeCompactionPolicy.*.
Behavior changed:
- No.
- Yes. By default, small MOW cumulative output rowsets remain in the cumulative layer until the normal size or delete-version promotion condition is met. Operators can lower compaction_promotion_version_count to restore the previous safeguard.
Does this need documentation?
- No.
- Yes.

QA validation

Please validate both local and cloud modes:

With the default configuration, create an MOW table and generate more than 1,000 tiny versions, then run cumulative compaction. When the output remains below the size threshold and has no delete version, verify that the cumulative point is not promoted solely by version span and base compaction is not triggered repeatedly.
Set compaction_promotion_version_count to a small value such as 5 and verify that version-count promotion can still be enabled. For local mode, configure/restart the BE before testing an existing tablet; cloud mode reads the value on each evaluation.
Verify that delete bitmap aggregation and removal introduced by [fix](mow) reduce mow delete bitmap count: agg and remove delete bitmap of pre rowsets when cu compaction #49383 keeps historical delete bitmap count bounded and query results correct.
Verify that reaching the size threshold or compacting a delete version still promotes the cumulative point.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: None Related PR: apache#49383 Problem Summary: Size-based cumulative compaction promotes a merge-on-write output rowset to the base layer once its version span exceeds compaction_promotion_version_count, even when the rowset is only a few hundred KB. This causes high-frequency tiny loads to enter base compaction prematurely and trigger unnecessary base compactions. PR apache#49383 makes cumulative compaction aggregate and remove historical delete bitmap versions, so raw version span is no longer needed as a proxy for delete bitmap pressure. Remove the version-count promotion from local and cloud policies and remove its obsolete configuration. Size-based and delete-version promotion remain unchanged. ### Release note MOW cumulative compaction no longer promotes small output rowsets based only on version count. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=TestSizeBasedCumulativeCompactionPolicy.*:TestCloudSizeBasedCumulativeCompactionPolicy.* -j80 - build-support/check-format.sh - build-support/check-build-hygiene.sh - Behavior changed: Yes. Small MOW cumulative output rowsets remain in the cumulative layer until normal size or delete-version promotion applies. - Does this need documentation: No

hello-stephen · 2026-08-26T14:17:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Yukang-Lian · 2026-08-27T02:56:01Z

run buildall

hello-stephen · 2026-08-27T03:54:32Z

TPC-H: Total hot run time: 17234 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0bc9c745456541d4f612edd67d4dd958980e34e9, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17615	3109	3071	3071
q2	2107	269	226	226
q3	10219	918	516	516
q4	4687	255	205	205
q5	7670	578	385	385
q6	140	117	98	98
q7	528	506	394	394
q8	9236	877	960	877
q9	3456	2403	2396	2396
q10	6516	895	750	750
q11	395	192	183	183
q12	612	259	198	198
q13	18174	1578	1207	1207
q14	162	155	141	141
q15	q16	457	404	377	377
q17	1422	936	836	836
q18	3145	2365	2341	2341
q19	1269	906	766	766
q20	375	291	210	210
q21	5576	1830	1862	1830
q22	341	267	227	227
Total cold run time: 94102 ms
Total hot run time: 17234 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3487	3431	3401	3401
q2	516	401	364	364
q3	2211	2347	2139	2139
q4	1201	1187	888	888
q5	2193	2153	2113	2113
q6	176	129	89	89
q7	1023	972	817	817
q8	1608	1424	1413	1413
q9	3186	3090	3091	3090
q10	1891	1835	1608	1608
q11	358	268	251	251
q12	452	431	346	346
q13	1518	1536	1156	1156
q14	165	172	159	159
q15	q16	390	406	360	360
q17	3622	3254	3256	3254
q18	4936	4463	5037	4463
q19	960	882	866	866
q20	1014	994	840	840
q21	3805	3092	3258	3092
q22	398	353	325	325
Total cold run time: 35110 ms
Total hot run time: 31034 ms

hello-stephen · 2026-08-27T04:01:01Z

TPC-DS: Total hot run time: 81438 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0bc9c745456541d4f612edd67d4dd958980e34e9, data reload: false
query5	4291	428	344	344
query6	386	147	127	127
query7	4906	416	232	232
query8	302	123	119	119
query9	8672	2867	2865	2865
query10	407	221	181	181
query11	5381	1025	910	910
query12	113	74	77	74
query13	1193	443	323	323
query14	6123	2185	2034	2034
query14_1	1954	1948	1952	1948
query15	169	117	111	111
query16	898	375	336	336
query17	796	463	394	394
query18	2320	318	231	231
query19	152	138	107	107
query20	72	66	66	66
query21	196	99	84	84
query22	5386	5366	5231	5231
query23	6687	6241	6130	6130
query23_1	6031	6090	6161	6090
query24	7249	1065	780	780
query24_1	786	785	778	778
query25	438	292	244	244
query26	1236	239	128	128
query27	2776	416	245	245
query28	4712	1476	1460	1460
query29	916	450	338	338
query30	251	159	133	133
query31	825	390	327	327
query32	125	71	71	71
query33	451	206	175	175
query34	998	824	473	473
query35	418	400	343	343
query36	558	549	519	519
query37	117	77	71	71
query38	996	838	804	804
query39	508	478	478	478
query39_1	483	478	469	469
query40	208	89	74	74
query41	54	50	51	50
query42	78	72	74	72
query43	240	242	214	214
query44	1015	535	556	535
query45	116	104	97	97
query46	793	834	517	517
query47	780	764	727	727
query48	304	298	217	217
query49	529	232	175	175
query50	770	268	196	196
query51	8017	7888	8009	7888
query52	72	70	66	66
query53	210	211	151	151
query54	245	184	164	164
query55	86	59	56	56
query56	205	187	190	187
query57	686	646	717	646
query58	209	168	176	168
query59	1252	1250	1094	1094
query60	242	197	174	174
query61	133	128	135	128
query62	367	207	189	189
query63	172	141	144	141
query64	2847	817	558	558
query65	1636	1587	1638	1587
query66	1871	273	201	201
query67	10043	9697	9801	9697
query68	3009	1140	726	726
query69	343	221	192	192
query70	656	609	608	608
query71	240	176	160	160
query72	2360	1809	1171	1171
query73	618	596	332	332
query74	1984	1206	1140	1140
query75	1173	1097	956	956
query76	2351	739	532	532
query77	251	263	216	216
query78	4056	3623	3232	3232
query79	2763	835	582	582
query80	1585	336	278	278
query81	516	160	132	132
query82	633	133	99	99
query83	297	215	193	193
query84	301	116	91	91
query85	813	365	299	299
query86	474	180	176	176
query87	1043	985	894	894
query88	2844	2102	2114	2102
query89	293	198	174	174
query90	2014	131	127	127
query91	136	124	101	101
query92	86	66	71	66
query93	1473	1137	747	747
query94	643	249	221	221
query95	517	252	291	252
query96	840	592	263	263
query97	1087	1087	991	991
query98	159	137	131	131
query99	417	349	304	304
Total cold run time: 178863 ms
Total hot run time: 81438 ms

hello-stephen · 2026-08-27T04:05:23Z

ClickBench: Total hot run time: 14.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0bc9c745456541d4f612edd67d4dd958980e34e9, data reload: false
query1	0.00	0.01	0.00
query2	0.08	0.04	0.04
query3	0.24	0.11	0.11
query4	1.60	0.09	0.10
query5	0.19	0.15	0.16
query6	1.25	0.71	0.68
query7	0.03	0.00	0.00
query8	0.05	0.03	0.03
query9	0.28	0.21	0.22
query10	0.34	0.35	0.34
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.31	0.30	0.31
query14	0.47	0.46	0.45
query15	0.36	0.34	0.34
query16	0.23	0.21	0.22
query17	0.70	0.68	0.71
query18	0.19	0.17	0.16
query19	1.18	1.15	1.17
query20	0.01	0.01	0.01
query21	15.45	0.15	0.12
query22	5.08	0.04	0.04
query23	16.22	0.25	0.11
query24	3.01	0.32	0.27
query25	0.13	0.03	0.04
query26	0.83	0.16	0.13
query27	0.04	0.03	0.03
query28	3.66	0.52	0.26
query29	12.52	3.16	2.55
query30	0.25	0.12	0.12
query31	2.75	0.36	0.17
query32	3.53	0.31	0.23
query33	1.52	1.38	1.39
query34	15.39	2.19	1.77
query35	1.76	1.72	1.75
query36	0.47	0.29	0.29
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.08
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 90.84 s
Total hot run time: 14.63 s

hello-stephen · 2026-08-27T06:04:48Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.65% (29291/46757)
Line Coverage	47.62% (306283/643117)
Region Coverage	43.28% (247425/571634)
Branch Coverage	44.84% (115152/256828)

Yukang-Lian · 2026-08-27T07:01:15Z

run cloud_p0

Yukang-Lian · 2026-08-27T07:01:22Z

run vault_p0

hello-stephen · 2026-08-27T08:15:57Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.11% (34467/45285)
Line Coverage	61.02% (387957/635755)
Region Coverage	57.13% (325302/569451)
Branch Coverage	57.98% (148475/256095)

### What problem does this PR solve? Issue Number: None Related PR: apache#49383 Problem Summary: The MOW version-count promotion protects against excessive delete bitmap metadata, but its default threshold of 1000 can promote tiny cumulative output rowsets and trigger frequent base compactions. Retain the local and cloud promotion logic as an operational fallback, while changing the default threshold to the maximum int64 value so version count alone does not promote rowsets by default. Operators can lower compaction_promotion_version_count if the fallback is needed. ### Release note MOW cumulative compaction no longer promotes output rowsets by version count under the default configuration. The existing configuration remains available as a fallback. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=TestSizeBasedCumulativeCompactionPolicy.*:TestCloudSizeBasedCumulativeCompactionPolicy.* -j80 - build-support/check-format.sh - build-support/check-build-hygiene.sh - Behavior changed: Yes. Version-count promotion is disabled by default but remains configurable. - Does this need documentation: No

Yukang-Lian · 2026-09-01T11:30:08Z

run buildall

hello-stephen · 2026-09-01T18:56:35Z

TPC-H: Total hot run time: 16753 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3cb81f90bd57e2b4d41f21f29a0be5378f03c014, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17553	3132	3126	3126
q2	2105	290	213	213
q3	10216	901	518	518
q4	4673	253	200	200
q5	7681	561	392	392
q6	136	110	93	93
q7	523	511	389	389
q8	9238	970	852	852
q9	3423	2389	2381	2381
q10	6520	847	715	715
q11	403	200	184	184
q12	624	259	201	201
q13	18114	1538	1149	1149
q14	157	146	141	141
q15	q16	425	399	370	370
q17	1360	890	760	760
q18	3037	2239	2247	2239
q19	1133	841	778	778
q20	364	285	199	199
q21	4829	1622	1900	1622
q22	323	270	231	231
Total cold run time: 92837 ms
Total hot run time: 16753 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3483	3416	3440	3416
q2	490	435	397	397
q3	2177	2264	2180	2180
q4	1195	1170	906	906
q5	2185	2136	2099	2099
q6	168	121	85	85
q7	1047	971	906	906
q8	1617	1425	1425	1425
q9	3144	3105	3103	3103
q10	1836	1778	1630	1630
q11	352	270	257	257
q12	455	434	348	348
q13	1485	1549	1164	1164
q14	169	169	174	169
q15	q16	403	401	366	366
q17	3615	3337	3317	3317
q18	4864	4434	4752	4434
q19	848	822	929	822
q20	1025	990	817	817
q21	3937	3134	3306	3134
q22	398	343	326	326
Total cold run time: 34893 ms
Total hot run time: 31301 ms

hello-stephen · 2026-09-01T19:03:05Z

TPC-DS: Total hot run time: 82422 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3cb81f90bd57e2b4d41f21f29a0be5378f03c014, data reload: false
query5	4319	424	347	347
query6	376	143	135	135
query7	4922	431	228	228
query8	314	125	136	125
query9	8702	2965	2943	2943
query10	391	220	193	193
query11	5379	1035	898	898
query12	118	72	71	71
query13	1207	475	321	321
query14	5992	2224	2133	2133
query14_1	1995	1987	1991	1987
query15	180	117	112	112
query16	931	397	375	375
query17	826	457	394	394
query18	2340	337	252	252
query19	185	155	129	129
query20	75	71	72	71
query21	204	103	89	89
query22	5387	5304	5271	5271
query23	6684	6268	6075	6075
query23_1	6028	6202	6127	6127
query24	7297	1102	778	778
query24_1	801	812	789	789
query25	437	322	265	265
query26	1235	239	129	129
query27	2789	430	258	258
query28	4678	1491	1521	1491
query29	959	445	361	361
query30	253	162	134	134
query31	846	410	331	331
query32	135	84	78	78
query33	474	235	182	182
query34	994	842	494	494
query35	428	406	352	352
query36	578	562	531	531
query37	122	86	76	76
query38	1004	865	815	815
query39	509	486	473	473
query39_1	459	481	486	481
query40	207	88	76	76
query41	56	52	50	50
query42	72	74	74	74
query43	242	243	210	210
query44	1018	542	547	542
query45	108	102	97	97
query46	814	824	528	528
query47	764	749	696	696
query48	302	300	236	236
query49	544	240	178	178
query50	759	259	197	197
query51	7874	8025	7953	7953
query52	67	68	65	65
query53	194	196	144	144
query54	253	188	175	175
query55	77	60	59	59
query56	203	176	158	158
query57	796	645	639	639
query58	211	162	167	162
query59	1222	1211	1107	1107
query60	221	187	183	183
query61	132	135	172	135
query62	367	205	173	173
query63	177	142	144	142
query64	2717	711	566	566
query65	1621	1712	1567	1567
query66	1799	267	203	203
query67	9878	9776	9574	9574
query68	2765	1221	731	731
query69	343	224	202	202
query70	682	612	613	612
query71	247	183	162	162
query72	2334	1794	1600	1600
query73	684	578	336	336
query74	1580	1235	1152	1152
query75	1201	1112	972	972
query76	2288	732	531	531
query77	256	261	215	215
query78	4055	3724	3272	3272
query79	2671	798	570	570
query80	1622	381	265	265
query81	492	154	132	132
query82	633	123	96	96
query83	291	204	193	193
query84	296	113	92	92
query85	800	365	305	305
query86	391	172	173	172
query87	993	973	894	894
query88	2771	2136	2107	2107
query89	282	197	179	179
query90	1975	130	132	130
query91	133	133	104	104
query92	80	72	72	72
query93	1683	1059	748	748
query94	637	261	226	226
query95	518	316	233	233
query96	855	610	280	280
query97	1097	1077	1006	1006
query98	167	136	133	133
query99	433	342	313	313
Total cold run time: 177949 ms
Total hot run time: 82422 ms

hello-stephen · 2026-09-01T19:07:28Z

ClickBench: Total hot run time: 14.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3cb81f90bd57e2b4d41f21f29a0be5378f03c014, data reload: false
query1	0.00	0.00	0.00
query2	0.08	0.04	0.04
query3	0.25	0.10	0.10
query4	1.60	0.10	0.10
query5	0.18	0.15	0.15
query6	1.25	0.72	0.67
query7	0.04	0.01	0.00
query8	0.04	0.04	0.03
query9	0.31	0.22	0.23
query10	0.36	0.37	0.35
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.32	0.32	0.31
query14	0.45	0.45	0.45
query15	0.35	0.35	0.36
query16	0.24	0.22	0.25
query17	0.70	0.73	0.67
query18	0.19	0.18	0.17
query19	1.21	1.09	1.07
query20	0.02	0.01	0.01
query21	15.46	0.15	0.12
query22	5.08	0.04	0.04
query23	16.17	0.27	0.12
query24	3.10	0.31	0.26
query25	0.12	0.04	0.04
query26	0.74	0.16	0.13
query27	0.04	0.03	0.04
query28	3.60	0.59	0.26
query29	12.43	3.17	2.56
query30	0.26	0.11	0.12
query31	2.76	0.37	0.17
query32	3.55	0.32	0.22
query33	1.36	1.41	1.50
query34	15.39	2.19	1.78
query35	1.74	1.73	1.73
query36	0.48	0.30	0.29
query37	0.06	0.03	0.04
query38	0.05	0.03	0.02
query39	0.03	0.03	0.03
query40	0.12	0.08	0.07
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.58 s
Total hot run time: 14.63 s

Yukang-Lian marked this pull request as ready for review August 26, 2026 16:17

Yukang-Lian requested review from gavinchou and luwei16 as code owners August 26, 2026 16:17

Yukang-Lian changed the title ~~[improvement](compaction) Remove MOW version-count promotion~~[improvement](compaction) Disable MOW version-count promotion by defaultSep 1, 2026

Navigation Menu

Uh oh!

[improvement](compaction) Disable MOW version-count promotion by default - #67178

[improvement](compaction) Disable MOW version-count promotion by default#67178
Yukang-Lian wants to merge 2 commits into
apache:masterfrom
Yukang-Lian:codex/remove-mow-cumu-version-promotion

Yukang-Lian commented Aug 26, 2026•
edited
Loading

Uh oh!

hello-stephen commented Aug 26, 2026

Uh oh!

Yukang-Lian commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

Yukang-Lian commented Aug 27, 2026

Uh oh!

Yukang-Lian commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

Yukang-Lian commented Sep 1, 2026

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Yukang-Lian commented Aug 26, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

QA validation

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 26, 2026

Uh oh!

Yukang-Lian commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

BE UT Coverage Report

Uh oh!

Yukang-Lian commented Aug 27, 2026

Uh oh!

Yukang-Lian commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

BE Regression && UT Coverage Report

Uh oh!

Yukang-Lian commented Sep 1, 2026

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Yukang-Lian commented Aug 26, 2026•
edited
Loading