[enhance](compaction) optimize mow base compaction parameters by luwei16 · Pull Request #52321 · apache/doris

luwei16 · 2025-06-25T13:07:38Z

1 Increase base compaction frequency

In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes.

2. Increase the limit on compaction score per run

The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.

Thearas · 2025-06-25T13:07:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

luwei16 · 2025-06-25T13:07:51Z

run buildall

luwei16 · 2025-06-26T04:04:46Z

run beut

…-mow-compact-new

luwei16 · 2025-06-26T14:43:01Z

run buildall

github-actions · 2025-06-26T17:07:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-06-26T17:07:46Z

PR approved by anyone and no changes requested.

luwei16 · 2025-06-29T10:52:05Z

run buildall

luwei16 · 2025-06-29T11:08:51Z

run buildall

doris-robot · 2025-06-29T12:42:34Z

TPC-H: Total hot run time: 33801 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 199db7c5eb2f75ab1c61b2c78b68ea0868c614f2, data reload: false
------ Round 1 ----------------------------------
q1	17586	5260	5045	5045
q2	1936	278	187	187
q3	10356	1315	728	728
q4	10229	1030	516	516
q5	7659	2463	2330	2330
q6	183	163	130	130
q7	896	750	592	592
q8	9303	1272	1143	1143
q9	6967	5137	5021	5021
q10	6882	2383	1965	1965
q11	492	287	275	275
q12	341	359	212	212
q13	17755	3675	3104	3104
q14	231	228	204	204
q15	547	463	483	463
q16	424	425	379	379
q17	593	861	367	367
q18	7681	7181	7053	7053
q19	1256	967	538	538
q20	327	369	216	216
q21	3659	2546	2355	2355
q22	1046	1012	978	978
Total cold run time: 106349 ms
Total hot run time: 33801 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5129	5022	5103	5022
q2	247	318	217	217
q3	2210	2651	2272	2272
q4	1368	1794	1330	1330
q5	4209	4290	4395	4290
q6	221	171	131	131
q7	2030	1945	1796	1796
q8	2612	2525	2509	2509
q9	7130	7109	7237	7109
q10	3083	3214	2816	2816
q11	578	519	484	484
q12	660	782	606	606
q13	3560	3980	3380	3380
q14	277	306	289	289
q15	512	486	468	468
q16	448	483	421	421
q17	1121	1547	1341	1341
q18	7333	7191	7200	7191
q19	756	896	1042	896
q20	1946	1986	1860	1860
q21	4763	4312	4369	4312
q22	1101	1028	999	999
Total cold run time: 51294 ms
Total hot run time: 49739 ms

doris-robot · 2025-06-29T12:54:07Z

TPC-DS: Total hot run time: 184434 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 199db7c5eb2f75ab1c61b2c78b68ea0868c614f2, data reload: false
query1	997	399	382	382
query2	6540	1646	1683	1646
query3	6743	221	214	214
query4	26358	23723	22895	22895
query5	4376	576	448	448
query6	300	213	205	205
query7	4633	494	296	296
query8	287	234	204	204
query9	8625	2648	2640	2640
query10	448	319	266	266
query11	15509	15063	15312	15063
query12	162	111	99	99
query13	1644	504	397	397
query14	9383	5602	5576	5576
query15	206	186	169	169
query16	7489	619	452	452
query17	1167	700	566	566
query18	1991	390	289	289
query19	187	190	158	158
query20	118	117	112	112
query21	208	119	113	113
query22	4143	4165	3929	3929
query23	33871	32971	33148	32971
query24	8523	2378	2392	2378
query25	558	508	426	426
query26	1248	267	153	153
query27	2745	522	348	348
query28	4354	2155	2121	2121
query29	750	595	453	453
query30	280	219	190	190
query31	939	842	741	741
query32	75	69	63	63
query33	571	379	334	334
query34	801	837	526	526
query35	771	882	744	744
query36	930	978	868	868
query37	108	100	77	77
query38	4101	4110	4026	4026
query39	1493	1442	1432	1432
query40	205	117	103	103
query41	55	57	52	52
query42	123	105	106	105
query43	479	504	480	480
query44	1298	816	817	816
query45	174	168	163	163
query46	852	1006	625	625
query47	1737	1810	1737	1737
query48	389	420	299	299
query49	734	527	376	376
query50	625	692	409	409
query51	4100	4131	4061	4061
query52	108	105	106	105
query53	223	247	178	178
query54	587	566	514	514
query55	82	75	80	75
query56	295	287	300	287
query57	1199	1189	1108	1108
query58	264	258	265	258
query59	2542	2702	2570	2570
query60	316	314	311	311
query61	119	123	122	122
query62	805	711	641	641
query63	217	211	191	191
query64	4233	981	634	634
query65	4269	4207	4170	4170
query66	1085	404	310	310
query67	15692	15745	15270	15270
query68	7972	901	540	540
query69	518	304	267	267
query70	1130	1111	1108	1108
query71	462	314	296	296
query72	5541	4816	4813	4813
query73	732	647	353	353
query74	9260	9223	9097	9097
query75	3800	3150	2699	2699
query76	3594	1142	708	708
query77	795	370	285	285
query78	10164	10321	9278	9278
query79	2185	835	584	584
query80	597	494	429	429
query81	486	261	222	222
query82	446	126	95	95
query83	250	252	232	232
query84	246	98	95	95
query85	851	346	302	302
query86	380	285	303	285
query87	4479	4409	4382	4382
query88	3493	2250	2312	2250
query89	382	311	286	286
query90	1859	197	200	197
query91	138	140	112	112
query92	73	58	57	57
query93	1630	914	590	590
query94	662	416	296	296
query95	374	285	284	284
query96	479	573	276	276
query97	2706	2743	2619	2619
query98	228	202	202	202
query99	1458	1400	1255	1255
Total cold run time: 273809 ms
Total hot run time: 184434 ms

doris-robot · 2025-06-29T12:59:24Z

ClickBench: Total hot run time: 30 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 199db7c5eb2f75ab1c61b2c78b68ea0868c614f2, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.08	0.08
query4	1.62	0.11	0.11
query5	0.43	0.42	0.42
query6	1.15	0.66	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.62	0.50	0.52
query10	0.57	0.56	0.57
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.63	0.61
query14	0.79	0.79	0.80
query15	0.88	0.86	0.87
query16	0.39	0.38	0.40
query17	1.06	1.09	1.08
query18	0.23	0.21	0.21
query19	1.91	1.88	1.82
query20	0.01	0.01	0.01
query21	15.42	0.89	0.55
query22	0.76	1.27	0.78
query23	14.74	1.39	0.65
query24	6.95	1.62	1.10
query25	0.50	0.27	0.19
query26	0.59	0.16	0.13
query27	0.07	0.05	0.05
query28	9.89	0.89	0.44
query29	12.53	3.98	3.36
query30	0.26	0.10	0.06
query31	2.86	0.58	0.38
query32	3.24	0.54	0.47
query33	3.14	3.04	3.19
query34	16.11	5.36	4.73
query35	4.82	4.84	4.81
query36	0.67	0.50	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.05 s
Total hot run time: 30 s

hello-stephen · 2025-06-29T14:28:51Z

BE UT Coverage Report

Increment line coverage 50.00% (5/10) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.03% (15366/26945)
Line Coverage	46.12% (139448/302371)
Region Coverage	45.45% (70656/155473)
Branch Coverage	40.20% (37317/92820)

dataroaring

LGTM

github-actions · 2025-07-01T03:35:17Z

PR approved by at least one committer and no changes requested.

## 1 Increase base compaction frequency In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes. ## 2. Increase the limit on compaction score per run The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.

…ters #52321 (#52605) Cherry-picked from #52321 Co-authored-by: Luwei <luwei@selectdb.com>

…ters #52321 (#52606) Cherry-picked from #52321 Co-authored-by: Luwei <luwei@selectdb.com>

…ters apache#52321 (apache#52605) Cherry-picked from apache#52321 Co-authored-by: Luwei <luwei@selectdb.com>

…#52321) ## 1 Increase base compaction frequency In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes. ## 2. Increase the limit on compaction score per run The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.

[enhance](compaction) optimize mow base compaction parameters
fa353b3

luwei16 requested review from dataroaring and gavinchou as code owners June 25, 2025 13:07

dataroaring added dev/3.0.x dev/3.1.x usercase Important user case type label labels Jun 25, 2025

Merge remote-tracking branch 'doris-origin/master' into luwei/improve…
64b1614
…-mow-compact-new

gavinchou previously approved these changes Jun 26, 2025
View reviewed changes

gavinchou added the cloud label Jun 26, 2025

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 26, 2025

github-actionsBot added the reviewed label Jun 26, 2025

fix
062558c

luwei16 dismissed gavinchou’s stale review via 062558cJune 29, 2025 10:51

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 29, 2025

fix
199db7c

dataroaring approved these changes Jul 1, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 1, 2025

Yukang-Lian approved these changes Jul 1, 2025
View reviewed changes

dataroaring merged commit 0f8a933 into apache:masterJul 1, 2025

github-actionsBot mentioned this pull request Jul 1, 2025
branch-3.0: [enhance](compaction) optimize mow base compaction parameters #52321 #52605
Merged

github-actionsBot mentioned this pull request Jul 1, 2025
branch-3.1: [enhance](compaction) optimize mow base compaction parameters #52321 #52606
Merged

dataroaring pushed a commit that referenced this pull request Jul 2, 2025
branch-3.0: [enhance](compaction) optimize mow base compaction parame…
5551e58
…ters #52321 (#52605) Cherry-picked from #52321 Co-authored-by: Luwei <luwei@selectdb.com>

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jul 2, 2025

morrySnow pushed a commit that referenced this pull request Jul 2, 2025
branch-3.1: [enhance](compaction) optimize mow base compaction parame…
55187ab
…ters #52321 (#52606) Cherry-picked from #52321 Co-authored-by: Luwei <luwei@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jul 2, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
branch-3.0: [enhance](compaction) optimize mow base compaction parame…
44fcf65
…ters apache#52321 (apache#52605) Cherry-picked from apache#52321 Co-authored-by: Luwei <luwei@selectdb.com>

gavinchou mentioned this pull request Aug 15, 2025
Release Note 3.0.7 #54868
Closed

Uh oh!

Conversation

luwei16 commented Jun 25, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1 Increase base compaction frequency

2. Increase the limit on compaction score per run

Uh oh!

Thearas commented Jun 25, 2025

Uh oh!

luwei16 commented Jun 25, 2025

Uh oh!

luwei16 commented Jun 26, 2025

Uh oh!

luwei16 commented Jun 26, 2025

Uh oh!

github-actionsBot commented Jun 26, 2025

Uh oh!

github-actionsBot commented Jun 26, 2025

Uh oh!

luwei16 commented Jun 29, 2025

Uh oh!

luwei16 commented Jun 29, 2025

Uh oh!

doris-robot commented Jun 29, 2025

Uh oh!

doris-robot commented Jun 29, 2025

Uh oh!

doris-robot commented Jun 29, 2025

Uh oh!

hello-stephen commented Jun 29, 2025

BE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

luwei16 commented Jun 25, 2025•
edited
Loading