Skip to content

[Enhancement](Compaction) Support auto set cumu compaction threads num base on cpu num - #53133

Merged
dataroaring merged 1 commit into
apache:masterfrom
Yukang-Lian:Auto-Set-Cumu-Threads
Jul 14, 2025
Merged

[Enhancement](Compaction) Support auto set cumu compaction threads num base on cpu num#53133
dataroaring merged 1 commit into
apache:masterfrom
Yukang-Lian:Auto-Set-Cumu-Threads

Conversation

@Yukang-Lian

@Yukang-LianYukang-Lian commented Jul 11, 2025

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Jul 11, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring

dataroaring commented Jul 12, 2025

Copy link
Copy Markdown
Contributor

run buildall

dataroaring
dataroaring approved these changes Jul 12, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2025
@github-actions

github-actionsBot commented Jul 12, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jul 12, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Jul 12, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33205 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f81a4bed28c412bb6b1b309649c139b4afdb3f4c, data reload: false
------ Round 1 ----------------------------------
q1	17606	5250	5112	5112
q2	1937	297	180	180
q3	10327	1284	741	741
q4	10253	990	527	527
q5	7733	2368	2308	2308
q6	183	162	129	129
q7	920	768	600	600
q8	9319	1232	1102	1102
q9	6754	5104	5083	5083
q10	6930	2368	1974	1974
q11	500	295	278	278
q12	351	351	217	217
q13	17754	3684	3030	3030
q14	230	234	214	214
q15	545	480	481	480
q16	431	430	371	371
q17	584	873	363	363
q18	7510	7116	7032	7032
q19	1351	936	559	559
q20	333	345	219	219
q21	3657	3233	2390	2390
q22	358	324	296	296
Total cold run time: 105566 ms
Total hot run time: 33205 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5261	5121	5102	5102
q2	243	320	232	232
q3	2134	2653	2303	2303
q4	1373	1764	1299	1299
q5	4270	4221	4448	4221
q6	209	166	131	131
q7	1929	2068	1866	1866
q8	2640	2595	2545	2545
q9	7247	7255	7213	7213
q10	3108	3281	2890	2890
q11	589	515	554	515
q12	698	914	748	748
q13	3582	3961	3371	3371
q14	289	296	300	296
q15	521	477	473	473
q16	441	487	457	457
q17	1182	1589	1379	1379
q18	7779	7786	7743	7743
q19	842	963	1081	963
q20	1981	1958	1807	1807
q21	4753	4421	4295	4295
q22	641	569	525	525
Total cold run time: 51712 ms
Total hot run time: 50374 ms

@doris-robot

doris-robot commented Jul 12, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 187033 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f81a4bed28c412bb6b1b309649c139b4afdb3f4c, data reload: false
query1	981	396	385	385
query2	6486	1682	1695	1682
query3	6744	217	220	217
query4	26281	23731	23623	23623
query5	4959	593	455	455
query6	309	215	206	206
query7	4627	484	283	283
query8	287	223	212	212
query9	8637	2649	2654	2649
query10	506	334	262	262
query11	15981	15020	14830	14830
query12	173	101	101	101
query13	1638	530	390	390
query14	9378	6027	5884	5884
query15	202	194	172	172
query16	7510	423	248	248
query17	1305	708	564	564
query18	1980	390	294	294
query19	187	188	154	154
query20	127	144	110	110
query21	212	131	106	106
query22	4086	4055	3968	3968
query23	33880	33029	33126	33029
query24	8374	2398	2340	2340
query25	512	475	396	396
query26	1084	259	147	147
query27	2765	500	333	333
query28	4330	2124	2090	2090
query29	721	540	429	429
query30	285	222	190	190
query31	898	837	737	737
query32	69	62	61	61
query33	568	319	282	282
query34	792	840	535	535
query35	603	642	570	570
query36	931	974	899	899
query37	116	98	77	77
query38	4158	4216	4107	4107
query39	1455	1463	1411	1411
query40	205	120	106	106
query41	57	53	51	51
query42	119	110	110	110
query43	513	500	490	490
query44	1393	823	835	823
query45	177	172	161	161
query46	830	1022	619	619
query47	1727	1781	1723	1723
query48	388	424	312	312
query49	751	475	382	382
query50	653	678	438	438
query51	5351	5759	5370	5370
query52	105	104	102	102
query53	228	253	185	185
query54	563	567	495	495
query55	86	81	83	81
query56	308	299	281	281
query57	1188	1180	1122	1122
query58	256	250	264	250
query59	2720	2819	2520	2520
query60	336	325	306	306
query61	154	151	150	150
query62	781	708	666	666
query63	222	195	194	194
query64	4175	1187	832	832
query65	4228	4190	4193	4190
query66	1002	407	319	319
query67	15700	15568	15195	15195
query68	7844	883	528	528
query69	514	315	271	271
query70	1218	1094	1111	1094
query71	422	324	300	300
query72	5555	4750	4876	4750
query73	665	630	356	356
query74	8917	9096	8716	8716
query75	3498	3170	2697	2697
query76	3280	1131	715	715
query77	669	381	296	296
query78	10869	10915	10254	10254
query79	1584	814	586	586
query80	556	507	454	454
query81	450	260	223	223
query82	177	122	100	100
query83	260	256	239	239
query84	257	97	84	84
query85	804	446	405	405
query86	330	297	292	292
query87	4354	4506	4390	4390
query88	3032	2286	2270	2270
query89	379	317	291	291
query90	1969	214	224	214
query91	135	141	114	114
query92	73	61	54	54
query93	1521	946	589	589
query94	674	310	198	198
query95	369	294	282	282
query96	492	556	278	278
query97	2675	2752	2601	2601
query98	234	203	200	200
query99	1309	1374	1291	1291
Total cold run time: 273438 ms
Total hot run time: 187033 ms

@doris-robot

doris-robot commented Jul 12, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f81a4bed28c412bb6b1b309649c139b4afdb3f4c, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.25	0.07	0.08
query4	1.61	0.11	0.10
query5	0.44	0.43	0.42
query6	1.16	0.67	0.68
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.61	0.51	0.51
query10	0.57	0.56	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.79	0.82	0.81
query15	0.90	0.88	0.86
query16	0.39	0.38	0.40
query17	1.04	1.06	1.03
query18	0.22	0.21	0.21
query19	1.99	1.86	1.85
query20	0.01	0.02	0.01
query21	15.38	0.91	0.56
query22	0.75	1.20	0.72
query23	14.88	1.40	0.59
query24	7.67	0.82	0.74
query25	0.47	0.18	0.09
query26	0.58	0.17	0.13
query27	0.06	0.06	0.05
query28	9.26	0.88	0.45
query29	12.54	3.89	3.26
query30	0.26	0.09	0.06
query31	2.83	0.61	0.39
query32	3.23	0.55	0.47
query33	3.08	3.04	3.06
query34	15.91	5.40	4.75
query35	4.86	4.82	4.81
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.04 s
Total hot run time: 29.33 s

@hello-stephen

hello-stephen commented Jul 12, 2025

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage57.45% (15615/27182)
Line Coverage46.16% (139374/301968)
Region Coverage35.49% (104163/293538)
Branch Coverage38.03% (45941/120798)

@Yukang-Lian

Yukang-Lian commented Jul 12, 2025

Copy link
Copy Markdown
CollaboratorAuthor

run cloud_p0

@dataroaring
dataroaring merged commit a07a01b into apache:masterJul 14, 2025
github-actionsBot pushed a commit that referenced this pull request Jul 14, 2025
@Yukang-LianYour Name
[Enhancement](Compaction) Support auto set cumu compaction threads nu…
ab98c0a
…m base on cpu num (#53133)
@github-actionsgithub-actionsBot mentioned this pull request Jul 14, 2025
Merged
github-actionsBot pushed a commit that referenced this pull request Jul 14, 2025
@Yukang-LianYour Name
[Enhancement](Compaction) Support auto set cumu compaction threads nu…
9e586d3
…m base on cpu num (#53133)
@github-actionsgithub-actionsBot mentioned this pull request Jul 14, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jul 14, 2025
@github-actions@Yukang-Lian
branch-3.0: [Enhancement](Compaction) Support auto set cumu compactio…
fbe7400
…n threads num base on cpu num #53133 (#53215)
Cherry-picked from #53133
Co-authored-by: abmdocrt <lianyukang@selectdb.com>
morrySnow pushed a commit that referenced this pull request Jul 15, 2025
@github-actions@Yukang-Lian
branch-3.1: [Enhancement](Compaction) Support auto set cumu compactio…
7a5e166
…n threads num base on cpu num #53133 (#53216)
Cherry-picked from #53133
Co-authored-by: abmdocrt <lianyukang@selectdb.com>
@gavinchougavinchou mentioned this pull request Aug 15, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@luwei16luwei16luwei16 approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.7-mergeddev/3.1.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Yukang-Lian@Thearas@dataroaring@doris-robot@hello-stephen@luwei16@morrySnow