[enhance](metrics)add metrics to show compaction task num by koarz · Pull Request #50706 · apache/doris

koarz · 2025-05-08T08:53:26Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

These metrics will be updated when the thread pool submits a task, the thread pool starts executing the task, and the thread pool is updated when the task is finished executing, where running uses increment to update instead of getting the number of running threads because the task executed by Defer is still part of the task Defer is finished executing the task before it is considered to be finished, and this leads to a situation where when the pool is finished executing the task, it is not considered to be finished until it is finished. When the last task finishes, the value of active_thread_num in Defer is 1, and the corresponding metric keeps 1, but it should actually be 0.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-05-08T08:53:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

koarz · 2025-05-08T08:53:43Z

run buildall

koarz · 2025-05-08T09:18:10Z

run buildall

doris-robot · 2025-05-08T10:20:10Z

TPC-H: Total hot run time: 34875 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a2b475e3415ed2bcb4867bf3e5b510145535dec, data reload: false
------ Round 1 ----------------------------------
q1	27015	5076	5048	5048
q2	2094	291	191	191
q3	10380	1312	738	738
q4	10222	1013	536	536
q5	7547	2289	2377	2289
q6	184	165	134	134
q7	926	774	629	629
q8	9337	1336	1143	1143
q9	6885	5159	5124	5124
q10	6887	2302	1893	1893
q11	498	287	280	280
q12	354	364	222	222
q13	17752	3675	3082	3082
q14	251	222	222	222
q15	523	491	474	474
q16	438	439	383	383
q17	618	867	365	365
q18	7574	7169	7265	7169
q19	1825	949	541	541
q20	343	336	221	221
q21	4506	3427	3208	3208
q22	1046	1019	983	983
Total cold run time: 117205 ms
Total hot run time: 34875 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5252	5061	5086	5061
q2	238	324	229	229
q3	2208	2686	2335	2335
q4	1438	1970	1418	1418
q5	4499	4463	4415	4415
q6	263	181	140	140
q7	2054	1930	1736	1736
q8	2602	2507	2794	2507
q9	7268	6935	7187	6935
q10	2993	3162	2798	2798
q11	592	505	491	491
q12	730	787	625	625
q13	3577	3950	3360	3360
q14	289	298	281	281
q15	525	483	480	480
q16	459	499	453	453
q17	1166	1450	1464	1450
q18	8028	7667	7472	7472
q19	794	830	881	830
q20	1925	2006	1942	1942
q21	5349	4703	4736	4703
q22	1120	1053	1008	1008
Total cold run time: 53369 ms
Total hot run time: 50669 ms

doris-robot · 2025-05-08T10:32:25Z

TPC-DS: Total hot run time: 192463 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a2b475e3415ed2bcb4867bf3e5b510145535dec, data reload: false
query1	1410	1101	1071	1071
query2	6177	1842	1816	1816
query3	11007	4561	4404	4404
query4	53867	26225	23235	23235
query5	5085	524	464	464
query6	336	211	201	201
query7	4882	511	282	282
query8	313	251	232	232
query9	5538	2567	2561	2561
query10	442	334	284	284
query11	14929	15070	14975	14975
query12	165	115	107	107
query13	1026	536	417	417
query14	10189	6302	6585	6302
query15	212	201	198	198
query16	7188	627	517	517
query17	1126	753	597	597
query18	1663	422	326	326
query19	202	200	169	169
query20	128	123	120	120
query21	240	126	106	106
query22	4466	4691	4432	4432
query23	34259	33654	33786	33654
query24	6621	2387	2448	2387
query25	469	468	397	397
query26	700	283	158	158
query27	2345	498	332	332
query28	3346	2117	2114	2114
query29	578	561	429	429
query30	265	215	194	194
query31	867	844	763	763
query32	71	62	68	62
query33	456	351	322	322
query34	778	897	543	543
query35	840	862	749	749
query36	924	1006	912	912
query37	116	107	77	77
query38	4168	4110	4282	4110
query39	1518	1475	1436	1436
query40	221	123	105	105
query41	56	54	53	53
query42	128	107	102	102
query43	487	506	475	475
query44	1347	823	830	823
query45	178	175	171	171
query46	838	1039	663	663
query47	1891	1924	1788	1788
query48	386	411	316	316
query49	685	507	428	428
query50	677	713	412	412
query51	4224	4248	4197	4197
query52	115	108	100	100
query53	229	261	188	188
query54	570	585	507	507
query55	88	87	85	85
query56	302	302	300	300
query57	1177	1174	1155	1155
query58	267	253	255	253
query59	2631	2801	2732	2732
query60	335	328	313	313
query61	137	127	126	126
query62	731	725	680	680
query63	231	195	189	189
query64	1866	1069	688	688
query65	4345	4247	4187	4187
query66	717	409	312	312
query67	16165	15608	15410	15410
query68	7465	895	518	518
query69	540	306	261	261
query70	1157	1087	1102	1087
query71	506	321	294	294
query72	5793	4819	4899	4819
query73	1511	701	350	350
query74	9003	9045	8579	8579
query75	3849	3211	2704	2704
query76	4166	1185	813	813
query77	636	360	288	288
query78	10116	10122	9217	9217
query79	2518	812	561	561
query80	613	508	427	427
query81	493	259	215	215
query82	452	125	93	93
query83	359	243	236	236
query84	297	102	83	83
query85	802	352	360	352
query86	383	298	259	259
query87	4403	4492	4249	4249
query88	3655	2214	2193	2193
query89	405	309	277	277
query90	1842	202	205	202
query91	140	146	116	116
query92	75	61	55	55
query93	1990	944	585	585
query94	665	410	303	303
query95	361	293	283	283
query96	483	559	268	268
query97	3199	3292	3090	3090
query98	229	208	211	208
query99	1443	1388	1265	1265
Total cold run time: 299485 ms
Total hot run time: 192463 ms

doris-robot · 2025-05-08T10:37:38Z

ClickBench: Total hot run time: 28.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a2b475e3415ed2bcb4867bf3e5b510145535dec, data reload: false
query1	0.04	0.04	0.04
query2	0.12	0.10	0.11
query3	0.26	0.19	0.19
query4	1.60	0.19	0.11
query5	0.58	0.57	0.54
query6	1.17	0.73	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.52	0.52
query10	0.58	0.58	0.57
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.58
query14	0.78	0.80	0.81
query15	0.88	0.85	0.84
query16	0.37	0.39	0.38
query17	1.03	1.01	1.03
query18	0.21	0.20	0.20
query19	1.94	1.75	1.83
query20	0.01	0.01	0.02
query21	15.39	0.88	0.55
query22	0.76	1.21	0.63
query23	15.00	1.39	0.64
query24	7.06	1.26	0.60
query25	0.57	0.09	0.07
query26	0.65	0.16	0.14
query27	0.05	0.04	0.05
query28	9.20	0.86	0.43
query29	12.52	3.90	3.29
query30	0.25	0.09	0.06
query31	2.82	0.58	0.39
query32	3.23	0.53	0.48
query33	3.06	3.13	3.09
query34	15.86	5.10	4.50
query35	4.52	4.50	4.44
query36	0.67	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.05	0.04
query39	0.03	0.02	0.02
query40	0.18	0.15	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.24 s
Total hot run time: 28.78 s

hello-stephen · 2025-05-08T11:51:30Z

BE UT Coverage Report

Increment line coverage 40.91% (18/44) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.68% (14894/26751)
Line Coverage	44.48% (131369/295321)
Region Coverage	43.38% (66174/152536)
Branch Coverage	38.06% (33792/88790)

hello-stephen · 2025-05-08T14:05:43Z

BE Regression && UT Coverage Report

Increment line coverage 63.64% (28/44) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (20829/26250)
Line Coverage	72.57% (213946/294824)
Region Coverage	70.75% (126566/178898)
Branch Coverage	64.39% (65240/101322)

koarz · 2025-05-09T09:41:25Z

run cloud_p0

luwei16 · 2025-05-12T09:38:59Z

 _submitted_base_compactions[tablet->tablet_id()] = compaction;
 }
 st = _base_compaction_thread_pool->submit_func([=, this, compaction = std::move(compaction)]() {
+ DorisMetrics::instance()->base_compaction_task_running_total->increment(1);


Can the compaction_task_running_total metirc directly use the member function of the thread pool class?

github-actions · 2025-05-12T11:31:20Z

PR approved by anyone and no changes requested.

koarz · 2025-05-13T03:47:08Z

run buildall

koarz · 2025-05-13T03:54:57Z

run buildall

doris-robot · 2025-05-13T06:46:16Z

BE UT Coverage Report

Increment line coverage 40.91% (18/44) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.79% (14896/26698)
Line Coverage	44.59% (131843/295646)
Region Coverage	43.65% (66284/151857)
Branch Coverage	38.26% (33962/88760)

koarz · 2025-05-13T08:26:40Z

run performance

koarz · 2025-05-13T08:26:52Z

run external

koarz · 2025-05-13T08:27:01Z

run cloud_p0

hello-stephen · 2025-05-13T09:50:25Z

BE Regression && UT Coverage Report

Increment line coverage 63.64% (28/44) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.31% (20830/26264)
Line Coverage	72.50% (214298/295582)
Region Coverage	70.67% (126053/178362)
Branch Coverage	64.43% (65310/101370)

doris-robot · 2025-05-13T09:59:43Z

TPC-H: Total hot run time: 34698 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a2cc946b32adbb5b1d5042f1dac54896b35a6b7, data reload: false
------ Round 1 ----------------------------------
q1	26486	5136	5172	5136
q2	2106	296	190	190
q3	10369	1273	727	727
q4	10225	1027	525	525
q5	7558	2410	2417	2410
q6	186	167	139	139
q7	980	748	610	610
q8	9337	1463	1211	1211
q9	6865	5309	5275	5275
q10	6854	2315	1915	1915
q11	504	292	277	277
q12	351	354	219	219
q13	17769	3731	3115	3115
q14	233	248	225	225
q15	535	484	481	481
q16	447	432	374	374
q17	595	917	350	350
q18	7705	7298	7175	7175
q19	1901	983	577	577
q20	329	347	219	219
q21	4249	3325	2536	2536
q22	1089	1013	1012	1012
Total cold run time: 116673 ms
Total hot run time: 34698 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5392	5267	5248	5248
q2	253	331	225	225
q3	2156	2693	2331	2331
q4	1390	2015	1489	1489
q5	4529	4436	4397	4397
q6	231	168	124	124
q7	2044	1928	1767	1767
q8	2636	2643	2587	2587
q9	7292	7047	7218	7047
q10	3035	3243	2796	2796
q11	577	521	488	488
q12	676	783	630	630
q13	3617	3964	3307	3307
q14	292	305	268	268
q15	516	475	470	470
q16	463	489	445	445
q17	1233	1571	1447	1447
q18	7809	7624	7423	7423
q19	881	924	879	879
q20	1981	2008	1851	1851
q21	5364	4698	4763	4698
q22	1152	1122	1064	1064
Total cold run time: 53519 ms
Total hot run time: 50981 ms

doris-robot · 2025-05-13T10:11:40Z

TPC-DS: Total hot run time: 193568 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a2cc946b32adbb5b1d5042f1dac54896b35a6b7, data reload: false
query1	1406	1119	1073	1073
query2	6309	1857	1866	1857
query3	11180	4662	4559	4559
query4	25775	23893	23222	23222
query5	4787	609	460	460
query6	316	216	204	204
query7	3987	499	291	291
query8	304	243	236	236
query9	8484	2652	2659	2652
query10	494	315	269	269
query11	15360	15128	14828	14828
query12	168	107	105	105
query13	1551	506	389	389
query14	9214	5995	5978	5978
query15	201	185	163	163
query16	7594	651	481	481
query17	1131	741	610	610
query18	2034	408	295	295
query19	188	203	158	158
query20	127	128	118	118
query21	209	131	120	120
query22	4419	4492	4288	4288
query23	34705	33883	33630	33630
query24	8600	2439	2451	2439
query25	537	513	420	420
query26	1040	274	154	154
query27	3011	550	354	354
query28	4819	2198	2144	2144
query29	675	617	497	497
query30	275	224	190	190
query31	930	897	821	821
query32	78	64	62	62
query33	535	353	305	305
query34	788	887	516	516
query35	782	836	744	744
query36	960	981	906	906
query37	114	103	81	81
query38	4243	4235	4235	4235
query39	1510	1451	1462	1451
query40	213	120	104	104
query41	63	64	52	52
query42	123	113	117	113
query43	507	520	499	499
query44	1386	849	832	832
query45	189	176	167	167
query46	856	1028	668	668
query47	1801	1883	1773	1773
query48	416	447	330	330
query49	736	523	433	433
query50	683	698	423	423
query51	4314	4321	4129	4129
query52	114	107	101	101
query53	230	263	194	194
query54	597	588	536	536
query55	82	85	87	85
query56	312	293	286	286
query57	1229	1213	1163	1163
query58	267	259	263	259
query59	2606	2810	2784	2784
query60	349	323	330	323
query61	128	155	125	125
query62	779	737	665	665
query63	230	186	197	186
query64	3473	1076	671	671
query65	4454	4376	4429	4376
query66	962	414	313	313
query67	16041	15487	15530	15487
query68	9027	886	516	516
query69	497	310	271	271
query70	1257	1090	1111	1090
query71	490	307	291	291
query72	5891	4716	4758	4716
query73	710	617	354	354
query74	9093	9242	8994	8994
query75	4118	3237	2742	2742
query76	3679	1180	756	756
query77	808	375	276	276
query78	10072	10120	9443	9443
query79	2722	791	578	578
query80	656	548	428	428
query81	482	257	223	223
query82	437	122	100	100
query83	287	247	242	242
query84	291	111	91	91
query85	800	361	307	307
query86	333	298	288	288
query87	4556	4526	4463	4463
query88	2944	2306	2289	2289
query89	439	317	287	287
query90	1933	226	223	223
query91	145	144	113	113
query92	73	63	56	56
query93	2387	941	569	569
query94	667	415	296	296
query95	377	298	282	282
query96	496	578	284	284
query97	3192	3250	3120	3120
query98	221	211	203	203
query99	1449	1370	1295	1295
Total cold run time: 282509 ms
Total hot run time: 193568 ms

doris-robot · 2025-05-13T10:16:50Z

ClickBench: Total hot run time: 29.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a2cc946b32adbb5b1d5042f1dac54896b35a6b7, data reload: false
query1	0.04	0.04	0.03
query2	0.13	0.10	0.11
query3	0.26	0.19	0.19
query4	1.59	0.20	0.20
query5	0.59	0.61	0.60
query6	1.20	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.52	0.51
query10	0.57	0.56	0.56
query11	0.15	0.11	0.12
query12	0.16	0.12	0.11
query13	0.61	0.60	0.59
query14	0.78	0.80	0.80
query15	0.89	0.85	0.85
query16	0.40	0.39	0.37
query17	1.05	1.05	1.03
query18	0.23	0.22	0.21
query19	1.92	1.86	1.85
query20	0.02	0.00	0.01
query21	15.40	0.89	0.54
query22	0.77	1.25	0.60
query23	14.93	1.39	0.57
query24	6.62	2.05	0.67
query25	0.51	0.22	0.08
query26	0.60	0.16	0.14
query27	0.05	0.06	0.05
query28	8.95	0.90	0.46
query29	12.53	3.97	3.26
query30	0.25	0.10	0.06
query31	2.81	0.59	0.38
query32	3.22	0.56	0.45
query33	3.06	3.03	3.07
query34	15.80	5.13	4.59
query35	4.57	4.55	4.51
query36	0.66	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.42 s
Total hot run time: 29.04 s

koarz · 2025-05-13T10:29:58Z

run cloud_p0

dataroaring

LGTM

github-actions · 2025-05-14T03:48:16Z

PR approved by at least one committer and no changes requested.

…50706 (#50883) Cherry-picked from #50706 Co-authored-by: koarz <lihao@selectdb.com>

koarz added 2 commits May 8, 2025 11:02

[enhance](metrics)add metrics to show compaction task num
db6fd26

1
83edefc

koarz requested review from dataroaring and gavinchou as code owners May 8, 2025 08:53

fix
8a2b475

luwei16 reviewed May 12, 2025
View reviewed changes

luwei16 approved these changes May 12, 2025
View reviewed changes

github-actionsBot added the reviewed label May 12, 2025

Yukang-Lian approved these changes May 13, 2025
View reviewed changes

Update doris_metrics.cpp
7e4f0eb

format
3a2cc94

dataroaring approved these changes May 14, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 14, 2025

dataroaring merged commit 4aa3b49 into apache:masterMay 14, 2025

dataroaring added the dev/3.0.x label May 14, 2025

github-actionsBot pushed a commit that referenced this pull request May 14, 2025
[enhance](metrics)add metrics to show compaction task num (#50706)
c8636e5

github-actionsBot mentioned this pull request May 14, 2025
branch-3.0: [enhance](metrics)add metrics to show compaction task num #50706 #50883
Merged

koarz deleted the compaction_metrics branch May 14, 2025 10:33

dataroaring pushed a commit that referenced this pull request May 14, 2025
branch-3.0: [enhance](metrics)add metrics to show compaction task num#…
246da2d
…50706 (#50883) Cherry-picked from #50706 Co-authored-by: koarz <lihao@selectdb.com>

dataroaring added dev/3.0.6-merged dev/2.1.x and removed dev/3.0.x labels May 14, 2025

github-actionsBot added the dev/2.1.x-conflict label May 14, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
[enhance](metrics)add metrics to show compaction task num (apache#50706)
5a6dedd

Uh oh!

Conversation

koarz commented May 8, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented May 8, 2025

Uh oh!

koarz commented May 8, 2025

Uh oh!

koarz commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

hello-stephen commented May 8, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 8, 2025

BE Regression && UT Coverage Report

Uh oh!

koarz commented May 9, 2025

Uh oh!

luwei16May 12, 2025

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 12, 2025

Uh oh!

koarz commented May 13, 2025

Uh oh!

koarz commented May 13, 2025

Uh oh!

doris-robot commented May 13, 2025

BE UT Coverage Report

Uh oh!

koarz commented May 13, 2025

Uh oh!

koarz commented May 13, 2025

Uh oh!

koarz commented May 13, 2025

Uh oh!

hello-stephen commented May 13, 2025

BE Regression && UT Coverage Report

Uh oh!

doris-robot commented May 13, 2025

Uh oh!

doris-robot commented May 13, 2025

Uh oh!

doris-robot commented May 13, 2025

Uh oh!

koarz commented May 13, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

koarz commented May 8, 2025•
edited
Loading