[fix](group commit) reduce cpu cost for group_commit get_block by mymeiyi · Pull Request #49822 · apache/doris

mymeiyi · 2025-04-07T03:18:13Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-07T03:18:18Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

mymeiyi · 2025-04-07T03:26:26Z

run buildall

doris-robot · 2025-04-07T04:20:52Z

TPC-H: Total hot run time: 34367 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3da47f3f6e3e421d67cd3d36a366ef6c65f64007, data reload: false
------ Round 1 ----------------------------------
q1	25929	5066	5017	5017
q2	2072	273	187	187
q3	10398	1286	707	707
q4	10219	1003	561	561
q5	7521	2404	2367	2367
q6	188	161	133	133
q7	916	758	627	627
q8	9313	1323	1159	1159
q9	6905	5193	5208	5193
q10	6885	2329	1901	1901
q11	482	290	276	276
q12	358	367	223	223
q13	17786	3760	3028	3028
q14	232	229	208	208
q15	545	488	481	481
q16	645	602	556	556
q17	618	866	382	382
q18	7589	7266	7111	7111
q19	1295	965	566	566
q20	355	339	225	225
q21	4481	3446	2510	2510
q22	1071	1004	949	949
Total cold run time: 115803 ms
Total hot run time: 34367 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5107	5113	5093	5093
q2	239	329	239	239
q3	2195	2661	2335	2335
q4	1424	1851	1496	1496
q5	4717	4406	4407	4406
q6	217	168	125	125
q7	1954	1921	1727	1727
q8	2626	2649	2597	2597
q9	7293	7112	7170	7112
q10	2960	3149	2746	2746
q11	578	525	499	499
q12	716	790	635	635
q13	3536	3914	3371	3371
q14	281	292	288	288
q15	538	509	506	506
q16	681	673	669	669
q17	1170	1583	1390	1390
q18	7836	7713	7420	7420
q19	840	828	890	828
q20	1907	1995	1884	1884
q21	5458	4916	4809	4809
q22	1112	1053	1017	1017
Total cold run time: 53385 ms
Total hot run time: 51192 ms

doris-robot · 2025-04-07T04:33:05Z

TPC-DS: Total hot run time: 192238 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3da47f3f6e3e421d67cd3d36a366ef6c65f64007, data reload: false
query1	1363	1051	996	996
query2	6281	1930	1866	1866
query3	10984	4487	4664	4487
query4	55371	24123	22801	22801
query5	5064	525	453	453
query6	375	189	186	186
query7	5028	501	278	278
query8	326	255	244	244
query9	6712	2600	2614	2600
query10	432	321	259	259
query11	15207	15082	14803	14803
query12	157	114	106	106
query13	1144	508	392	392
query14	10867	6368	6222	6222
query15	191	202	175	175
query16	7130	648	514	514
query17	1077	743	583	583
query18	1560	420	335	335
query19	210	205	182	182
query20	128	118	130	118
query21	216	138	104	104
query22	4676	4454	4440	4440
query23	33936	33264	33338	33264
query24	5930	2417	2386	2386
query25	446	461	430	430
query26	726	266	147	147
query27	1948	476	344	344
query28	2964	2472	2491	2472
query29	572	565	428	428
query30	278	219	230	219
query31	865	897	800	800
query32	84	61	63	61
query33	445	368	319	319
query34	788	861	545	545
query35	816	839	808	808
query36	950	998	934	934
query37	123	106	81	81
query38	4254	4316	4021	4021
query39	1478	1414	1456	1414
query40	223	136	104	104
query41	53	55	50	50
query42	123	109	109	109
query43	476	516	487	487
query44	1365	848	816	816
query45	181	185	173	173
query46	843	1017	664	664
query47	1840	1905	1815	1815
query48	401	424	319	319
query49	696	518	420	420
query50	669	692	417	417
query51	4222	4320	4144	4144
query52	106	104	98	98
query53	233	270	194	194
query54	598	594	532	532
query55	88	81	90	81
query56	310	299	296	296
query57	1161	1188	1156	1156
query58	280	266	253	253
query59	2682	2888	2779	2779
query60	348	323	314	314
query61	131	131	129	129
query62	742	745	668	668
query63	230	188	192	188
query64	1802	1071	721	721
query65	4392	4274	4273	4273
query66	779	402	304	304
query67	16032	15933	15346	15346
query68	6828	892	517	517
query69	545	301	263	263
query70	1160	1069	1156	1069
query71	501	316	305	305
query72	5972	4792	4772	4772
query73	1269	631	357	357
query74	8980	9166	8882	8882
query75	3964	3213	2702	2702
query76	4261	1193	770	770
query77	642	358	276	276
query78	10222	10232	9207	9207
query79	1962	805	561	561
query80	569	526	445	445
query81	475	259	221	221
query82	330	122	102	102
query83	258	312	236	236
query84	297	106	85	85
query85	788	359	304	304
query86	368	307	289	289
query87	4393	4344	4444	4344
query88	3412	2187	2204	2187
query89	402	305	282	282
query90	1916	202	211	202
query91	143	143	111	111
query92	73	60	61	60
query93	1277	968	581	581
query94	634	398	319	319
query95	367	293	289	289
query96	492	553	273	273
query97	3153	3199	3097	3097
query98	229	206	204	204
query99	1333	1401	1259	1259
Total cold run time: 299323 ms
Total hot run time: 192238 ms

doris-robot · 2025-04-07T04:38:23Z

ClickBench: Total hot run time: 31.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3da47f3f6e3e421d67cd3d36a366ef6c65f64007, data reload: false
query1	0.04	0.04	0.03
query2	0.14	0.11	0.11
query3	0.26	0.20	0.20
query4	1.59	0.20	0.20
query5	0.60	0.56	0.58
query6	1.19	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.51	0.50
query10	0.61	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.58	0.61
query14	2.72	2.81	2.83
query15	0.94	0.84	0.84
query16	0.40	0.38	0.37
query17	1.03	1.01	1.01
query18	0.21	0.19	0.19
query19	1.88	1.97	1.80
query20	0.01	0.01	0.02
query21	15.37	0.91	0.54
query22	0.74	1.20	0.69
query23	14.93	1.35	0.61
query24	7.36	1.25	1.20
query25	0.49	0.23	0.08
query26	0.65	0.16	0.12
query27	0.05	0.05	0.04
query28	9.44	0.86	0.43
query29	12.56	4.08	3.41
query30	0.25	0.09	0.07
query31	2.83	0.61	0.38
query32	3.23	0.54	0.45
query33	3.09	3.10	3.10
query34	15.60	5.19	4.50
query35	4.49	4.45	4.53
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.41 s
Total hot run time: 31.65 s

mymeiyi · 2025-04-07T06:03:22Z

run p0

hello-stephen · 2025-04-07T06:16:36Z

BE UT Coverage Report

Increment line coverage 0.00% (0/22) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.23% (13998/26800)
Line Coverage	40.97% (120615/294387)
Region Coverage	39.69% (61349/154588)
Branch Coverage	34.31% (30628/89274)

mymeiyi · 2025-04-07T07:36:03Z

run buildall

doris-robot · 2025-04-07T08:43:44Z

TPC-H: Total hot run time: 34214 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5103a6e2860c8456865f2035cb58790c78c0954c, data reload: false
------ Round 1 ----------------------------------
q1	26063	5095	5006	5006
q2	2062	285	184	184
q3	10397	1236	675	675
q4	10211	1007	516	516
q5	7545	2373	2382	2373
q6	191	170	136	136
q7	951	755	621	621
q8	9343	1321	1103	1103
q9	6816	5168	5181	5168
q10	6807	2322	1884	1884
q11	473	272	266	266
q12	350	356	221	221
q13	17783	3680	3121	3121
q14	221	214	204	204
q15	544	484	501	484
q16	639	616	577	577
q17	586	877	358	358
q18	7541	7133	7063	7063
q19	1739	979	573	573
q20	331	338	233	233
q21	3943	3362	2465	2465
q22	1093	1024	983	983
Total cold run time: 115629 ms
Total hot run time: 34214 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5185	5123	5140	5123
q2	239	327	225	225
q3	2173	2658	2309	2309
q4	1448	1851	1552	1552
q5	4450	4388	4401	4388
q6	221	166	125	125
q7	1996	1945	1775	1775
q8	2592	2608	2576	2576
q9	7285	7238	7084	7084
q10	2998	3189	2745	2745
q11	579	520	490	490
q12	674	769	651	651
q13	3556	3872	3249	3249
q14	274	312	291	291
q15	542	489	465	465
q16	644	698	637	637
q17	1169	1570	1413	1413
q18	7771	7616	7374	7374
q19	809	876	1072	876
q20	2085	1974	1878	1878
q21	5125	4846	4821	4821
q22	1117	1094	1040	1040
Total cold run time: 52932 ms
Total hot run time: 51087 ms

doris-robot · 2025-04-07T08:55:39Z

TPC-DS: Total hot run time: 193538 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5103a6e2860c8456865f2035cb58790c78c0954c, data reload: false
query1	1382	1050	1048	1048
query2	6246	2000	1952	1952
query3	11128	4665	4881	4665
query4	25425	23714	23138	23138
query5	4899	636	445	445
query6	307	201	206	201
query7	3999	486	288	288
query8	302	233	214	214
query9	8506	2553	2563	2553
query10	490	318	252	252
query11	15219	15188	14972	14972
query12	159	103	105	103
query13	1540	505	390	390
query14	9087	6204	6118	6118
query15	202	181	166	166
query16	7244	638	481	481
query17	1160	749	600	600
query18	2006	427	341	341
query19	198	199	202	199
query20	130	131	121	121
query21	221	124	107	107
query22	4521	4865	4351	4351
query23	34453	33554	33361	33361
query24	8540	2411	2427	2411
query25	517	472	404	404
query26	1167	269	154	154
query27	2939	495	340	340
query28	4966	2449	2471	2449
query29	745	547	465	465
query30	287	237	202	202
query31	915	877	796	796
query32	72	62	58	58
query33	525	364	318	318
query34	798	896	519	519
query35	835	862	791	791
query36	972	1027	898	898
query37	114	102	75	75
query38	4189	4135	4260	4135
query39	1501	1437	1431	1431
query40	216	121	106	106
query41	51	54	52	52
query42	119	110	111	110
query43	527	517	500	500
query44	1298	813	800	800
query45	182	173	161	161
query46	842	1032	649	649
query47	1838	1905	1791	1791
query48	380	438	306	306
query49	753	532	405	405
query50	636	689	446	446
query51	4283	4263	4234	4234
query52	107	112	97	97
query53	241	261	192	192
query54	591	578	518	518
query55	82	78	85	78
query56	303	306	282	282
query57	1179	1214	1159	1159
query58	270	265	264	264
query59	2906	2812	2886	2812
query60	331	323	317	317
query61	154	152	157	152
query62	768	758	720	720
query63	236	191	189	189
query64	4278	1114	799	799
query65	4477	4315	4297	4297
query66	1532	430	342	342
query67	16194	15675	15342	15342
query68	8750	881	518	518
query69	486	303	265	265
query70	1204	1110	1108	1108
query71	471	310	301	301
query72	5788	4748	4934	4748
query73	737	661	347	347
query74	8898	8940	8974	8940
query75	3909	3221	2672	2672
query76	3719	1200	766	766
query77	798	369	294	294
query78	10007	10223	9310	9310
query79	1915	825	558	558
query80	586	508	514	508
query81	463	259	226	226
query82	451	122	97	97
query83	262	252	236	236
query84	245	101	82	82
query85	784	357	318	318
query86	334	301	290	290
query87	4468	4438	4375	4375
query88	3361	2213	2193	2193
query89	388	313	289	289
query90	1904	208	211	208
query91	142	141	115	115
query92	73	60	54	54
query93	1540	941	585	585
query94	678	431	318	318
query95	374	297	288	288
query96	477	566	278	278
query97	3166	3244	3120	3120
query98	233	212	206	206
query99	1415	1398	1293	1293
Total cold run time: 280706 ms
Total hot run time: 193538 ms

doris-robot · 2025-04-07T09:00:57Z

ClickBench: Total hot run time: 30.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5103a6e2860c8456865f2035cb58790c78c0954c, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.19
query4	1.58	0.20	0.20
query5	0.61	0.59	0.59
query6	1.20	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.53
query10	0.59	0.60	0.55
query11	0.16	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.59	0.61
query14	2.81	2.70	2.70
query15	0.95	0.85	0.85
query16	0.40	0.38	0.37
query17	1.03	1.07	1.00
query18	0.21	0.20	0.20
query19	1.87	1.90	1.84
query20	0.01	0.01	0.01
query21	15.35	0.88	0.54
query22	0.76	1.09	0.65
query23	15.07	1.38	0.59
query24	7.01	2.02	0.50
query25	0.47	0.14	0.19
query26	0.60	0.16	0.13
query27	0.05	0.05	0.05
query28	9.50	0.85	0.43
query29	12.52	3.99	3.34
query30	0.24	0.09	0.06
query31	2.84	0.57	0.39
query32	3.23	0.55	0.47
query33	2.99	3.07	3.05
query34	15.65	5.13	4.52
query35	4.56	4.54	4.54
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.2 s
Total hot run time: 30.91 s

doris-robot · 2025-04-07T11:02:24Z

BE UT Coverage Report

Increment line coverage 0.00% (0/22) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.26% (14022/26832)
Line Coverage	41.04% (121049/294956)
Region Coverage	39.78% (61611/154882)
Branch Coverage	34.45% (30844/89538)

mymeiyi · 2025-04-11T03:38:47Z

run buildall

doris-robot · 2025-04-11T04:14:33Z

TPC-H: Total hot run time: 35526 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a023e7737c776901c1893fd5eea3c26e314f3df8, data reload: false
------ Round 1 ----------------------------------
q1	26046	5310	5275	5275
q2	2081	293	187	187
q3	10375	1394	737	737
q4	10240	1079	560	560
q5	7582	2729	2649	2649
q6	204	174	133	133
q7	1006	793	652	652
q8	9330	1425	1258	1258
q9	7078	5311	5314	5311
q10	6944	2367	1929	1929
q11	519	293	283	283
q12	382	390	227	227
q13	17796	3832	3152	3152
q14	248	236	217	217
q15	561	508	488	488
q16	666	628	597	597
q17	623	902	360	360
q18	7591	7147	7156	7147
q19	1570	1090	587	587
q20	347	366	222	222
q21	4317	3693	2568	2568
q22	1086	1058	987	987
Total cold run time: 116592 ms
Total hot run time: 35526 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5593	5520	5506	5506
q2	253	348	244	244
q3	2255	2761	2485	2485
q4	1628	2100	1557	1557
q5	4633	4468	4420	4420
q6	264	183	136	136
q7	2092	1981	1804	1804
q8	2891	2809	2777	2777
q9	7154	7105	7133	7105
q10	3164	3296	2802	2802
q11	625	517	506	506
q12	713	781	640	640
q13	3658	4153	3388	3388
q14	290	305	287	287
q15	552	485	478	478
q16	679	704	645	645
q17	1231	1782	1407	1407
q18	7762	7613	7460	7460
q19	861	865	1150	865
q20	1988	2073	1896	1896
q21	5630	5032	4982	4982
q22	1111	1083	1044	1044
Total cold run time: 55027 ms
Total hot run time: 52434 ms

doris-robot · 2025-04-11T04:26:27Z

TPC-DS: Total hot run time: 193284 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a023e7737c776901c1893fd5eea3c26e314f3df8, data reload: false
query1	1403	1081	1046	1046
query2	6161	1991	1974	1974
query3	11149	4678	4929	4678
query4	25327	23808	23140	23140
query5	4823	629	486	486
query6	316	192	199	192
query7	3994	498	276	276
query8	291	239	225	225
query9	8517	2608	2596	2596
query10	483	330	281	281
query11	15280	15213	14888	14888
query12	161	110	117	110
query13	1561	527	380	380
query14	9079	6288	6125	6125
query15	201	178	170	170
query16	7610	620	471	471
query17	1122	740	604	604
query18	2007	405	313	313
query19	201	195	172	172
query20	122	124	118	118
query21	203	125	117	117
query22	4351	4483	4297	4297
query23	34560	33471	33502	33471
query24	8682	2453	2424	2424
query25	511	480	399	399
query26	732	292	157	157
query27	3050	503	345	345
query28	4790	2499	2424	2424
query29	663	591	473	473
query30	284	226	202	202
query31	915	861	784	784
query32	74	67	65	65
query33	551	374	319	319
query34	819	886	519	519
query35	830	878	786	786
query36	973	1016	905	905
query37	132	103	85	85
query38	4278	4300	4151	4151
query39	1554	1461	1442	1442
query40	225	125	110	110
query41	54	58	55	55
query42	121	110	103	103
query43	524	525	510	510
query44	1368	821	829	821
query45	190	168	169	168
query46	833	1025	639	639
query47	1797	1918	1869	1869
query48	369	409	301	301
query49	756	507	437	437
query50	655	710	405	405
query51	4265	4360	4279	4279
query52	122	106	97	97
query53	236	271	192	192
query54	601	580	515	515
query55	84	81	84	81
query56	322	308	288	288
query57	1134	1193	1091	1091
query58	267	263	274	263
query59	2761	2755	2664	2664
query60	332	332	316	316
query61	134	127	156	127
query62	787	732	671	671
query63	225	191	183	183
query64	3078	1057	739	739
query65	4498	4343	4414	4343
query66	1015	423	309	309
query67	16176	15529	15353	15353
query68	9321	896	506	506
query69	490	297	270	270
query70	1213	1099	1135	1099
query71	491	331	299	299
query72	5315	4825	4905	4825
query73	747	647	349	349
query74	9205	9017	8692	8692
query75	4342	3200	2714	2714
query76	3719	1186	762	762
query77	829	391	279	279
query78	10015	10386	9343	9343
query79	1789	804	562	562
query80	634	514	434	434
query81	485	257	223	223
query82	437	125	100	100
query83	263	258	255	255
query84	243	108	93	93
query85	777	357	314	314
query86	333	297	280	280
query87	4378	4560	4392	4392
query88	3326	2247	2254	2247
query89	406	308	282	282
query90	1921	217	211	211
query91	148	148	113	113
query92	81	63	59	59
query93	1190	952	588	588
query94	662	391	311	311
query95	374	296	286	286
query96	491	568	272	272
query97	3163	3256	3073	3073
query98	231	206	204	204
query99	1458	1414	1308	1308
Total cold run time: 279166 ms
Total hot run time: 193284 ms

doris-robot · 2025-04-11T04:31:45Z

ClickBench: Total hot run time: 31.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a023e7737c776901c1893fd5eea3c26e314f3df8, data reload: false
query1	0.03	0.03	0.03
query2	0.13	0.10	0.10
query3	0.25	0.20	0.20
query4	1.59	0.19	0.20
query5	0.59	0.60	0.60
query6	1.19	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.53	0.51
query10	0.59	0.59	0.57
query11	0.16	0.10	0.10
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	2.81	2.79	2.69
query15	0.92	0.84	0.85
query16	0.38	0.38	0.39
query17	1.05	1.00	1.04
query18	0.20	0.19	0.19
query19	1.90	1.99	1.80
query20	0.01	0.01	0.01
query21	15.35	0.87	0.56
query22	0.74	1.14	0.69
query23	14.98	1.38	0.61
query24	6.89	1.25	0.85
query25	0.52	0.20	0.08
query26	0.58	0.16	0.14
query27	0.05	0.05	0.05
query28	9.10	0.94	0.43
query29	12.54	3.99	3.31
query30	0.24	0.09	0.06
query31	2.84	0.59	0.39
query32	3.23	0.54	0.47
query33	3.08	3.03	3.05
query34	16.17	5.14	4.54
query35	4.53	4.56	4.50
query36	0.66	0.51	0.49
query37	0.09	0.06	0.05
query38	0.06	0.04	0.03
query39	0.02	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.2 s
Total hot run time: 31.18 s

hello-stephen · 2025-04-11T06:36:38Z

BE UT Coverage Report

Increment line coverage 0.00% (0/22) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.46% (14106/26887)
Line Coverage	41.27% (121966/295527)
Region Coverage	40.03% (62114/155182)
Branch Coverage	34.68% (31102/89690)

mymeiyi · 2025-04-11T08:00:41Z

run cloud_p0

mymeiyi · 2025-04-11T11:18:15Z

run cloud_p0

yiguolei · 2025-04-27T09:51:28Z

+ _runtime_filter_timer = std::make_shared<pipeline::RuntimeFilterTimer>(
+ MonotonicMillis(), load_block_queue->get_group_commit_interval_ms(),
+ _timer_dependency);
+ std::vector<std::shared_ptr<pipeline::RuntimeFilterTimer>> timers;


为啥会用到runtime filter？

group commit需要按照提交时间被唤醒，现在只有runtime filter支持这种

这种地方都得用注释写清楚，否则，别人还以为我们的group commit 跟runtime filter 有啥关联

mymeiyi · 2025-05-19T03:23:34Z

run buildall

doris-robot · 2025-05-19T04:08:12Z

TPC-H: Total hot run time: 33561 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18509efc38e4dc855667894f374bce30e7973650, data reload: false
------ Round 1 ----------------------------------
q1	25928	5617	4961	4961
q2	2080	266	187	187
q3	10410	1228	684	684
q4	10220	997	522	522
q5	7694	2383	2318	2318
q6	174	159	130	130
q7	900	737	615	615
q8	9379	1356	1089	1089
q9	7106	5084	5215	5084
q10	6879	2303	1887	1887
q11	505	291	260	260
q12	337	347	221	221
q13	17754	3617	3035	3035
q14	236	224	212	212
q15	529	486	476	476
q16	424	430	372	372
q17	574	851	358	358
q18	7477	7097	7239	7097
q19	1537	925	532	532
q20	321	330	218	218
q21	4139	2509	2327	2327
q22	1050	1027	976	976
Total cold run time: 115653 ms
Total hot run time: 33561 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5080	5032	5046	5032
q2	242	327	226	226
q3	2160	2632	2288	2288
q4	1409	1839	1351	1351
q5	4401	4389	4361	4361
q6	217	166	135	135
q7	2058	1951	1790	1790
q8	2645	2520	2501	2501
q9	7217	7196	7023	7023
q10	3044	3174	2727	2727
q11	589	519	507	507
q12	660	753	627	627
q13	3514	3896	3301	3301
q14	279	301	267	267
q15	543	499	491	491
q16	444	471	425	425
q17	1139	1486	1398	1398
q18	7692	7515	7440	7440
q19	786	768	840	768
q20	1980	2060	1869	1869
q21	4755	4532	4413	4413
q22	1077	1121	1038	1038
Total cold run time: 51931 ms
Total hot run time: 49978 ms

doris-robot · 2025-05-19T04:20:04Z

TPC-DS: Total hot run time: 192651 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18509efc38e4dc855667894f374bce30e7973650, data reload: false
query1	1402	1077	1059	1059
query2	6440	1862	1866	1862
query3	11119	4647	4514	4514
query4	24694	23314	22840	22840
query5	3717	618	463	463
query6	307	218	226	218
query7	3987	486	292	292
query8	308	256	246	246
query9	8519	2640	2650	2640
query10	446	322	283	283
query11	15268	15031	14791	14791
query12	172	109	108	108
query13	1567	542	405	405
query14	9160	6145	6118	6118
query15	248	187	171	171
query16	7592	658	498	498
query17	1195	721	584	584
query18	2026	413	322	322
query19	221	202	171	171
query20	135	127	122	122
query21	210	128	113	113
query22	4488	4653	4484	4484
query23	34511	33653	33807	33653
query24	8417	2456	2424	2424
query25	530	477	385	385
query26	1432	271	152	152
query27	2757	512	346	346
query28	4645	2200	2166	2166
query29	686	576	458	458
query30	273	225	194	194
query31	926	856	782	782
query32	71	61	63	61
query33	556	360	348	348
query34	826	896	541	541
query35	832	815	758	758
query36	1004	1016	881	881
query37	108	98	75	75
query38	4222	4165	4123	4123
query39	1532	1487	1438	1438
query40	203	122	108	108
query41	55	55	52	52
query42	124	111	113	111
query43	521	521	473	473
query44	1344	855	871	855
query45	181	172	168	168
query46	855	1038	662	662
query47	1882	1906	1859	1859
query48	405	430	330	330
query49	751	556	435	435
query50	674	704	406	406
query51	4269	4226	4313	4226
query52	121	108	96	96
query53	221	263	188	188
query54	598	584	537	537
query55	90	90	84	84
query56	323	324	300	300
query57	1218	1219	1164	1164
query58	281	268	265	265
query59	2654	2776	2755	2755
query60	349	329	315	315
query61	134	133	156	133
query62	794	716	678	678
query63	227	188	193	188
query64	3988	1049	715	715
query65	4452	4373	4332	4332
query66	1123	405	316	316
query67	16094	15532	15451	15451
query68	8259	884	553	553
query69	479	311	267	267
query70	1226	1145	1088	1088
query71	458	323	297	297
query72	5709	4763	4641	4641
query73	684	596	363	363
query74	9173	9139	8928	8928
query75	3899	3204	2664	2664
query76	3765	1188	738	738
query77	786	368	280	280
query78	10110	10171	9388	9388
query79	2303	816	582	582
query80	572	603	458	458
query81	490	253	213	213
query82	461	129	92	92
query83	248	244	235	235
query84	249	100	83	83
query85	802	367	313	313
query86	375	298	279	279
query87	4374	4417	4278	4278
query88	3831	2396	2358	2358
query89	418	314	283	283
query90	1853	211	207	207
query91	143	158	112	112
query92	75	62	53	53
query93	1789	954	613	613
query94	654	427	321	321
query95	365	285	271	271
query96	493	560	284	284
query97	2713	2822	2656	2656
query98	231	210	211	210
query99	1361	1373	1252	1252
Total cold run time: 278828 ms
Total hot run time: 192651 ms

doris-robot · 2025-05-19T04:25:14Z

ClickBench: Total hot run time: 28.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18509efc38e4dc855667894f374bce30e7973650, data reload: false
query1	0.03	0.03	0.03
query2	0.13	0.10	0.11
query3	0.25	0.19	0.19
query4	1.60	0.19	0.10
query5	0.45	0.40	0.43
query6	1.15	0.66	0.67
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.56	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	0.79	0.81	0.81
query15	0.87	0.85	0.85
query16	0.39	0.38	0.40
query17	1.06	1.04	1.02
query18	0.22	0.21	0.20
query19	1.95	1.82	1.84
query20	0.02	0.01	0.01
query21	15.42	0.87	0.53
query22	0.76	1.21	0.63
query23	14.94	1.39	0.61
query24	6.65	1.06	0.51
query25	0.52	0.25	0.06
query26	0.55	0.16	0.15
query27	0.04	0.05	0.06
query28	9.24	0.91	0.44
query29	12.53	3.98	3.36
query30	0.26	0.10	0.07
query31	2.81	0.58	0.38
query32	3.23	0.55	0.46
query33	3.09	3.10	3.07
query34	15.84	5.12	4.55
query35	4.57	4.56	4.60
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 102.64 s
Total hot run time: 28.74 s

hello-stephen · 2025-05-19T05:02:28Z

BE UT Coverage Report

Increment line coverage 0.00% (0/22) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.81% (14886/26671)
Line Coverage	44.60% (131908/295739)
Region Coverage	43.73% (66413/151859)
Branch Coverage	38.33% (34025/88762)

hello-stephen · 2025-05-19T06:06:32Z

BE Regression && UT Coverage Report

Increment line coverage 95.45% (21/22) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.39% (20845/26258)
Line Coverage	72.62% (214775/295750)
Region Coverage	70.82% (126361/178417)
Branch Coverage	64.60% (65501/101398)

hello-stephen · 2025-05-19T12:12:45Z

BE Regression && UT Coverage Report

Increment line coverage 4.55% (1/22) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.00% (19420/26242)
Line Coverage	62.59% (185076/295710)
Region Coverage	60.62% (108141/178401)
Branch Coverage	53.12% (53867/101398)

github-actions · 2025-05-20T02:05:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-20T02:05:23Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

…block #49822 (#51079) pick #49822

…mit get_block apache#49822 (apache#51079)" This reverts commit 212cd8f.

…roup_commit get_block apache#49822 (apache#51079)"" This reverts commit e79a574.

### What problem does this PR solve? 1. after #49822, the group_commit_scan_operator is blocked by time dependency until timeout, this pr fix it 2. modify some debug log 3. print the create plan error to client

1. after apache#49822, the group_commit_scan_operator is blocked by time dependency until timeout, this pr fix it 2. modify some debug log 3. print the create plan error to client

…e#49822)

### What problem does this PR solve? 1. after apache#49822, the group_commit_scan_operator is blocked by time dependency until timeout, this pr fix it 2. modify some debug log 3. print the create plan error to client

mymeiyi force-pushed the group-commit-get-block branch 2 times, most recently from a2f90ac to 3da47f3Compare April 7, 2025 03:25

mymeiyi force-pushed the group-commit-get-block branch from 3da47f3 to 5103a6eCompare April 7, 2025 07:35

Gabriel39 reviewed Apr 7, 2025
View reviewed changes

Comment threadbe/src/pipeline/dependency.h

mymeiyi force-pushed the group-commit-get-block branch 2 times, most recently from ba2cbcb to a023e77Compare April 11, 2025 03:25

yiguolei reviewed Apr 27, 2025
View reviewed changes

[fix](group commit) reduce cpu cost for group_commit get_block
18509ef

mymeiyi force-pushed the group-commit-get-block branch from a023e77 to 18509efCompare May 19, 2025 03:23

yiguolei approved these changes May 20, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 20, 2025

github-actionsBot added the reviewed label May 20, 2025

dataroaring approved these changes May 20, 2025
View reviewed changes

dataroaring added dev/3.0.x dev/2.1.x usercase Important user case type label and removed dev/2.1.x labels May 20, 2025

dataroaring merged commit 427b917 into apache:masterMay 20, 2025

github-actionsBot pushed a commit that referenced this pull request May 20, 2025
[fix](group commit) reduce cpu cost for group_commit get_block (#49822)
1e18002

github-actionsBot mentioned this pull request May 20, 2025
branch-3.0: [fix](group commit) reduce cpu cost for group_commit get_block #49822 #51069
Closed

mymeiyi mentioned this pull request May 20, 2025
branch-3.0: [fix](group commit) reduce cpu cost for group_commit get_block #49822 #51079
Merged

dataroaring pushed a commit that referenced this pull request May 21, 2025
branch-3.0: [fix](group commit) reduce cpu cost for group_commit get_…
212cd8f
…block #49822 (#51079) pick #49822

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 21, 2025

mymeiyi added a commit to mymeiyi/doris that referenced this pull request May 28, 2025
Revert "branch-3.0: [fix](group commit) reduce cpu cost for group_com…
e79a574
…mit get_block apache#49822 (apache#51079)" This reverts commit 212cd8f.

mymeiyi added a commit to mymeiyi/doris that referenced this pull request May 28, 2025
Revert "Revert "branch-3.0: [fix](group commit) reduce cpu cost for g…
dc340a6
…roup_commit get_block apache#49822 (apache#51079)"" This reverts commit e79a574.

mymeiyi mentioned this pull request May 29, 2025
[fix](group commit) fix group_commit get_block too slow #51358
Merged
16 tasks

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
[fix](group commit) reduce cpu cost for group_commit get_block (apach…
8fa69f0
…e#49822)

dataroaring mentioned this pull request Sep 9, 2025
[Bug] [stream-load] Doris 3.0.5 stream load cpu usage highly than Doris 2.1.6 #51103
Closed
3 tasks

Uh oh!

Conversation

mymeiyi commented Apr 7, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 7, 2025

Uh oh!

mymeiyi commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

mymeiyi commented Apr 7, 2025

Uh oh!

hello-stephen commented Apr 7, 2025

BE UT Coverage Report

Uh oh!

mymeiyi commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

Uh oh!

doris-robot commented Apr 7, 2025

BE UT Coverage Report

Uh oh!

mymeiyi commented Apr 11, 2025

Uh oh!

doris-robot commented Apr 11, 2025

Uh oh!

doris-robot commented Apr 11, 2025

Uh oh!

doris-robot commented Apr 11, 2025

Uh oh!

hello-stephen commented Apr 11, 2025

BE UT Coverage Report

Uh oh!

mymeiyi commented Apr 11, 2025

Uh oh!

mymeiyi commented Apr 11, 2025

Uh oh!

yiguoleiApr 27, 2025

Choose a reason for hiding this comment

Uh oh!

mymeiyiMay 12, 2025

Choose a reason for hiding this comment

Uh oh!

yiguoleiMay 20, 2025

Choose a reason for hiding this comment

Uh oh!

mymeiyi commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

hello-stephen commented May 19, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 19, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented May 19, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented May 20, 2025

Uh oh!

github-actionsBot commented May 20, 2025

Uh oh!