Branch-3.0 [opt](nereids) avoid printing duplicated table stats in profile #51354 by englefly · Pull Request #51658 · apache/doris

englefly · 2025-06-12T02:43:51Z

What problem does this PR solve?

pick #51354
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-06-12T02:44:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

englefly · 2025-06-12T02:44:32Z

run buildall

englefly · 2025-07-07T12:42:03Z

run buildall

englefly · 2025-07-07T23:46:39Z

run buildall

doris-robot · 2025-07-08T00:24:48Z

TPC-H: Total hot run time: 39854 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dbaebbac61fac22e3dbe02198a348249253f100d, data reload: false
------ Round 1 ----------------------------------
q1	17597	6814	6731	6731
q2	2088	193	180	180
q3	10494	1135	1171	1135
q4	10215	774	750	750
q5	7771	2886	2852	2852
q6	209	132	132	132
q7	977	618	607	607
q8	9351	1949	2040	1949
q9	6648	6347	6440	6347
q10	7044	2273	2309	2273
q11	457	257	255	255
q12	396	212	215	212
q13	17768	2974	2990	2974
q14	241	199	206	199
q15	510	503	455	455
q16	519	385	375	375
q17	988	531	612	531
q18	7200	6510	6591	6510
q19	1393	1052	1172	1052
q20	475	198	207	198
q21	3868	3157	3158	3157
q22	1130	980	997	980
Total cold run time: 107339 ms
Total hot run time: 39854 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6595	6602	6636	6602
q2	341	240	239	239
q3	2968	2957	2934	2934
q4	2112	1892	1880	1880
q5	5865	5857	5831	5831
q6	210	134	134	134
q7	2309	1857	1913	1857
q8	3490	3654	3617	3617
q9	8980	8985	8887	8887
q10	3529	3526	3505	3505
q11	592	501	505	501
q12	819	604	617	604
q13	8827	3216	3182	3182
q14	308	270	269	269
q15	532	468	467	467
q16	474	453	430	430
q17	1869	1634	1637	1634
q18	8202	7837	7862	7837
q19	1681	1514	1582	1514
q20	2068	1823	1853	1823
q21	5310	5057	5010	5010
q22	1132	1053	1021	1021
Total cold run time: 68213 ms
Total hot run time: 59778 ms

…he#51354) ### What problem does this PR solve? Query related tables are collected by function org.apache.doris.common.profile.Profile#updateSummary(). This function is invoked multiple times during a query execution. And hence, tables are added to Profile.physicalRelations multiple times. In this pr, Profile.physicalRelations is defined as Set instead of List to remove duplications. (cherry picked from commit 8e41d4d)

doris-robot · 2025-07-08T00:37:08Z

TPC-DS: Total hot run time: 197013 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dbaebbac61fac22e3dbe02198a348249253f100d, data reload: false
query1	1294	919	911	911
query2	6264	1858	1885	1858
query3	10883	4498	4580	4498
query4	33347	23548	23777	23548
query5	3527	477	446	446
query6	256	173	173	173
query7	3985	306	315	306
query8	275	227	222	222
query9	9584	2598	2547	2547
query10	450	273	273	273
query11	18139	15118	15147	15118
query12	156	103	105	103
query13	1559	440	418	418
query14	9444	7045	6996	6996
query15	256	181	198	181
query16	8070	496	487	487
query17	1628	613	592	592
query18	2110	325	313	313
query19	209	170	167	167
query20	128	114	115	114
query21	206	112	107	107
query22	4620	4456	4349	4349
query23	35173	34236	34402	34236
query24	11271	2937	2915	2915
query25	677	424	417	417
query26	1370	177	170	170
query27	3148	365	357	357
query28	7423	2187	2173	2173
query29	917	465	489	465
query30	267	165	167	165
query31	1044	867	828	828
query32	93	53	53	53
query33	775	308	314	308
query34	1035	516	540	516
query35	865	721	734	721
query36	1118	970	965	965
query37	128	61	67	61
query38	4156	3937	3994	3937
query39	1517	1477	1470	1470
query40	253	102	104	102
query41	52	52	50	50
query42	117	106	102	102
query43	504	492	495	492
query44	1317	829	824	824
query45	191	170	164	164
query46	1148	719	692	692
query47	2000	1880	1921	1880
query48	495	381	392	381
query49	909	412	391	391
query50	834	440	435	435
query51	7484	7217	7361	7217
query52	103	91	95	91
query53	260	183	196	183
query54	1170	469	488	469
query55	82	81	78	78
query56	303	246	244	244
query57	1314	1236	1180	1180
query58	227	217	217	217
query59	3252	2995	3119	2995
query60	297	273	278	273
query61	116	124	114	114
query62	859	714	694	694
query63	221	193	195	193
query64	5135	706	700	700
query65	3262	3169	3197	3169
query66	978	292	298	292
query67	15853	15455	15522	15455
query68	4597	569	558	558
query69	465	275	272	272
query70	1138	1037	1101	1037
query71	344	261	252	252
query72	6186	4085	4022	4022
query73	744	343	354	343
query74	10325	9005	9129	9005
query75	3346	2613	2668	2613
query76	2719	1099	1155	1099
query77	367	276	272	272
query78	10568	9610	9615	9610
query79	2323	593	614	593
query80	955	417	430	417
query81	547	218	218	218
query82	878	93	89	89
query83	258	143	141	141
query84	236	79	90	79
query85	1566	320	285	285
query86	456	303	284	284
query87	4427	4267	4292	4267
query88	4364	2343	2337	2337
query89	426	292	295	292
query90	1986	186	190	186
query91	179	151	147	147
query92	65	51	48	48
query93	2356	544	548	544
query94	836	298	300	298
query95	364	253	254	253
query96	610	280	281	280
query97	3314	3169	3114	3114
query98	224	206	202	202
query99	1513	1295	1304	1295
Total cold run time: 304785 ms
Total hot run time: 197013 ms

englefly · 2025-07-08T00:38:45Z

run buildall

doris-robot · 2025-07-08T00:57:55Z

TPC-H: Total hot run time: 39701 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bda34ac77d06481a8ae547b6ebf2e22065746a1e, data reload: false
------ Round 1 ----------------------------------
q1	17580	6722	6607	6607
q2	2060	229	185	185
q3	10466	1123	1171	1123
q4	10217	722	704	704
q5	7740	3133	2874	2874
q6	218	132	131	131
q7	970	617	612	612
q8	9344	1934	2038	1934
q9	6647	6395	6408	6395
q10	6980	2266	2283	2266
q11	467	275	259	259
q12	399	215	217	215
q13	17781	2972	3009	2972
q14	247	202	207	202
q15	523	491	454	454
q16	506	398	373	373
q17	980	543	558	543
q18	7440	6561	6756	6561
q19	1403	1087	1049	1049
q20	464	197	207	197
q21	4096	3209	3039	3039
q22	1149	1006	1045	1006
Total cold run time: 107677 ms
Total hot run time: 39701 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6674	6547	6619	6547
q2	324	233	229	229
q3	2908	2795	2793	2793
q4	1902	1755	1653	1653
q5	5403	5460	5419	5419
q6	199	127	124	124
q7	2112	1699	1680	1680
q8	3235	3395	3427	3395
q9	8541	8461	8525	8461
q10	3492	3411	3406	3406
q11	591	512	478	478
q12	778	553	591	553
q13	8277	3001	3013	3001
q14	280	258	248	248
q15	504	455	456	455
q16	467	443	422	422
q17	1794	1580	1566	1566
q18	7834	7366	7224	7224
q19	1636	1517	1447	1447
q20	2044	1814	1803	1803
q21	5024	4853	4834	4834
q22	1082	1043	998	998
Total cold run time: 65101 ms
Total hot run time: 56736 ms

doris-robot · 2025-07-08T01:10:00Z

TPC-DS: Total hot run time: 189789 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bda34ac77d06481a8ae547b6ebf2e22065746a1e, data reload: false
query1	992	370	354	354
query2	6515	1878	1852	1852
query3	6706	214	217	214
query4	34131	23678	23328	23328
query5	4681	440	443	440
query6	279	167	194	167
query7	4622	306	309	306
query8	269	217	220	217
query9	9675	2553	2546	2546
query10	496	270	264	264
query11	18235	15224	15162	15162
query12	154	102	106	102
query13	1634	426	413	413
query14	9444	7345	6801	6801
query15	257	174	183	174
query16	8053	481	502	481
query17	1636	589	553	553
query18	2139	316	314	314
query19	372	164	163	163
query20	114	108	108	108
query21	210	106	108	106
query22	4444	4172	4234	4172
query23	34660	33587	33484	33484
query24	11233	2901	2846	2846
query25	707	415	420	415
query26	1445	177	170	170
query27	2953	347	355	347
query28	7919	2119	2108	2108
query29	956	464	445	445
query30	323	164	163	163
query31	1017	799	776	776
query32	103	59	60	59
query33	804	305	306	305
query34	936	510	523	510
query35	875	714	745	714
query36	1104	958	949	949
query37	139	71	71	71
query38	4021	3803	3791	3791
query39	1516	1427	1443	1427
query40	296	109	103	103
query41	62	58	56	56
query42	117	102	104	102
query43	518	468	488	468
query44	1337	798	795	795
query45	186	171	171	171
query46	1161	757	702	702
query47	1925	1810	1819	1810
query48	478	381	383	381
query49	1202	431	414	414
query50	824	415	420	415
query51	7338	7119	7160	7119
query52	107	96	92	92
query53	266	195	187	187
query54	1249	478	468	468
query55	83	77	80	77
query56	275	274	266	266
query57	1292	1191	1146	1146
query58	248	216	233	216
query59	3031	2958	2783	2783
query60	324	260	262	260
query61	148	111	153	111
query62	888	679	667	667
query63	218	183	195	183
query64	5348	656	636	636
query65	3273	3206	3218	3206
query66	1442	326	309	309
query67	15918	15839	15459	15459
query68	4707	566	565	565
query69	458	268	263	263
query70	1144	1093	1120	1093
query71	338	253	259	253
query72	6378	4117	4107	4107
query73	767	349	355	349
query74	10320	9177	8910	8910
query75	3425	2641	2654	2641
query76	2959	1166	1013	1013
query77	433	271	278	271
query78	10458	9615	9559	9559
query79	2509	594	614	594
query80	1121	451	429	429
query81	534	224	223	223
query82	775	92	90	90
query83	228	147	152	147
query84	248	79	86	79
query85	1327	307	295	295
query86	422	286	311	286
query87	4445	4226	4237	4226
query88	4080	2331	2317	2317
query89	404	298	298	298
query90	2040	188	191	188
query91	185	147	154	147
query92	68	53	53	53
query93	1518	558	544	544
query94	910	310	297	297
query95	361	263	268	263
query96	606	276	277	276
query97	3280	3109	3137	3109
query98	214	203	189	189
query99	1533	1314	1296	1296
Total cold run time: 303518 ms
Total hot run time: 189789 ms

doris-robot · 2025-07-08T01:15:24Z

ClickBench: Total hot run time: 29.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bda34ac77d06481a8ae547b6ebf2e22065746a1e, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.22	0.07	0.06
query4	1.62	0.10	0.10
query5	0.54	0.51	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.03
query8	0.04	0.03	0.04
query9	0.56	0.51	0.50
query10	0.56	0.54	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	0.78	0.79	0.80
query15	0.86	0.82	0.84
query16	0.40	0.38	0.38
query17	1.02	1.04	0.97
query18	0.24	0.21	0.22
query19	1.90	1.88	1.78
query20	0.01	0.01	0.01
query21	15.39	0.61	0.58
query22	2.06	1.77	2.46
query23	16.98	1.01	0.93
query24	2.95	1.55	0.47
query25	0.22	0.14	0.06
query26	0.48	0.14	0.15
query27	0.05	0.06	0.03
query28	10.47	0.50	0.48
query29	12.61	3.22	3.22
query30	0.24	0.06	0.06
query31	2.85	0.39	0.40
query32	3.23	0.47	0.45
query33	2.94	3.00	3.04
query34	17.12	4.54	4.53
query35	4.58	4.53	4.56
query36	0.67	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.19 s
Total hot run time: 29.72 s

englefly · 2025-07-08T02:17:10Z

run buildall

doris-robot · 2025-07-08T03:13:50Z

TPC-H: Total hot run time: 39708 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bda34ac77d06481a8ae547b6ebf2e22065746a1e, data reload: false
------ Round 1 ----------------------------------
q1	17597	6735	6643	6643
q2	2077	193	192	192
q3	10557	1157	1177	1157
q4	10319	728	743	728
q5	7747	2892	2773	2773
q6	213	138	129	129
q7	976	616	603	603
q8	9366	1955	2061	1955
q9	6616	6377	6421	6377
q10	7038	2253	2285	2253
q11	467	259	267	259
q12	401	207	214	207
q13	17786	3009	2986	2986
q14	242	206	220	206
q15	507	472	478	472
q16	505	389	374	374
q17	977	603	539	539
q18	7224	6719	6534	6534
q19	1402	1113	1110	1110
q20	473	196	202	196
q21	3874	3025	3125	3025
q22	1071	994	990	990
Total cold run time: 107435 ms
Total hot run time: 39708 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6547	6555	6584	6555
q2	332	244	228	228
q3	2911	2855	2958	2855
q4	1992	1818	1773	1773
q5	5672	5700	5726	5700
q6	215	128	127	127
q7	2232	1802	1805	1802
q8	3371	3574	3517	3517
q9	8810	8888	8909	8888
q10	3761	3492	3517	3492
q11	588	507	505	505
q12	793	602	604	602
q13	10141	3129	3197	3129
q14	290	284	265	265
q15	512	472	465	465
q16	494	457	472	457
q17	1849	1617	1610	1610
q18	8367	7787	7708	7708
q19	1691	1567	1620	1567
q20	2096	1851	1802	1802
q21	5200	5015	4962	4962
q22	1111	1053	1010	1010
Total cold run time: 68975 ms
Total hot run time: 59019 ms

doris-robot · 2025-07-08T03:26:07Z

TPC-DS: Total hot run time: 196583 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bda34ac77d06481a8ae547b6ebf2e22065746a1e, data reload: false
query1	1301	905	897	897
query2	6186	1879	1865	1865
query3	10863	4319	4404	4319
query4	33321	24122	23280	23280
query5	4911	463	455	455
query6	280	187	173	173
query7	3992	318	321	318
query8	280	219	219	219
query9	9572	2574	2567	2567
query10	505	277	253	253
query11	18000	14993	15163	14993
query12	149	104	104	104
query13	1541	418	416	416
query14	9383	7368	6789	6789
query15	278	205	201	201
query16	8028	475	484	475
query17	1557	602	590	590
query18	2173	330	314	314
query19	243	163	168	163
query20	122	117	113	113
query21	209	104	120	104
query22	4740	4556	4504	4504
query23	34582	34612	34386	34386
query24	10957	2939	2913	2913
query25	630	429	414	414
query26	768	174	175	174
query27	2330	356	360	356
query28	6464	2137	2147	2137
query29	789	468	460	460
query30	256	157	160	157
query31	1039	807	866	807
query32	97	58	58	58
query33	800	293	296	293
query34	928	530	519	519
query35	908	764	723	723
query36	1125	938	992	938
query37	110	66	68	66
query38	4072	3944	3943	3943
query39	1568	1459	1483	1459
query40	200	103	101	101
query41	51	49	49	49
query42	119	105	107	105
query43	535	476	475	475
query44	1251	809	815	809
query45	190	169	169	169
query46	1163	730	743	730
query47	2022	1925	2001	1925
query48	486	394	390	390
query49	853	400	380	380
query50	858	426	426	426
query51	7473	7182	7303	7182
query52	104	92	89	89
query53	262	183	184	183
query54	1173	466	476	466
query55	83	84	80	80
query56	261	253	241	241
query57	1299	1247	1216	1216
query58	238	212	212	212
query59	3241	3046	3040	3040
query60	285	253	269	253
query61	109	107	110	107
query62	873	710	703	703
query63	236	201	194	194
query64	3895	682	641	641
query65	3371	3274	3284	3274
query66	793	307	311	307
query67	16174	15714	15385	15385
query68	4434	594	603	594
query69	433	265	261	261
query70	1152	1065	1130	1065
query71	379	259	256	256
query72	6400	3910	4006	3910
query73	755	347	350	347
query74	10163	9289	9210	9210
query75	3420	2619	2649	2619
query76	2757	1054	1084	1054
query77	380	290	267	267
query78	10581	9622	9633	9622
query79	2294	630	610	610
query80	1023	429	422	422
query81	569	225	221	221
query82	623	85	86	85
query83	252	140	142	140
query84	240	86	76	76
query85	1826	296	295	295
query86	490	294	279	279
query87	4387	4241	4278	4241
query88	4354	2382	2331	2331
query89	417	287	288	287
query90	1903	186	184	184
query91	186	144	148	144
query92	63	48	51	48
query93	2513	569	563	563
query94	837	280	306	280
query95	359	262	258	258
query96	607	281	291	281
query97	3311	3171	3134	3134
query98	220	211	205	205
query99	1551	1295	1247	1247
Total cold run time: 301534 ms
Total hot run time: 196583 ms

doris-robot · 2025-07-08T03:31:30Z

ClickBench: Total hot run time: 30.24 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bda34ac77d06481a8ae547b6ebf2e22065746a1e, data reload: false
query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.51	0.50	0.51
query6	1.13	0.73	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.50	0.51
query10	0.54	0.54	0.56
query11	0.15	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.59	0.60
query14	0.77	0.80	0.80
query15	0.84	0.83	0.81
query16	0.39	0.38	0.39
query17	0.98	1.05	1.00
query18	0.23	0.22	0.22
query19	1.85	1.81	1.76
query20	0.01	0.02	0.01
query21	15.42	0.56	0.54
query22	2.68	1.77	1.65
query23	16.90	0.95	0.91
query24	3.30	1.18	2.00
query25	0.15	0.14	0.07
query26	0.67	0.13	0.14
query27	0.04	0.04	0.05
query28	9.22	0.54	0.49
query29	12.57	3.24	3.21
query30	0.25	0.06	0.06
query31	2.85	0.39	0.38
query32	3.25	0.47	0.45
query33	3.00	3.04	3.01
query34	16.90	4.48	4.47
query35	4.49	4.55	4.53
query36	0.68	0.48	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.61 s
Total hot run time: 30.24 s

englefly · 2025-07-08T08:20:48Z

run p0

englefly · 2025-07-08T08:20:55Z

run cloud_p0

englefly · 2025-07-08T23:59:58Z

run p0

englefly · 2025-07-09T00:00:04Z

run cloud_p0

dataroaring

LGTM

englefly requested review from BiteTheDDDDt, CalvinKirs, dataroaring, gavinchou, morningman, w41ter, xiaokang and yiguolei as code owners June 12, 2025 02:43

englefly changed the base branch from master to branch-3.0June 12, 2025 02:44

englefly closed this Jun 26, 2025

englefly reopened this Jul 7, 2025

englefly added 2 commits July 8, 2025 08:36

fmt
f8b15cb

englefly force-pushed the br-3.0-pick51354 branch from dbaebba to f8b15cbCompare July 8, 2025 00:36

0708
bda34ac

dataroaring approved these changes Jul 15, 2025
View reviewed changes

dataroaring merged commit 53e5c9f into apache:branch-3.0Jul 15, 2025

Uh oh!

Conversation

englefly commented Jun 12, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jun 12, 2025

Uh oh!

englefly commented Jun 12, 2025

Uh oh!

englefly commented Jul 7, 2025

Uh oh!

englefly commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

englefly commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

englefly commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

englefly commented Jul 8, 2025

Uh oh!

englefly commented Jul 8, 2025

Uh oh!

englefly commented Jul 8, 2025

Uh oh!

englefly commented Jul 9, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants