[feat](nereids)push Limit to local agg by englefly · Pull Request #34853 · apache/doris

englefly · 2024-05-14T11:45:57Z

Proposed changes

for a pattern: topn(n)->globalAgg->localAgg
this pr tries to add a filter on global/localAgg which means only the first n tuples are counted, and others could be ignored.

inorder to obtain this benefit, optimizer will change limit node to topn node if the limit number is less than topnOptLimitThreshold.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-14T11:46:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

englefly · 2024-05-14T11:46:05Z

run buildall

github-actions · 2024-05-14T11:51:42Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-05-15T08:35:56Z

clang-tidy review says "All clean, LGTM! 👍"

morrySnow · 2024-05-15T08:49:06Z

add description please

github-actions · 2024-05-15T10:37:05Z

clang-tidy review says "All clean, LGTM! 👍"

morrySnow · 2024-05-21T12:26:08Z

add description

englefly · 2024-05-22T01:37:48Z

run buildall

github-actions · 2024-05-22T01:42:09Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-22T03:01:59Z

TPC-H: Total hot run time: 42152 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9d6017c9dbf124d3af8c1cb73e23ad03eae6a3a, data reload: false
------ Round 1 ----------------------------------
q1	17606	5006	4316	4316
q2	2023	196	195	195
q3	10444	1163	1215	1163
q4	10194	861	835	835
q5	7461	2699	2707	2699
q6	222	134	134	134
q7	1043	602	610	602
q8	9246	2193	2105	2105
q9	9147	6760	6757	6757
q10	9178	3943	3882	3882
q11	459	242	251	242
q12	458	222	229	222
q13	17252	3210	3237	3210
q14	285	234	219	219
q15	528	467	468	467
q16	517	380	398	380
q17	989	669	649	649
q18	8569	7852	7999	7852
q19	6185	1585	1493	1493
q20	644	327	339	327
q21	5252	4185	4128	4128
q22	354	291	275	275
Total cold run time: 118056 ms
Total hot run time: 42152 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4545	4465	4439	4439
q2	380	281	266	266
q3	3174	2927	2755	2755
q4	1868	1607	1688	1607
q5	5536	5535	5498	5498
q6	212	121	128	121
q7	2336	1962	1978	1962
q8	3293	3426	3505	3426
q9	8713	8851	8642	8642
q10	3926	3857	3845	3845
q11	613	494	496	494
q12	812	617	603	603
q13	15955	3115	3197	3115
q14	300	284	270	270
q15	530	497	493	493
q16	507	418	429	418
q17	1787	1515	1466	1466
q18	7769	7650	7633	7633
q19	1667	1586	1515	1515
q20	2024	1789	1776	1776
q21	11576	4809	4951	4809
q22	563	497	460	460
Total cold run time: 78086 ms
Total hot run time: 55613 ms

doris-robot · 2024-05-22T03:13:02Z

TPC-DS: Total hot run time: 181914 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9d6017c9dbf124d3af8c1cb73e23ad03eae6a3a, data reload: false
query1	913	394	385	385
query2	6434	2308	2437	2308
query3	6654	205	205	205
query4	23729	21263	21312	21263
query5	4140	420	416	416
query6	273	177	176	176
query7	4580	288	286	286
query8	243	192	195	192
query9	8535	2401	2385	2385
query10	424	249	246	246
query11	14660	14211	14159	14159
query12	133	89	87	87
query13	1637	380	375	375
query14	9670	8529	7722	7722
query15	226	184	180	180
query16	7133	260	256	256
query17	1032	549	548	548
query18	1915	295	269	269
query19	200	153	150	150
query20	93	104	85	85
query21	194	132	127	127
query22	5045	4846	4818	4818
query23	34474	33661	33429	33429
query24	5028	3018	2884	2884
query25	468	402	370	370
query26	692	155	155	155
query27	1845	319	332	319
query28	3844	2043	2044	2043
query29	843	610	592	592
query30	251	183	178	178
query31	920	769	739	739
query32	59	51	60	51
query33	379	249	241	241
query34	865	462	479	462
query35	770	721	665	665
query36	1043	950	899	899
query37	112	71	74	71
query38	2866	2777	2777	2777
query39	1620	1564	1568	1564
query40	198	134	127	127
query41	57	45	48	45
query42	106	95	101	95
query43	601	568	546	546
query44	1063	725	735	725
query45	269	257	244	244
query46	1052	729	734	729
query47	1938	1905	1895	1895
query48	375	309	297	297
query49	780	401	396	396
query50	768	393	375	375
query51	6880	6823	6782	6782
query52	108	90	88	88
query53	354	280	295	280
query54	558	429	432	429
query55	73	70	74	70
query56	235	225	225	225
query57	1231	1138	1147	1138
query58	226	200	200	200
query59	3732	3134	3396	3134
query60	250	236	235	235
query61	93	92	87	87
query62	577	460	483	460
query63	308	283	282	282
query64	2614	1840	1806	1806
query65	3162	3128	3125	3125
query66	688	355	350	350
query67	15248	14660	14844	14660
query68	4550	549	545	545
query69	483	307	310	307
query70	1173	1113	1147	1113
query71	430	274	273	273
query72	7234	2704	2515	2515
query73	723	317	318	317
query74	6647	6257	6204	6204
query75	3342	2644	2605	2605
query76	2334	1053	908	908
query77	449	273	277	273
query78	10735	10249	10287	10249
query79	2301	520	522	520
query80	873	446	445	445
query81	533	240	247	240
query82	809	101	102	101
query83	255	173	177	173
query84	267	93	88	88
query85	1151	326	313	313
query86	464	312	304	304
query87	3312	3129	3176	3129
query88	4067	2335	2333	2333
query89	477	383	387	383
query90	2073	196	191	191
query91	136	107	110	107
query92	66	51	50	50
query93	2045	511	500	500
query94	1133	188	188	188
query95	531	307	335	307
query96	592	263	268	263
query97	3203	2964	2972	2964
query98	243	228	219	219
query99	1225	898	885	885
Total cold run time: 261342 ms
Total hot run time: 181914 ms

doris-robot · 2024-05-22T03:18:31Z

ClickBench: Total hot run time: 30.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9d6017c9dbf124d3af8c1cb73e23ad03eae6a3a, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.08
query5	0.50	0.48	0.50
query6	1.12	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.50	0.49
query10	0.55	0.57	0.55
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.60	0.59	0.59
query14	0.76	0.77	0.78
query15	0.82	0.81	0.81
query16	0.36	0.37	0.36
query17	0.95	1.04	0.98
query18	0.22	0.25	0.24
query19	1.84	1.72	1.71
query20	0.01	0.01	0.01
query21	15.49	0.68	0.66
query22	4.09	7.55	1.85
query23	18.25	1.38	1.34
query24	1.67	0.28	0.21
query25	0.15	0.09	0.09
query26	0.26	0.17	0.16
query27	0.08	0.08	0.08
query28	13.34	1.02	1.00
query29	12.74	3.30	3.26
query30	0.24	0.06	0.06
query31	2.86	0.38	0.38
query32	3.29	0.47	0.48
query33	2.85	2.94	2.90
query34	17.03	4.43	4.41
query35	4.53	4.53	4.54
query36	0.68	0.49	0.46
query37	0.18	0.16	0.16
query38	0.17	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.93 s
Total hot run time: 30.66 s

github-actions · 2024-05-22T06:41:49Z

clang-tidy review says "All clean, LGTM! 👍"

englefly · 2024-05-22T08:56:10Z

run buildall

github-actions · 2024-05-22T09:01:07Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-22T10:01:07Z

TPC-H: Total hot run time: 40308 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c1ead2da8d951d6396d7c5af18194600ae996f3, data reload: false
------ Round 1 ----------------------------------
q1	17596	4919	4265	4265
q2	2018	188	204	188
q3	10551	1233	1199	1199
q4	10394	838	754	754
q5	7557	2753	2618	2618
q6	222	138	151	138
q7	973	618	631	618
q8	9474	2139	2092	2092
q9	9103	6646	6622	6622
q10	9430	3762	3714	3714
q11	461	252	245	245
q12	483	231	225	225
q13	18829	3012	3041	3012
q14	252	228	229	228
q15	503	472	472	472
q16	547	384	387	384
q17	995	622	749	622
q18	8160	7504	7486	7486
q19	4501	1572	1458	1458
q20	667	317	315	315
q21	5028	3899	3372	3372
q22	348	281	282	281
Total cold run time: 118092 ms
Total hot run time: 40308 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4364	4254	4229	4229
q2	374	263	272	263
q3	2970	2800	2692	2692
q4	1884	1565	1557	1557
q5	5288	5301	5296	5296
q6	215	128	130	128
q7	2168	1691	1783	1691
q8	3222	3322	3329	3322
q9	8400	8312	8328	8312
q10	3873	3699	3684	3684
q11	587	497	504	497
q12	763	599	628	599
q13	17179	3040	3036	3036
q14	281	253	270	253
q15	521	475	482	475
q16	476	422	412	412
q17	1772	1501	1479	1479
q18	7882	7538	7510	7510
q19	1661	1562	1581	1562
q20	1984	1771	1801	1771
q21	4888	4815	4728	4728
q22	571	508	478	478
Total cold run time: 71323 ms
Total hot run time: 53974 ms

doris-robot · 2024-05-22T10:12:13Z

TPC-DS: Total hot run time: 171080 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c1ead2da8d951d6396d7c5af18194600ae996f3, data reload: false
query1	920	387	375	375
query2	6450	2487	2351	2351
query3	6649	211	205	205
query4	20028	17376	17353	17353
query5	4176	423	426	423
query6	252	169	170	169
query7	4601	300	291	291
query8	248	203	206	203
query9	8363	2393	2372	2372
query10	454	296	272	272
query11	10598	10257	10109	10109
query12	139	89	91	89
query13	1645	368	368	368
query14	8528	7307	7755	7307
query15	262	168	170	168
query16	8176	268	266	266
query17	1828	550	539	539
query18	2118	281	277	277
query19	283	165	163	163
query20	93	88	84	84
query21	197	131	132	131
query22	4384	4157	3898	3898
query23	33770	33087	34008	33087
query24	11733	2886	2830	2830
query25	547	351	362	351
query26	1269	151	150	150
query27	2901	293	295	293
query28	7517	2037	2049	2037
query29	858	603	631	603
query30	319	173	176	173
query31	968	751	748	748
query32	87	53	56	53
query33	773	273	259	259
query34	1054	473	481	473
query35	699	569	553	553
query36	1071	928	910	910
query37	212	80	80	80
query38	2910	2793	2802	2793
query39	849	791	792	791
query40	284	121	118	118
query41	48	44	41	41
query42	101	95	100	95
query43	588	564	553	553
query44	1207	738	767	738
query45	184	164	162	162
query46	1074	692	734	692
query47	1841	1772	1738	1738
query48	371	297	290	290
query49	1178	388	390	388
query50	786	416	399	399
query51	6983	6832	6704	6704
query52	110	100	89	89
query53	351	292	284	284
query54	921	434	422	422
query55	75	72	73	72
query56	260	242	242	242
query57	1108	1053	1065	1053
query58	238	209	211	209
query59	3386	3323	3193	3193
query60	301	256	250	250
query61	94	87	90	87
query62	844	684	653	653
query63	311	281	284	281
query64	9751	2234	1807	1807
query65	3197	3108	3134	3108
query66	1274	334	325	325
query67	15245	15041	14835	14835
query68	6406	529	539	529
query69	503	269	276	269
query70	1223	1157	1135	1135
query71	510	269	270	269
query72	7765	5666	2737	2737
query73	821	318	320	318
query74	6130	5612	5664	5612
query75	4024	2599	2632	2599
query76	4521	994	961	961
query77	677	270	272	270
query78	10392	9901	10038	9901
query79	3029	512	520	512
query80	1276	503	434	434
query81	524	246	245	245
query82	763	206	203	203
query83	215	175	171	171
query84	265	86	84	84
query85	1436	286	263	263
query86	463	310	295	295
query87	3357	3094	3079	3079
query88	4393	2315	2328	2315
query89	501	401	388	388
query90	2015	195	192	192
query91	125	95	97	95
query92	62	51	48	48
query93	4047	511	501	501
query94	1135	237	186	186
query95	419	320	317	317
query96	610	266	267	266
query97	3191	2999	2967	2967
query98	239	218	212	212
query99	1663	1283	1282	1282
Total cold run time: 282740 ms
Total hot run time: 171080 ms

doris-robot · 2024-05-22T10:17:45Z

ClickBench: Total hot run time: 30.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c1ead2da8d951d6396d7c5af18194600ae996f3, data reload: false
query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.06
query4	1.65	0.07	0.08
query5	0.50	0.48	0.50
query6	1.12	0.72	0.71
query7	0.03	0.01	0.01
query8	0.04	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.55	0.53
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.77	0.78
query15	0.83	0.82	0.80
query16	0.37	0.34	0.37
query17	0.96	0.95	0.95
query18	0.21	0.25	0.25
query19	1.82	1.69	1.63
query20	0.02	0.01	0.01
query21	15.74	0.66	0.65
query22	4.00	8.03	1.83
query23	18.25	1.38	1.27
query24	1.64	0.26	0.22
query25	0.14	0.09	0.08
query26	0.25	0.17	0.17
query27	0.08	0.08	0.07
query28	13.48	1.02	1.02
query29	13.05	3.32	3.31
query30	0.24	0.06	0.06
query31	2.84	0.40	0.38
query32	3.27	0.47	0.46
query33	2.92	2.87	2.91
query34	17.26	4.40	4.48
query35	4.52	4.56	4.47
query36	0.65	0.45	0.47
query37	0.17	0.15	0.15
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.74 s
Total hot run time: 30.31 s

englefly · 2024-05-27T02:37:06Z

run buildall

doris-robot · 2024-05-27T03:14:32Z

TPC-H: Total hot run time: 41660 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0736491208510c40800cb07c07a7c2395bfe67bc, data reload: false
------ Round 1 ----------------------------------
q1	17617	4383	4177	4177
q2	2009	190	185	185
q3	10449	1277	1244	1244
q4	10185	918	790	790
q5	7463	2724	2696	2696
q6	216	134	135	134
q7	992	629	627	627
q8	9222	2147	2095	2095
q9	9357	6670	6630	6630
q10	9448	3901	3826	3826
q11	520	235	226	226
q12	415	226	219	219
q13	18198	3283	3222	3222
q14	280	219	240	219
q15	517	483	471	471
q16	499	389	375	375
q17	962	685	696	685
q18	8339	7957	7748	7748
q19	2066	1544	1498	1498
q20	649	298	317	298
q21	5117	4016	4125	4016
q22	347	285	279	279
Total cold run time: 114867 ms
Total hot run time: 41660 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4550	4358	4391	4358
q2	385	270	265	265
q3	3146	2865	2953	2865
q4	2010	1638	1641	1638
q5	5378	5471	5499	5471
q6	217	128	124	124
q7	2192	1841	1817	1817
q8	3205	3393	3355	3355
q9	8680	8632	8619	8619
q10	4043	3728	3830	3728
q11	586	513	491	491
q12	806	613	630	613
q13	16858	3113	3139	3113
q14	303	276	281	276
q15	529	495	482	482
q16	499	435	434	434
q17	1792	1512	1528	1512
q18	7759	7599	7546	7546
q19	1682	1591	1553	1553
q20	2034	1772	1795	1772
q21	4815	4715	4711	4711
q22	589	500	485	485
Total cold run time: 72058 ms
Total hot run time: 55228 ms

doris-robot · 2024-05-27T03:25:08Z

TPC-DS: Total hot run time: 171056 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0736491208510c40800cb07c07a7c2395bfe67bc, data reload: false
query1	922	392	371	371
query2	6429	2240	2346	2240
query3	6655	217	209	209
query4	19493	17393	17407	17393
query5	4113	432	414	414
query6	242	155	151	151
query7	4603	294	289	289
query8	237	191	197	191
query9	8615	2375	2365	2365
query10	460	269	288	269
query11	10393	10198	10051	10051
query12	136	98	87	87
query13	1655	372	368	368
query14	9403	5929	6860	5929
query15	211	171	168	168
query16	7131	286	258	258
query17	1062	528	525	525
query18	1905	272	273	272
query19	202	152	150	150
query20	92	86	84	84
query21	193	127	131	127
query22	4261	3937	3850	3850
query23	33547	33152	32850	32850
query24	6984	2840	2816	2816
query25	566	357	362	357
query26	704	152	146	146
query27	2024	291	288	288
query28	3926	2033	2037	2033
query29	864	604	586	586
query30	246	168	175	168
query31	945	767	740	740
query32	91	87	53	53
query33	501	266	260	260
query34	852	472	469	469
query35	674	589	563	563
query36	1068	928	932	928
query37	116	75	79	75
query38	2865	2776	2744	2744
query39	872	778	778	778
query40	194	120	122	120
query41	46	44	44	44
query42	101	98	98	98
query43	586	550	549	549
query44	1052	721	735	721
query45	184	168	164	164
query46	1060	721	708	708
query47	1829	1748	1773	1748
query48	356	287	289	287
query49	765	381	416	381
query50	770	392	400	392
query51	6898	6713	6759	6713
query52	100	97	91	91
query53	359	287	284	284
query54	540	448	426	426
query55	76	74	73	73
query56	261	251	248	248
query57	1087	1057	1062	1057
query58	230	217	228	217
query59	3380	3157	3045	3045
query60	274	259	250	250
query61	88	90	91	90
query62	770	669	638	638
query63	307	277	280	277
query64	8564	2216	1693	1693
query65	3170	3102	3106	3102
query66	906	333	334	333
query67	15018	14992	14676	14676
query68	4541	538	533	533
query69	447	283	269	269
query70	1177	1144	1178	1144
query71	381	274	272	272
query72	7343	5278	5269	5269
query73	718	319	319	319
query74	5956	5616	5567	5567
query75	3316	2610	2630	2610
query76	2153	944	1016	944
query77	399	270	272	270
query78	10064	9809	9722	9722
query79	1964	513	507	507
query80	989	441	442	441
query81	517	242	243	242
query82	959	197	196	196
query83	262	168	171	168
query84	260	83	88	83
query85	1036	276	267	267
query86	481	314	309	309
query87	3375	3123	3091	3091
query88	3326	2351	2344	2344
query89	472	391	373	373
query90	1995	190	192	190
query91	127	98	93	93
query92	59	48	49	48
query93	1963	505	489	489
query94	1140	185	184	184
query95	399	312	314	312
query96	585	268	265	265
query97	3119	3015	3006	3006
query98	262	220	228	220
query99	1528	1275	1269	1269
Total cold run time: 255542 ms
Total hot run time: 171056 ms

doris-robot · 2024-05-27T03:30:38Z

ClickBench: Total hot run time: 31.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0736491208510c40800cb07c07a7c2395bfe67bc, data reload: false
query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.24	0.06	0.07
query4	1.65	0.10	0.09
query5	0.49	0.51	0.50
query6	1.11	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.51	0.50
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.81	0.77	0.77
query15	0.84	0.82	0.82
query16	0.37	0.37	0.35
query17	1.02	0.94	0.98
query18	0.96	0.94	0.94
query19	1.78	1.71	1.68
query20	0.01	0.01	0.01
query21	15.51	0.69	0.68
query22	3.50	7.70	2.03
query23	18.29	1.30	1.22
query24	1.79	0.26	0.20
query25	0.14	0.08	0.09
query26	0.25	0.16	0.16
query27	0.08	0.07	0.07
query28	13.32	1.01	0.99
query29	13.27	3.32	3.24
query30	0.24	0.06	0.06
query31	2.86	0.38	0.38
query32	3.30	0.46	0.47
query33	2.94	2.86	2.93
query34	17.10	4.46	4.41
query35	4.49	4.48	4.63
query36	0.65	0.46	0.46
query37	0.17	0.15	0.16
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.14	0.15
query41	0.09	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.88 s
Total hot run time: 31.22 s

englefly · 2024-07-03T07:07:28Z

run buildall

doris-robot · 2024-07-03T07:29:51Z

TPC-H: Total hot run time: 39942 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 774a94b676576d7a81f265c613319dea5f758a9a, data reload: false
------ Round 1 ----------------------------------
q1	17838	6359	4309	4309
q2	2023	194	192	192
q3	10625	1213	1070	1070
q4	10562	710	900	710
q5	7573	2664	2679	2664
q6	230	145	145	145
q7	955	604	625	604
q8	9640	2106	2091	2091
q9	8964	6524	6477	6477
q10	8955	3762	3721	3721
q11	449	241	236	236
q12	468	244	236	236
q13	17772	3040	3026	3026
q14	262	236	222	222
q15	519	476	507	476
q16	510	369	378	369
q17	971	655	704	655
q18	8003	7437	7368	7368
q19	7235	1573	1462	1462
q20	663	326	342	326
q21	4903	3245	3341	3245
q22	392	347	338	338
Total cold run time: 119512 ms
Total hot run time: 39942 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4412	4281	4251	4251
q2	363	266	265	265
q3	2998	2665	2717	2665
q4	1849	1547	1657	1547
q5	5445	5287	5254	5254
q6	246	133	135	133
q7	2099	1727	1720	1720
q8	3203	3325	3302	3302
q9	8362	8338	8329	8329
q10	3900	3665	3690	3665
q11	565	490	484	484
q12	796	634	626	626
q13	17578	3011	3014	3011
q14	293	253	263	253
q15	513	491	477	477
q16	473	409	420	409
q17	1779	1482	1462	1462
q18	7625	7641	7308	7308
q19	1698	1523	1612	1523
q20	1995	1788	1781	1781
q21	4865	4707	4795	4707
q22	621	553	539	539
Total cold run time: 71678 ms
Total hot run time: 53711 ms

doris-robot · 2024-07-03T07:40:50Z

TPC-DS: Total hot run time: 170725 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 774a94b676576d7a81f265c613319dea5f758a9a, data reload: false
query1	942	375	383	375
query2	6470	2299	2219	2219
query3	6660	205	211	205
query4	19780	17431	17154	17154
query5	4164	484	485	484
query6	271	175	164	164
query7	4584	289	285	285
query8	312	317	314	314
query9	8628	2364	2355	2355
query10	617	305	286	286
query11	10320	10005	9925	9925
query12	129	89	85	85
query13	1656	378	365	365
query14	10197	7355	7761	7355
query15	238	192	195	192
query16	7889	264	268	264
query17	1909	559	536	536
query18	1959	279	267	267
query19	199	162	153	153
query20	91	80	92	80
query21	223	143	125	125
query22	4303	4021	4266	4021
query23	33569	33199	32971	32971
query24	11943	2787	2799	2787
query25	680	368	362	362
query26	1777	150	148	148
query27	2902	264	271	264
query28	7741	2061	2053	2053
query29	1141	641	611	611
query30	285	152	156	152
query31	959	726	725	725
query32	95	50	52	50
query33	770	286	293	286
query34	951	467	466	466
query35	694	596	561	561
query36	1079	940	930	930
query37	195	78	80	78
query38	2862	2732	2761	2732
query39	843	809	799	799
query40	282	120	125	120
query41	56	56	59	56
query42	123	102	105	102
query43	587	558	548	548
query44	1269	734	736	734
query45	184	159	163	159
query46	1069	713	702	702
query47	1857	1760	1755	1755
query48	367	311	290	290
query49	1186	409	411	409
query50	784	403	399	399
query51	6892	6867	6745	6745
query52	110	88	95	88
query53	361	304	288	288
query54	1031	451	445	445
query55	76	73	74	73
query56	297	268	264	264
query57	1144	1052	1063	1052
query58	263	252	258	252
query59	3502	3480	3213	3213
query60	319	315	284	284
query61	98	96	91	91
query62	807	635	661	635
query63	321	292	295	292
query64	9856	2262	1778	1778
query65	3174	3101	3102	3101
query66	1387	334	330	330
query67	15535	14877	14919	14877
query68	4630	542	544	542
query69	460	322	317	317
query70	1163	1160	1135	1135
query71	383	274	269	269
query72	7063	2813	2641	2641
query73	736	324	329	324
query74	5907	5550	5530	5530
query75	3376	2683	2671	2671
query76	2572	982	954	954
query77	462	305	309	305
query78	10219	9768	9629	9629
query79	2215	511	508	508
query80	1110	481	469	469
query81	549	220	224	220
query82	723	141	141	141
query83	252	171	175	171
query84	245	85	85	85
query85	1957	344	267	267
query86	492	340	317	317
query87	3294	3051	3109	3051
query88	3877	2463	2449	2449
query89	472	378	377	377
query90	1847	190	192	190
query91	129	99	99	99
query92	69	48	48	48
query93	2418	519	510	510
query94	1281	188	191	188
query95	409	315	315	315
query96	600	279	277	277
query97	3154	3010	3011	3010
query98	213	198	200	198
query99	1696	1252	1272	1252
Total cold run time: 276937 ms
Total hot run time: 170725 ms

doris-robot · 2024-07-03T07:46:19Z

ClickBench: Total hot run time: 30.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 774a94b676576d7a81f265c613319dea5f758a9a, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.22	0.04	0.04
query4	1.69	0.06	0.06
query5	0.50	0.49	0.49
query6	1.13	0.71	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.48	0.49
query10	0.55	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.58	0.59	0.60
query14	0.78	0.76	0.77
query15	0.85	0.80	0.81
query16	0.36	0.37	0.37
query17	1.03	1.03	1.05
query18	0.22	0.20	0.21
query19	1.79	1.71	1.70
query20	0.01	0.00	0.01
query21	15.46	0.77	0.66
query22	4.23	7.13	1.56
query23	18.32	1.48	1.27
query24	2.08	0.23	0.22
query25	0.16	0.09	0.09
query26	0.26	0.17	0.18
query27	0.09	0.08	0.08
query28	13.33	1.02	1.00
query29	12.59	3.23	3.24
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.30	0.48	0.46
query33	2.87	2.91	2.88
query34	17.11	4.42	4.41
query35	4.46	4.44	4.52
query36	0.65	0.47	0.46
query37	0.18	0.16	0.16
query38	0.16	0.14	0.16
query39	0.04	0.03	0.04
query40	0.17	0.15	0.14
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.54 s
Total hot run time: 30.1 s

englefly · 2024-07-03T15:52:15Z

run p0

englefly · 2024-07-03T15:52:31Z

run cloud_p1

englefly · 2024-07-04T01:41:35Z

run buildall

doris-robot · 2024-07-04T02:01:30Z

TPC-H: Total hot run time: 39917 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 368929706d3861fe09489809e48106e1dc87762a, data reload: false
------ Round 1 ----------------------------------
q1	18524	4531	4378	4378
q2	2972	199	194	194
q3	11211	1158	1138	1138
q4	10689	824	722	722
q5	7635	2741	2664	2664
q6	226	140	143	140
q7	959	607	615	607
q8	9254	2027	2070	2027
q9	8842	6482	6436	6436
q10	8983	3753	3719	3719
q11	442	243	244	243
q12	436	256	236	236
q13	18843	2982	2989	2982
q14	260	236	232	232
q15	525	495	486	486
q16	497	377	375	375
q17	962	619	730	619
q18	7977	7452	7400	7400
q19	7423	1456	1491	1456
q20	665	332	329	329
q21	5086	3199	3342	3199
q22	400	335	345	335
Total cold run time: 122811 ms
Total hot run time: 39917 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4385	4273	4263	4263
q2	371	276	262	262
q3	2955	2725	2696	2696
q4	1907	1653	1653	1653
q5	5279	5309	5262	5262
q6	219	134	134	134
q7	2135	1729	1771	1729
q8	3214	3333	3339	3333
q9	8352	8317	8343	8317
q10	3861	3710	3755	3710
q11	573	482	491	482
q12	797	625	621	621
q13	17510	3035	3032	3032
q14	280	262	251	251
q15	528	480	471	471
q16	469	414	427	414
q17	1788	1477	1476	1476
q18	7694	7444	7331	7331
q19	1736	1498	1455	1455
q20	1998	1766	1774	1766
q21	4948	4880	4784	4784
q22	613	565	514	514
Total cold run time: 71612 ms
Total hot run time: 53956 ms

doris-robot · 2024-07-04T02:12:52Z

TPC-DS: Total hot run time: 172310 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 368929706d3861fe09489809e48106e1dc87762a, data reload: false
query1	924	382	369	369
query2	6458	2377	2264	2264
query3	6668	203	210	203
query4	20133	17548	17170	17170
query5	4183	496	485	485
query6	281	188	173	173
query7	4595	289	279	279
query8	305	291	267	267
query9	8740	2466	2446	2446
query10	596	320	281	281
query11	10513	9932	10051	9932
query12	132	89	81	81
query13	1638	363	367	363
query14	10110	6814	7509	6814
query15	235	184	204	184
query16	7774	321	320	320
query17	1554	583	535	535
query18	1950	282	281	281
query19	195	152	160	152
query20	91	84	85	84
query21	225	134	129	129
query22	4188	3998	3856	3856
query23	33638	33065	33111	33065
query24	12193	2827	2780	2780
query25	687	404	405	404
query26	1835	153	149	149
query27	3030	274	272	272
query28	7760	2128	2114	2114
query29	1163	645	626	626
query30	288	157	153	153
query31	972	765	747	747
query32	98	57	57	57
query33	795	306	299	299
query34	971	469	474	469
query35	693	598	563	563
query36	1097	969	930	930
query37	285	84	94	84
query38	2915	2737	2756	2737
query39	843	795	837	795
query40	278	129	123	123
query41	58	54	55	54
query42	122	187	100	100
query43	610	564	529	529
query44	1301	758	739	739
query45	195	163	163	163
query46	1071	749	740	740
query47	1879	1758	1757	1757
query48	369	291	300	291
query49	1185	411	415	411
query50	773	397	408	397
query51	6862	6622	6774	6622
query52	110	91	97	91
query53	359	302	292	292
query54	1023	462	442	442
query55	74	74	76	74
query56	282	274	268	268
query57	1148	1049	1040	1040
query58	250	259	252	252
query59	3342	3416	3225	3225
query60	318	295	311	295
query61	100	111	95	95
query62	839	666	659	659
query63	317	288	290	288
query64	9859	2235	1794	1794
query65	3165	3109	3101	3101
query66	1382	339	327	327
query67	15491	14847	14972	14847
query68	8889	562	553	553
query69	715	433	334	334
query70	1215	1152	1099	1099
query71	498	282	280	280
query72	7163	5357	5472	5357
query73	833	327	328	327
query74	5814	5496	5472	5472
query75	4199	2702	2684	2684
query76	5122	974	925	925
query77	685	308	303	303
query78	9611	10089	8848	8848
query79	8139	513	519	513
query80	1299	482	482	482
query81	573	217	223	217
query82	761	142	135	135
query83	200	173	168	168
query84	275	86	86	86
query85	1464	291	331	291
query86	422	317	320	317
query87	3326	3140	3098	3098
query88	4433	2372	2395	2372
query89	522	383	382	382
query90	2022	193	192	192
query91	128	99	101	99
query92	62	50	50	50
query93	6253	516	524	516
query94	1276	215	217	215
query95	410	323	319	319
query96	616	270	271	270
query97	3189	3089	3006	3006
query98	214	210	204	204
query99	1617	1288	1268	1268
Total cold run time: 295458 ms
Total hot run time: 172310 ms

doris-robot · 2024-07-04T02:18:22Z

ClickBench: Total hot run time: 30.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 368929706d3861fe09489809e48106e1dc87762a, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.47	0.49
query6	1.13	0.73	0.74
query7	0.02	0.01	0.02
query8	0.04	0.05	0.05
query9	0.56	0.49	0.50
query10	0.54	0.53	0.53
query11	0.15	0.11	0.11
query12	0.15	0.13	0.12
query13	0.59	0.58	0.58
query14	0.80	0.81	0.77
query15	0.87	0.82	0.81
query16	0.35	0.35	0.35
query17	0.99	1.01	0.96
query18	0.22	0.21	0.20
query19	1.74	1.70	1.74
query20	0.02	0.01	0.01
query21	15.39	0.76	0.65
query22	4.59	7.67	1.41
query23	18.23	1.30	1.35
query24	2.10	0.23	0.24
query25	0.15	0.09	0.08
query26	0.27	0.18	0.17
query27	0.08	0.09	0.08
query28	13.25	1.02	0.99
query29	12.59	3.25	3.24
query30	0.26	0.05	0.05
query31	2.87	0.39	0.39
query32	3.27	0.48	0.48
query33	2.86	2.92	2.95
query34	17.11	4.47	4.45
query35	4.44	4.48	4.57
query36	0.66	0.46	0.47
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.55 s
Total hot run time: 30.04 s

HappenLee

LGTM

code is refactored

## Proposed changes for a pattern: topn(n)->globalAgg->localAgg this pr tries to add a filter on global/localAgg which means only the first n tuples are counted, and others could be ignored. inorder to obtain this benefit, optimizer will change limit node to topn node if the limit number is less than topnOptLimitThreshold. Issue Number: close #xxx  ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...

### What problem does this PR solve? Pr #34853 introduced PushTopnToAgg rule. But there is a limitation that Topn(limit) should output all group by keys. This pr removes this limitation by using the first group by key as order key.

Before merge apache#34853， should merge the pr firstly

## Proposed changes for a pattern: topn(n)->globalAgg->localAgg this pr tries to add a filter on global/localAgg which means only the first n tuples are counted, and others could be ignored. inorder to obtain this benefit, optimizer will change limit node to topn node if the limit number is less than topnOptLimitThreshold. Issue Number: close #xxx  ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...

### What problem does this PR solve? Pr apache#34853 introduced PushTopnToAgg rule. But there is a limitation that Topn(limit) should output all group by keys. This pr removes this limitation by using the first group by key as order key.

github-actionsBot added the meta-change label May 14, 2024

englefly force-pushed the limit-localAgg branch from a9fa9a9 to 25ba503Compare May 15, 2024 08:29

morrySnow previously requested changes May 21, 2024
View reviewed changes

Comment thread...-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalHashAggregate.java Outdated

englefly marked this pull request as draft May 22, 2024 06:31

englefly force-pushed the limit-localAgg branch from 863149b to 8c1ead2Compare May 22, 2024 08:55

englefly force-pushed the limit-localAgg branch from 8c1ead2 to ae56111Compare May 24, 2024 08:20

morrySnow added the not-merge/2.1 label May 27, 2024

xzj7019 reviewed May 27, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LimitAggToTopNAgg.java

englefly added 4 commits July 4, 2024 09:26

push limit to local agg
2781e49

update rt
b0f176b

rt
07b7748

update-shape
3689297

englefly force-pushed the limit-localAgg branch from 774a94b to 3689297Compare July 4, 2024 01:41

starocean999 approved these changes Jul 5, 2024
View reviewed changes

HappenLee approved these changes Jul 8, 2024
View reviewed changes

englefly merged commit 3bb5a8d into apache:masterJul 8, 2024

wm1581066 added the need more test Add more test label Jul 8, 2024

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024
Release Note 3.0.1 #39570
Closed

englefly mentioned this pull request Nov 18, 2024
[opt](nereids) optimize push limit to agg #44042
Merged
16 tasks

englefly deleted the limit-localAgg branch June 24, 2025 03:51

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[Exec](agg) Fix agg limit result error (apache#37025)
7090b23
Before merge apache#34853， should merge the pr firstly

Uh oh!

Conversation

englefly commented May 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 14, 2024

Uh oh!

englefly commented May 14, 2024

Uh oh!

github-actionsBot commented May 14, 2024

Uh oh!

github-actionsBot commented May 15, 2024

Uh oh!

morrySnow commented May 15, 2024

Uh oh!

github-actionsBot commented May 15, 2024

Uh oh!

morrySnow commented May 21, 2024

Uh oh!

Uh oh!

englefly commented May 22, 2024

Uh oh!

github-actionsBot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

github-actionsBot commented May 22, 2024

Uh oh!

englefly commented May 22, 2024

Uh oh!

github-actionsBot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

englefly commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

Uh oh!

englefly commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

englefly commented Jul 3, 2024

Uh oh!

englefly commented Jul 3, 2024

Uh oh!

englefly commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

englefly commented May 14, 2024•
edited
Loading