[feat](Nereids) push down predicates with multi columns through LogicalWindow and LogicalPartitionTopN by feiniaofeiafei · Pull Request #36984 · apache/doris

feiniaofeiafei · 2024-06-28T03:15:08Z

cherry-pick #36828 to branch-2.0

The requirement for predicate pushdown through the window operator is that the partition by slots of the window contains all slots in the predicate. The original implementation of doris only allows predicate pushdown with one slot. This PR relaxes this restriction and allows for predicate pushdown with multiple slots. The same applies to the predicate pushdown of the LogicalPartitionTopN operator. The following sql is an example.

select*from
(
select
row_number() over(partition by id, value1 order by value1) as num,
id,
value1
from
push_down_multi_column_predicate_through_window_t ) t
where
abs(id + value1)<4and num <=2;

…alWindow and LogicalPartitionTopN (apache#36828) The requirement for predicate pushdown through the window operator is that the partition by slots of the window contains all slots in the predicate. The original implementation of doris only allows predicate pushdown with one slot. This PR relaxes this restriction and allows for predicate pushdown with multiple slots. The same applies to the predicate pushdown of the LogicalPartitionTopN operator. The following sql is an example. ```sql select * from ( select row_number() over(partition by id, value1 order by value1) as num, id, value1 from push_down_multi_column_predicate_through_window_t ) t where abs(id + value1)<4 and num <= 2; ``` --------- Co-authored-by: feiniaofeiafei <moailing@selectdb.com>

doris-robot · 2024-06-28T03:15:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

feiniaofeiafei · 2024-06-28T03:15:18Z

run buildall

feiniaofeiafei · 2024-06-28T03:15:41Z

run buildall

feiniaofeiafei · 2024-06-28T03:19:38Z

run buildall

doris-robot · 2024-06-28T03:37:52Z

TPC-H: Total hot run time: 50686 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e597ff6e96942efc2a6a31f68d0ff0b6bbf6ad4, data reload: false
------ Round 1 ----------------------------------
q1	17817	4427	4429	4427
q2	2080	160	149	149
q3	10401	1920	1996	1920
q4	10398	1264	1336	1264
q5	9061	3911	3936	3911
q6	259	127	127	127
q7	2034	1640	1637	1637
q8	9371	2811	2724	2724
q9	10834	10348	10550	10348
q10	9008	3674	3712	3674
q11	448	258	271	258
q12	489	317	322	317
q13	19045	4118	4192	4118
q14	374	365	416	365
q15	554	460	464	460
q16	700	602	616	602
q17	1151	983	978	978
q18	7290	6967	6955	6955
q19	2980	1623	1600	1600
q20	957	354	314	314
q21	4506	4116	4087	4087
q22	560	465	451	451
Total cold run time: 120317 ms
Total hot run time: 50686 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4349	4316	4322	4316
q2	325	230	231	230
q3	4157	4150	4183	4150
q4	2762	2727	2748	2727
q5	7141	7049	7083	7049
q6	237	122	120	120
q7	3250	2819	2835	2819
q8	4347	4459	4461	4459
q9	16824	16875	16667	16667
q10	4224	4270	4242	4242
q11	787	673	696	673
q12	1040	836	855	836
q13	7003	3741	3740	3740
q14	448	433	425	425
q15	511	463	455	455
q16	741	709	676	676
q17	3826	3828	3882	3828
q18	8842	8746	8868	8746
q19	1719	1698	1658	1658
q20	2361	2161	2112	2112
q21	8621	8459	8466	8459
q22	1029	975	926	926
Total cold run time: 84544 ms
Total hot run time: 79313 ms

doris-robot · 2024-06-28T03:50:33Z

TPC-DS: Total hot run time: 204130 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e597ff6e96942efc2a6a31f68d0ff0b6bbf6ad4, data reload: false
query1	929	420	381	381
query2	6537	2870	2890	2870
query3	6921	207	208	207
query4	20694	17976	17991	17976
query5	19749	6495	6419	6419
query6	285	226	231	226
query7	4155	310	315	310
query8	426	411	401	401
query9	3133	2698	2605	2605
query10	402	310	294	294
query11	11238	10628	10661	10628
query12	116	81	74	74
query13	5595	689	692	689
query14	17763	13801	13341	13341
query15	363	246	255	246
query16	6489	287	260	260
query17	1514	1491	893	893
query18	2271	411	417	411
query19	212	146	156	146
query20	82	78	83	78
query21	193	94	99	94
query22	5186	4983	5003	4983
query23	32574	31888	32005	31888
query24	7022	6556	6499	6499
query25	533	436	428	428
query26	513	169	164	164
query27	1825	300	302	300
query28	6094	2396	2329	2329
query29	2902	2727	2855	2727
query30	242	166	173	166
query31	911	733	735	733
query32	72	68	63	63
query33	388	261	251	251
query34	837	484	487	484
query35	1114	936	917	917
query36	1441	1210	1192	1192
query37	87	61	63	61
query38	3082	2949	2918	2918
query39	1386	1325	1326	1325
query40	214	95	91	91
query41	47	45	44	44
query42	91	82	87	82
query43	724	625	793	625
query44	1153	725	741	725
query45	249	235	237	235
query46	1252	959	967	959
query47	1863	1979	1802	1802
query48	1039	720	717	717
query49	626	384	374	374
query50	863	629	589	589
query51	4755	4651	4568	4568
query52	96	79	83	79
query53	449	335	328	328
query54	2666	2443	2477	2443
query55	87	86	86	86
query56	234	241	218	218
query57	1178	1082	1077	1077
query58	217	213	197	197
query59	4273	4004	4008	4004
query60	212	201	196	196
query61	102	97	100	97
query62	851	448	467	448
query63	492	350	349	349
query64	2237	1533	1417	1417
query65	3632	3553	3564	3553
query66	795	385	385	385
query67	16183	15930	16210	15930
query68	8504	655	690	655
query69	568	375	370	370
query70	1607	1335	1454	1335
query71	381	327	320	320
query72	6649	3530	3526	3526
query73	736	322	329	322
query74	6367	5928	5855	5855
query75	5014	3759	3647	3647
query76	5099	1139	1184	1139
query77	780	257	252	252
query78	12480	11413	20585	11413
query79	6209	642	647	642
query80	1190	411	406	406
query81	500	238	239	238
query82	268	97	102	97
query83	186	146	136	136
query84	260	72	74	72
query85	1022	329	323	323
query86	350	334	311	311
query87	3226	3013	3056	3013
query88	4136	2317	2306	2306
query89	354	289	308	289
query90	1807	211	219	211
query91	190	149	141	141
query92	63	57	53	53
query93	958	550	588	550
query94	731	217	213	213
query95	1132	1105	1075	1075
query96	665	339	335	335
query97	6521	6312	6431	6312
query98	194	173	178	173
query99	3014	805	866	805
Total cold run time: 303017 ms
Total hot run time: 204130 ms

doris-robot · 2024-06-28T03:56:02Z

ClickBench: Total hot run time: 30.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e597ff6e96942efc2a6a31f68d0ff0b6bbf6ad4, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.06	0.05
query4	1.77	0.10	0.06
query5	0.54	0.53	0.52
query6	1.28	0.64	0.63
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.48	0.47
query10	0.54	0.53	0.54
query11	0.11	0.09	0.08
query12	0.12	0.09	0.08
query13	0.63	0.62	0.60
query14	0.78	0.78	0.78
query15	0.79	0.78	0.75
query16	0.37	0.40	0.36
query17	1.05	1.02	0.99
query18	0.22	0.27	0.24
query19	1.94	1.84	1.86
query20	0.01	0.01	0.01
query21	15.47	0.55	0.55
query22	2.09	2.29	1.62
query23	17.06	1.02	0.96
query24	5.25	1.10	1.05
query25	0.40	0.11	0.06
query26	0.57	0.16	0.17
query27	0.03	0.04	0.04
query28	7.71	0.74	0.71
query29	12.58	2.31	2.30
query30	0.61	0.53	0.53
query31	2.82	0.39	0.38
query32	3.38	0.49	0.50
query33	3.03	3.08	3.11
query34	15.26	4.81	4.80
query35	4.87	4.83	4.82
query36	1.06	1.02	1.02
query37	0.06	0.05	0.04
query38	0.03	0.03	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.06	0.02	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.01
Total cold run time: 103.64 s
Total hot run time: 30.86 s

doris-robot · 2024-06-28T03:59:13Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9e597ff6e96942efc2a6a31f68d0ff0b6bbf6ad4 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s

…alWindow and LogicalPartitionTopN

feiniaofeiafei · 2024-06-28T06:17:10Z

run buildall

doris-robot · 2024-06-28T06:35:50Z

TPC-H: Total hot run time: 49843 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91dffca63df8d99da45e856f264bf811cd07a40b, data reload: false
------ Round 1 ----------------------------------
q1	17542	4451	4335	4335
q2	2078	153	146	146
q3	10468	1882	1967	1882
q4	10312	1226	1336	1226
q5	8409	3875	3925	3875
q6	233	127	125	125
q7	2073	1608	1600	1600
q8	9520	2727	2710	2710
q9	13883	11079	10323	10323
q10	8673	3520	3538	3520
q11	422	247	238	238
q12	487	309	309	309
q13	18348	3946	4062	3946
q14	356	329	335	329
q15	497	452	458	452
q16	673	579	571	571
q17	1128	981	945	945
q18	7304	6856	6826	6826
q19	1780	1635	1621	1621
q20	570	315	315	315
q21	4443	4112	4117	4112
q22	531	463	437	437
Total cold run time: 119730 ms
Total hot run time: 49843 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4370	4314	4352	4314
q2	319	230	216	216
q3	4156	4175	4175	4175
q4	2753	2752	2770	2752
q5	7157	7121	7071	7071
q6	239	118	121	118
q7	3208	2841	2786	2786
q8	4366	4442	4443	4442
q9	16912	16778	16898	16778
q10	4270	4242	4283	4242
q11	747	706	709	706
q12	1031	846	877	846
q13	7012	3742	3769	3742
q14	470	426	434	426
q15	504	471	461	461
q16	748	690	698	690
q17	3846	3913	3820	3820
q18	8888	8709	8799	8709
q19	1730	1693	1702	1693
q20	2386	2122	2084	2084
q21	8479	8375	8451	8375
q22	1036	993	1004	993
Total cold run time: 84627 ms
Total hot run time: 79439 ms

doris-robot · 2024-06-28T06:48:30Z

TPC-DS: Total hot run time: 203647 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91dffca63df8d99da45e856f264bf811cd07a40b, data reload: false
query1	960	387	380	380
query2	6535	2855	2822	2822
query3	6915	206	203	203
query4	19869	18109	17894	17894
query5	19728	6530	6553	6530
query6	294	223	231	223
query7	4149	292	305	292
query8	449	397	452	397
query9	3139	2698	2599	2599
query10	409	306	308	306
query11	11322	10677	10786	10677
query12	128	80	80	80
query13	5595	692	695	692
query14	17798	13724	13735	13724
query15	357	250	246	246
query16	6458	305	260	260
query17	1688	1453	868	868
query18	2312	415	404	404
query19	223	149	152	149
query20	86	78	79	78
query21	192	92	98	92
query22	5229	4891	4953	4891
query23	32739	32149	32017	32017
query24	6972	6556	6505	6505
query25	522	436	440	436
query26	620	160	163	160
query27	1998	300	291	291
query28	6121	2346	2298	2298
query29	2961	2638	2772	2638
query30	242	169	168	168
query31	889	720	733	720
query32	62	63	62	62
query33	394	262	252	252
query34	835	474	479	474
query35	1116	924	910	910
query36	1310	1135	1181	1135
query37	90	59	60	59
query38	3065	2912	2977	2912
query39	1366	1346	1306	1306
query40	219	95	92	92
query41	47	44	44	44
query42	91	83	76	76
query43	814	723	779	723
query44	1147	715	706	706
query45	251	238	232	232
query46	1215	980	957	957
query47	1829	1750	1686	1686
query48	1004	721	715	715
query49	625	382	365	365
query50	863	617	601	601
query51	4687	4639	4685	4639
query52	113	79	104	79
query53	440	323	334	323
query54	2697	2449	2536	2449
query55	80	79	84	79
query56	226	206	208	206
query57	1233	1149	1193	1149
query58	218	214	192	192
query59	4264	4225	3935	3935
query60	210	213	209	209
query61	95	94	93	93
query62	813	474	418	418
query63	495	339	342	339
query64	2638	1547	1352	1352
query65	3609	3557	3554	3554
query66	795	376	371	371
query67	16379	16630	15232	15232
query68	8378	646	641	641
query69	552	348	341	341
query70	1606	1353	1461	1353
query71	405	308	320	308
query72	6501	3535	3464	3464
query73	723	318	317	317
query74	6219	5840	5846	5840
query75	5065	3822	3655	3655
query76	5079	1166	1210	1166
query77	859	261	252	252
query78	12500	11652	11746	11652
query79	7658	630	635	630
query80	1029	399	405	399
query81	490	234	233	233
query82	1491	97	99	97
query83	167	130	131	130
query84	262	70	70	70
query85	878	313	317	313
query86	344	298	291	291
query87	3249	3073	3025	3025
query88	5032	2329	2322	2322
query89	360	307	292	292
query90	1953	188	205	188
query91	183	139	139	139
query92	60	52	51	51
query93	4781	564	579	564
query94	729	214	206	206
query95	1102	1064	1027	1027
query96	638	319	325	319
query97	6513	6390	6385	6385
query98	197	176	186	176
query99	2836	835	879	835
Total cold run time: 310073 ms
Total hot run time: 203647 ms

doris-robot · 2024-06-28T06:53:58Z

ClickBench: Total hot run time: 30.84 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91dffca63df8d99da45e856f264bf811cd07a40b, data reload: false
query1	0.02	0.02	0.03
query2	0.06	0.03	0.03
query3	0.24	0.05	0.05
query4	1.79	0.06	0.06
query5	0.54	0.52	0.52
query6	1.23	0.67	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.49
query10	0.54	0.53	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.77	0.79
query15	0.77	0.77	0.76
query16	0.36	0.35	0.35
query17	1.02	1.03	1.04
query18	0.24	0.25	0.26
query19	1.93	1.89	1.84
query20	0.01	0.01	0.01
query21	15.46	0.56	0.55
query22	2.23	2.01	1.54
query23	17.20	1.05	0.87
query24	5.88	1.26	1.31
query25	0.40	0.09	0.06
query26	0.66	0.16	0.16
query27	0.05	0.04	0.03
query28	6.83	0.72	0.71
query29	12.60	2.36	2.32
query30	0.52	0.53	0.46
query31	2.82	0.39	0.37
query32	3.38	0.50	0.49
query33	3.07	3.04	3.04
query34	15.26	4.78	4.77
query35	4.84	4.82	4.85
query36	1.09	1.02	1.01
query37	0.05	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.62 s
Total hot run time: 30.84 s

doris-robot · 2024-06-28T06:57:13Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 91dffca63df8d99da45e856f264bf811cd07a40b with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.1 seconds inserted 10000000 Rows, about 473K ops/s

…alWindow and LogicalPartitionTopN

feiniaofeiafei · 2024-06-28T07:52:27Z

run buildall

doris-robot · 2024-06-28T08:27:09Z

TPC-H: Total hot run time: 50028 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 33cfd3d5a54ee04864c9a1ef8796bddba0acb422, data reload: false
------ Round 1 ----------------------------------
q1	17782	4396	4341	4341
q2	2064	153	146	146
q3	10279	1973	1885	1885
q4	10351	1231	1311	1231
q5	8417	3896	3886	3886
q6	227	152	126	126
q7	2020	1592	1624	1592
q8	9490	2698	2698	2698
q9	12686	10523	10533	10523
q10	8633	3509	3502	3502
q11	418	259	254	254
q12	471	304	301	301
q13	18372	3948	4049	3948
q14	362	329	324	324
q15	496	457	447	447
q16	691	561	570	561
q17	1115	965	911	911
q18	7263	6994	6957	6957
q19	1782	1661	1598	1598
q20	529	296	311	296
q21	4336	4160	4063	4063
q22	536	438	447	438
Total cold run time: 118320 ms
Total hot run time: 50028 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4331	4361	4298	4298
q2	318	232	234	232
q3	4171	4143	4120	4120
q4	2744	2751	2741	2741
q5	7112	7058	7072	7058
q6	238	117	118	117
q7	3192	2853	2843	2843
q8	4306	4414	4434	4414
q9	17353	16870	16977	16870
q10	4206	4247	4278	4247
q11	734	694	708	694
q12	1035	855	840	840
q13	6667	3737	3773	3737
q14	472	437	420	420
q15	514	462	458	458
q16	725	683	684	683
q17	3787	3836	3784	3784
q18	8847	8753	8931	8753
q19	1730	1716	1690	1690
q20	2374	2098	2113	2098
q21	8394	8488	8384	8384
q22	1048	970	984	970
Total cold run time: 84298 ms
Total hot run time: 79451 ms

doris-robot · 2024-06-28T08:39:56Z

TPC-DS: Total hot run time: 203289 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 33cfd3d5a54ee04864c9a1ef8796bddba0acb422, data reload: false
query1	922	426	384	384
query2	6543	2845	2508	2508
query3	6921	207	206	206
query4	20091	18145	18061	18061
query5	19727	6550	6508	6508
query6	303	226	238	226
query7	4226	300	299	299
query8	390	392	369	369
query9	3103	2686	2610	2610
query10	416	299	314	299
query11	11344	10733	10824	10733
query12	130	79	74	74
query13	5596	692	695	692
query14	17866	13319	13245	13245
query15	368	246	246	246
query16	6513	289	265	265
query17	1698	1467	914	914
query18	3369	426	409	409
query19	487	159	149	149
query20	78	76	80	76
query21	192	106	94	94
query22	5247	5028	5092	5028
query23	32902	32690	32481	32481
query24	7472	6549	6614	6549
query25	564	427	449	427
query26	513	171	162	162
query27	2069	305	298	298
query28	8592	2303	2275	2275
query29	2819	2669	2650	2650
query30	244	168	167	167
query31	932	744	747	744
query32	72	67	64	64
query33	398	287	262	262
query34	845	466	473	466
query35	1141	982	913	913
query36	1223	1113	1277	1113
query37	93	61	63	61
query38	3031	2939	2941	2939
query39	1373	1335	1331	1331
query40	203	93	101	93
query41	47	43	45	43
query42	86	80	85	80
query43	733	662	733	662
query44	1115	714	721	714
query45	249	249	241	241
query46	1231	961	978	961
query47	1794	1825	1808	1808
query48	1021	689	704	689
query49	617	379	374	374
query50	855	620	613	613
query51	4794	4640	4610	4610
query52	96	81	79	79
query53	450	326	321	321
query54	2658	2470	2474	2470
query55	86	77	83	77
query56	238	221	215	215
query57	1313	1178	1128	1128
query58	223	211	182	182
query59	4110	4072	3722	3722
query60	210	196	206	196
query61	101	96	98	96
query62	789	483	504	483
query63	491	339	335	335
query64	2526	1552	1425	1425
query65	3633	3527	3541	3527
query66	775	381	381	381
query67	16060	15743	14781	14781
query68	9053	663	693	663
query69	569	341	357	341
query70	1582	1437	1655	1437
query71	432	305	325	305
query72	6545	3527	3472	3472
query73	732	319	315	315
query74	6238	5865	5905	5865
query75	5378	3594	3629	3594
query76	5608	1172	1077	1077
query77	953	256	260	256
query78	12713	11848	11567	11567
query79	8050	623	640	623
query80	1231	407	407	407
query81	493	237	234	234
query82	1658	101	97	97
query83	178	135	136	135
query84	260	71	71	71
query85	881	325	332	325
query86	332	299	337	299
query87	3285	3026	3054	3026
query88	5198	2361	2350	2350
query89	384	316	292	292
query90	1962	211	205	205
query91	170	139	138	138
query92	57	53	54	53
query93	5484	573	586	573
query94	706	213	210	210
query95	1114	1060	1042	1042
query96	649	333	330	330
query97	6531	6378	6525	6378
query98	188	175	179	175
query99	2917	883	831	831
Total cold run time: 317848 ms
Total hot run time: 203289 ms

doris-robot · 2024-06-28T08:45:25Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 33cfd3d5a54ee04864c9a1ef8796bddba0acb422, data reload: false
query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.78	0.07	0.06
query5	0.54	0.52	0.51
query6	1.25	0.61	0.61
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.49	0.48
query10	0.54	0.54	0.54
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.78	0.77	0.79
query15	0.77	0.75	0.76
query16	0.37	0.39	0.39
query17	0.95	0.99	1.02
query18	0.21	0.28	0.22
query19	1.96	1.82	1.85
query20	0.02	0.01	0.01
query21	15.46	0.57	0.56
query22	2.04	1.95	1.42
query23	16.67	1.11	0.85
query24	5.48	1.24	1.34
query25	0.41	0.07	0.05
query26	0.62	0.16	0.17
query27	0.04	0.03	0.04
query28	7.06	0.74	0.76
query29	12.63	2.36	2.22
query30	0.60	0.53	0.54
query31	2.81	0.38	0.38
query32	3.39	0.51	0.50
query33	3.04	2.99	3.06
query34	15.24	4.83	4.80
query35	4.88	4.86	4.85
query36	1.09	1.01	1.00
query37	0.05	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.06	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 102.76 s
Total hot run time: 30.62 s

doris-robot · 2024-06-28T08:48:39Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 33cfd3d5a54ee04864c9a1ef8796bddba0acb422 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.4 seconds inserted 10000000 Rows, about 467K ops/s

…alWindow and LogicalPartitionTopN

feiniaofeiafei · 2024-07-01T07:01:04Z

run buildall

doris-robot · 2024-07-01T07:21:38Z

TPC-H: Total hot run time: 50266 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37a204165305766aeaa64fdca3736bbeac53ed54, data reload: false
------ Round 1 ----------------------------------
q1	17823	4386	4376	4376
q2	2079	160	154	154
q3	10256	1945	1937	1937
q4	10106	1291	1292	1291
q5	8408	3956	3987	3956
q6	256	129	127	127
q7	2015	1623	1609	1609
q8	9302	2748	2724	2724
q9	11161	10727	10359	10359
q10	8656	3525	3478	3478
q11	422	242	245	242
q12	477	310	308	308
q13	18369	3956	4042	3956
q14	367	326	328	326
q15	503	462	459	459
q16	690	579	569	569
q17	1125	965	965	965
q18	7355	6978	6962	6962
q19	1790	1651	1629	1629
q20	572	317	309	309
q21	4437	4104	4093	4093
q22	530	442	437	437
Total cold run time: 116699 ms
Total hot run time: 50266 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4343	4308	4308	4308
q2	328	235	222	222
q3	4228	4204	4150	4150
q4	2750	2745	2739	2739
q5	7245	7262	7196	7196
q6	242	124	123	123
q7	3250	2876	2855	2855
q8	4341	4448	4456	4448
q9	17441	17215	17137	17137
q10	4210	4298	4265	4265
q11	767	713	691	691
q12	1030	858	878	858
q13	6886	3738	3723	3723
q14	454	437	430	430
q15	518	456	464	456
q16	739	687	668	668
q17	3924	3921	3842	3842
q18	8905	8795	8763	8763
q19	1718	1710	1687	1687
q20	2384	2154	2100	2100
q21	8422	8391	8538	8391
q22	1069	975	956	956
Total cold run time: 85194 ms
Total hot run time: 80008 ms

doris-robot · 2024-07-01T07:34:21Z

TPC-DS: Total hot run time: 203871 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37a204165305766aeaa64fdca3736bbeac53ed54, data reload: false
query1	929	396	410	396
query2	6527	2829	2652	2652
query3	6923	206	207	206
query4	20263	17870	17902	17870
query5	19724	6497	6535	6497
query6	309	224	228	224
query7	4159	302	314	302
query8	423	400	385	385
query9	3146	2724	2660	2660
query10	434	304	310	304
query11	11413	10745	10758	10745
query12	125	75	76	75
query13	5611	699	693	693
query14	17895	13434	13188	13188
query15	358	245	246	245
query16	6459	297	266	266
query17	1693	1465	884	884
query18	2295	420	412	412
query19	203	152	149	149
query20	75	81	81	81
query21	189	91	95	91
query22	5072	5004	4954	4954
query23	32759	31964	31933	31933
query24	7033	6489	6533	6489
query25	540	447	410	410
query26	539	163	159	159
query27	1876	294	293	293
query28	6140	2360	2328	2328
query29	2868	2844	2590	2590
query30	241	167	170	167
query31	902	738	762	738
query32	68	66	60	60
query33	406	262	255	255
query34	841	468	479	468
query35	1159	955	960	955
query36	1351	1281	1134	1134
query37	90	59	60	59
query38	3077	2966	2936	2936
query39	1364	1362	1319	1319
query40	204	94	98	94
query41	44	44	43	43
query42	88	94	81	81
query43	716	631	650	631
query44	1122	711	717	711
query45	249	236	233	233
query46	1225	966	986	966
query47	1756	1853	1694	1694
query48	1011	728	697	697
query49	613	383	380	380
query50	849	629	625	625
query51	4817	4625	4688	4625
query52	106	80	91	80
query53	451	320	315	315
query54	2652	2522	2455	2455
query55	99	74	80	74
query56	209	221	215	215
query57	1091	1141	1089	1089
query58	218	207	193	193
query59	4263	4030	4183	4030
query60	210	219	214	214
query61	100	94	96	94
query62	902	548	577	548
query63	485	345	343	343
query64	2445	1581	1353	1353
query65	3659	3571	3553	3553
query66	821	368	392	368
query67	19023	15614	15773	15614
query68	7957	658	642	642
query69	573	349	343	343
query70	1724	1422	1411	1411
query71	406	298	313	298
query72	6525	3546	3534	3534
query73	740	321	328	321
query74	6360	5837	5852	5837
query75	4679	3750	3692	3692
query76	4558	1137	1214	1137
query77	546	259	272	259
query78	12743	11968	11688	11688
query79	8725	644	638	638
query80	3055	420	407	407
query81	525	233	230	230
query82	1615	97	97	97
query83	201	135	131	131
query84	256	72	71	71
query85	1443	324	321	321
query86	371	290	296	290
query87	3240	3006	3042	3006
query88	5336	2344	2313	2313
query89	383	281	272	272
query90	1726	212	223	212
query91	170	138	137	137
query92	58	52	55	52
query93	5030	559	555	555
query94	899	214	219	214
query95	1094	1060	1033	1033
query96	652	316	326	316
query97	6598	6481	6484	6481
query98	202	185	175	175
query99	2891	868	884	868
Total cold run time: 316188 ms
Total hot run time: 203871 ms

doris-robot · 2024-07-01T07:39:48Z

ClickBench: Total hot run time: 30.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37a204165305766aeaa64fdca3736bbeac53ed54, data reload: false
query1	0.02	0.02	0.02
query2	0.08	0.02	0.02
query3	0.25	0.05	0.05
query4	1.78	0.06	0.07
query5	0.55	0.52	0.51
query6	1.25	0.62	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.49	0.48
query10	0.54	0.52	0.54
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.62	0.61
query14	0.79	0.79	0.78
query15	0.78	0.75	0.77
query16	0.36	0.37	0.38
query17	1.02	0.99	1.01
query18	0.22	0.26	0.25
query19	1.94	1.85	1.89
query20	0.02	0.01	0.01
query21	15.45	0.56	0.54
query22	2.01	2.26	1.43
query23	17.22	0.93	0.87
query24	8.13	0.83	0.65
query25	0.35	0.09	0.05
query26	0.78	0.15	0.16
query27	0.04	0.04	0.04
query28	5.64	0.80	0.77
query29	12.65	2.22	2.26
query30	0.55	0.50	0.48
query31	2.81	0.39	0.37
query32	3.39	0.50	0.49
query33	3.09	3.07	3.02
query34	15.26	4.85	4.80
query35	4.90	4.86	4.89
query36	1.07	1.02	1.01
query37	0.05	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.74 s
Total hot run time: 30.08 s

doris-robot · 2024-07-01T07:43:00Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 37a204165305766aeaa64fdca3736bbeac53ed54 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s

morrySnow · 2024-07-04T12:11:48Z

run external

morrySnow · 2024-07-05T04:59:00Z

run external

…alWindow and LogicalPartitionTopN (apache#36984) cherry-pick apache#36828 to branch-2.0 The requirement for predicate pushdown through the window operator is that the partition by slots of the window contains all slots in the predicate. The original implementation of doris only allows predicate pushdown with one slot. This PR relaxes this restriction and allows for predicate pushdown with multiple slots. The same applies to the predicate pushdown of the LogicalPartitionTopN operator. The following sql is an example. select * from ( select row_number() over(partition by id, value1 order by value1) as num, id, value1 from push_down_multi_column_predicate_through_window_t ) t where abs(id + value1)<4 and num <= 2; Co-authored-by: feiniaofeiafei <moailing@selectdb.com>

github-actionsBot added area/nereids kind/test labels Jun 28, 2024

[feat](Nereids) push down predicates with multi columns through Logic…
91dffca
…alWindow and LogicalPartitionTopN

[feat](Nereids) push down predicates with multi columns through Logic…
33cfd3d
…alWindow and LogicalPartitionTopN

[feat](Nereids) push down predicates with multi columns through Logic…
37a2041
…alWindow and LogicalPartitionTopN

morrySnow merged commit a90b363 into apache:branch-2.0Jul 5, 2024

Uh oh!

Conversation

feiniaofeiafei commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

feiniaofeiafei commented Jun 28, 2024

Uh oh!

feiniaofeiafei commented Jun 28, 2024

Uh oh!

feiniaofeiafei commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

feiniaofeiafei commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

feiniaofeiafei commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

feiniaofeiafei commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

morrySnow commented Jul 4, 2024

Uh oh!

morrySnow commented Jul 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants