[nereids](topn-filter) support multi-topn filter (FE part) by englefly · Pull Request #31485 · apache/doris

englefly · 2024-02-27T11:39:57Z

Proposed changes

in previous version, topN should be the parent of scan. this restriction is removed.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-27T11:40:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

englefly · 2024-02-28T02:49:11Z

run buildall

englefly · 2024-02-28T03:00:55Z

run buildall

doris-robot · 2024-02-28T03:22:06Z

TPC-H: Total hot run time: 37960 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e99065b0563ce814a7e3123760814e39e9c96185, data reload: false
------ Round 1 ----------------------------------
q1	17678	4058	4020	4020
q2	2040	139	134	134
q3	10579	950	941	941
q4	4657	948	950	948
q5	7582	2903	3015	2903
q6	181	120	120	120
q7	1246	766	749	749
q8	9388	2023	2017	2017
q9	7389	6289	6283	6283
q10	8199	2514	2508	2508
q11	426	198	209	198
q12	765	319	316	316
q13	17952	3207	3191	3191
q14	282	253	242	242
q15	536	494	496	494
q16	467	399	432	399
q17	932	896	855	855
q18	6839	6063	6051	6051
q19	1571	1503	1521	1503
q20	540	276	268	268
q21	6724	3596	3530	3530
q22	788	295	290	290
Total cold run time: 106761 ms
Total hot run time: 37960 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4021	3996	3998	3996
q2	306	209	218	209
q3	2967	2974	2955	2955
q4	1827	1808	1835	1808
q5	5085	5068	5076	5068
q6	197	114	114	114
q7	2133	1698	1671	1671
q8	3118	3179	3210	3179
q9	8276	8230	8218	8218
q10	6128	3663	3652	3652
q11	499	436	432	432
q12	679	523	509	509
q13	6060	3063	3034	3034
q14	268	247	266	247
q15	540	495	487	487
q16	471	460	451	451
q17	1710	1680	1676	1676
q18	7919	7679	7496	7496
q19	1670	1646	1631	1631
q20	2099	1907	1896	1896
q21	4738	4725	4641	4641
q22	500	439	445	439
Total cold run time: 61211 ms
Total hot run time: 53809 ms

doris-robot · 2024-02-28T03:32:45Z

TPC-DS: Total hot run time: 168890 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e99065b0563ce814a7e3123760814e39e9c96185, data reload: false
query1	939	338	338	338
query2	6533	1701	1748	1701
query3	6701	216	210	210
query4	23355	20615	20516	20516
query5	4345	366	360	360
query6	262	165	174	165
query7	4613	307	304	304
query8	264	189	203	189
query9	8472	2238	2228	2228
query10	406	224	217	217
query11	14557	14182	14067	14067
query12	136	84	79	79
query13	1657	435	428	428
query14	8313	6152	6704	6152
query15	211	169	178	169
query16	7085	260	254	254
query17	1037	523	508	508
query18	1892	263	272	263
query19	191	148	150	148
query20	88	82	82	82
query21	195	121	115	115
query22	4789	4624	4554	4554
query23	30656	29960	30001	29960
query24	12043	3111	3089	3089
query25	667	359	356	356
query26	1892	153	168	153
query27	3052	324	322	322
query28	7056	1819	1804	1804
query29	1282	567	565	565
query30	289	132	140	132
query31	877	679	692	679
query32	95	61	56	56
query33	724	240	225	225
query34	1038	489	490	489
query35	856	738	754	738
query36	950	814	884	814
query37	271	62	62	62
query38	3087	3063	3037	3037
query39	1334	1262	1241	1241
query40	282	101	101	101
query41	38	36	37	36
query42	107	99	98	98
query43	465	418	425	418
query44	1064	694	699	694
query45	188	179	175	175
query46	1041	784	806	784
query47	1503	1475	1501	1475
query48	418	355	339	339
query49	1180	298	309	298
query50	782	384	382	382
query51	4416	4255	4317	4255
query52	108	89	88	88
query53	332	265	265	265
query54	295	233	228	228
query55	87	83	82	82
query56	219	196	212	196
query57	991	871	872	871
query58	214	190	197	190
query59	2233	2026	2083	2026
query60	244	219	214	214
query61	86	85	89	85
query62	597	366	364	364
query63	288	258	254	254
query64	6531	3032	3123	3032
query65	3209	3200	3218	3200
query66	1440	316	313	313
query67	14382	13889	14084	13889
query68	4906	540	557	540
query69	535	376	375	375
query70	1225	1209	1189	1189
query71	391	259	249	249
query72	6206	2820	2688	2688
query73	706	317	324	317
query74	6346	5981	5920	5920
query75	3114	2459	2408	2408
query76	2796	1049	1188	1049
query77	350	242	225	225
query78	9002	8607	8531	8531
query79	968	542	523	523
query80	512	364	346	346
query81	450	198	199	198
query82	240	89	85	85
query83	141	127	117	117
query84	258	80	79	79
query85	1057	359	341	341
query86	302	265	293	265
query87	3259	3076	3081	3076
query88	2744	2369	2313	2313
query89	391	321	343	321
query90	1809	166	167	166
query91	151	124	136	124
query92	55	49	47	47
query93	1018	523	506	506
query94	1100	178	183	178
query95	446	341	347	341
query96	568	266	266	266
query97	4252	4107	4141	4107
query98	215	204	196	196
query99	1050	699	683	683
Total cold run time: 261513 ms
Total hot run time: 168890 ms

doris-robot · 2024-02-28T03:38:31Z

ClickBench: Total hot run time: 32.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e99065b0563ce814a7e3123760814e39e9c96185, data reload: false
query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.52	0.52
query6	1.27	0.69	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.57	0.52	0.50
query10	0.56	0.55	0.56
query11	0.13	0.10	0.10
query12	0.12	0.10	0.11
query13	0.63	0.62	0.61
query14	0.80	0.81	0.80
query15	0.84	0.83	0.81
query16	0.36	0.39	0.38
query17	1.02	0.99	0.98
query18	0.27	0.26	0.25
query19	1.86	1.74	1.70
query20	0.02	0.01	0.02
query21	15.40	0.68	0.68
query22	3.43	4.00	2.62
query23	17.34	1.03	1.10
query24	16.20	0.67	0.68
query25	0.32	0.12	0.05
query26	0.90	0.16	0.16
query27	0.06	0.05	0.04
query28	8.91	0.88	0.89
query29	12.75	3.42	3.20
query30	0.63	0.55	0.52
query31	2.79	0.35	0.36
query32	3.34	0.49	0.48
query33	3.15	3.25	3.21
query34	15.36	4.74	4.73
query35	4.74	4.75	4.76
query36	1.14	1.06	1.07
query37	0.07	0.05	0.05
query38	0.03	0.02	0.03
query39	0.03	0.02	0.01
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 117.87 s
Total hot run time: 32.91 s

doris-robot · 2024-02-28T03:41:43Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e99065b0563ce814a7e3123760814e39e9c96185 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 17.8 seconds inserted 10000000 Rows, about 561K ops/s

englefly · 2024-02-28T03:59:31Z

run buildall

doris-robot · 2024-02-28T04:18:51Z

TPC-H: Total hot run time: 38016 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba6b404e6a0c4a7267d303c3915e6f755dc504be, data reload: false
------ Round 1 ----------------------------------
q1	18002	4107	4142	4107
q2	2674	143	150	143
q3	12142	934	935	934
q4	4653	933	959	933
q5	7611	2820	3007	2820
q6	179	122	120	120
q7	1241	769	748	748
q8	9253	2001	2013	2001
q9	7663	6257	6234	6234
q10	8157	2504	2481	2481
q11	408	217	213	213
q12	770	315	316	315
q13	17944	3209	3212	3209
q14	276	254	249	249
q15	535	500	491	491
q16	471	410	404	404
q17	936	845	821	821
q18	6715	6272	6154	6154
q19	1571	1510	1507	1507
q20	551	281	268	268
q21	6674	3583	3568	3568
q22	792	296	297	296
Total cold run time: 109218 ms
Total hot run time: 38016 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4027	4027	4008	4008
q2	313	215	213	213
q3	2987	2964	2943	2943
q4	1857	1772	1836	1772
q5	5138	5150	5109	5109
q6	198	116	116	116
q7	2157	1746	1691	1691
q8	3120	3193	3221	3193
q9	8236	8215	8197	8197
q10	6167	3645	3654	3645
q11	525	427	422	422
q12	680	535	551	535
q13	5138	3098	3053	3053
q14	279	250	257	250
q15	531	490	485	485
q16	491	452	497	452
q17	1690	1673	1688	1673
q18	7702	7496	7503	7496
q19	1642	1633	1630	1630
q20	2106	1890	1907	1890
q21	4753	4511	4590	4511
q22	512	453	439	439
Total cold run time: 60249 ms
Total hot run time: 53723 ms

doris-robot · 2024-02-28T04:29:35Z

TPC-DS: Total hot run time: 170045 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba6b404e6a0c4a7267d303c3915e6f755dc504be, data reload: false
query1	928	343	341	341
query2	6523	1700	1774	1700
query3	6699	211	210	210
query4	23275	20592	20551	20551
query5	4307	379	367	367
query6	260	181	170	170
query7	4599	314	295	295
query8	265	194	198	194
query9	8422	2257	2245	2245
query10	422	224	220	220
query11	14675	14192	14174	14174
query12	138	87	86	86
query13	1649	441	434	434
query14	8620	6571	6308	6308
query15	211	177	174	174
query16	7078	260	253	253
query17	1044	529	522	522
query18	1894	268	273	268
query19	199	153	155	153
query20	91	86	85	85
query21	194	132	123	123
query22	4715	4638	4565	4565
query23	30672	29877	30005	29877
query24	11954	3312	3350	3312
query25	648	372	362	362
query26	1818	158	161	158
query27	3021	309	312	309
query28	7040	1793	1791	1791
query29	1218	568	563	563
query30	285	135	148	135
query31	877	685	692	685
query32	95	61	59	59
query33	725	239	232	232
query34	1027	489	507	489
query35	846	766	749	749
query36	895	863	852	852
query37	257	63	66	63
query38	3111	2999	3018	2999
query39	1321	1268	1246	1246
query40	287	111	104	104
query41	40	36	35	35
query42	103	96	98	96
query43	470	444	434	434
query44	1056	685	702	685
query45	196	181	174	174
query46	1107	841	835	835
query47	1523	1449	1470	1449
query48	423	350	367	350
query49	1192	308	322	308
query50	841	386	387	386
query51	4397	4327	4303	4303
query52	101	87	95	87
query53	357	266	269	266
query54	317	234	237	234
query55	88	84	84	84
query56	222	203	201	201
query57	959	834	901	834
query58	216	199	206	199
query59	2192	1974	1949	1949
query60	253	223	222	222
query61	90	84	82	82
query62	602	371	362	362
query63	294	261	265	261
query64	6531	3055	3279	3055
query65	3280	3222	3211	3211
query66	1429	316	314	314
query67	14239	14100	14159	14100
query68	4923	561	560	560
query69	518	377	379	377
query70	1259	1164	1257	1164
query71	323	254	252	252
query72	6181	2815	2686	2686
query73	754	323	319	319
query74	6295	6043	6028	6028
query75	3193	2451	2474	2451
query76	2768	1132	1279	1132
query77	350	239	240	239
query78	9001	8551	8333	8333
query79	1435	536	530	530
query80	1111	363	349	349
query81	476	202	202	202
query82	1179	87	87	87
query83	221	130	122	122
query84	283	81	79	79
query85	1320	351	346	346
query86	374	281	298	281
query87	3229	3139	3144	3139
query88	2858	2409	2380	2380
query89	411	339	345	339
query90	1873	167	169	167
query91	157	143	128	128
query92	56	53	51	51
query93	1263	526	534	526
query94	1245	184	188	184
query95	453	357	356	356
query96	645	275	272	272
query97	4375	4175	4214	4175
query98	225	208	195	195
query99	1093	692	657	657
Total cold run time: 264670 ms
Total hot run time: 170045 ms

doris-robot · 2024-02-28T04:35:20Z

ClickBench: Total hot run time: 32.47 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba6b404e6a0c4a7267d303c3915e6f755dc504be, data reload: false
query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.52	0.51
query6	1.30	0.67	0.68
query7	0.02	0.02	0.01
query8	0.03	0.02	0.02
query9	0.55	0.50	0.52
query10	0.57	0.55	0.56
query11	0.13	0.09	0.09
query12	0.12	0.10	0.10
query13	0.63	0.61	0.62
query14	0.79	0.80	0.80
query15	0.83	0.82	0.83
query16	0.38	0.36	0.38
query17	0.99	0.96	0.99
query18	0.27	0.24	0.25
query19	1.83	1.69	1.72
query20	0.02	0.01	0.01
query21	15.41	0.66	0.62
query22	3.46	4.02	2.60
query23	17.53	1.11	0.86
query24	16.01	0.68	0.64
query25	0.32	0.09	0.06
query26	0.93	0.14	0.15
query27	0.04	0.05	0.04
query28	9.20	0.87	0.86
query29	12.62	3.36	3.24
query30	0.59	0.51	0.52
query31	2.77	0.35	0.37
query32	3.38	0.47	0.47
query33	3.18	3.18	3.20
query34	15.36	4.74	4.71
query35	4.74	4.72	4.75
query36	1.14	1.07	1.05
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 118.07 s
Total hot run time: 32.47 s

doris-robot · 2024-02-28T04:38:28Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ba6b404e6a0c4a7267d303c3915e6f755dc504be with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 16.8 seconds inserted 10000000 Rows, about 595K ops/s

englefly · 2024-02-28T07:12:06Z

run feut

morrySnow

add enough test cases please

englefly · 2024-02-29T05:33:40Z

run buildall

englefly · 2024-03-05T03:59:16Z

run buildall

doris-robot · 2024-03-05T04:48:41Z

TPC-H: Total hot run time: 38314 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b64e2814ed384421c1d0232aab75c43d26bdc080, data reload: false
------ Round 1 ----------------------------------
q1	18179	4218	4120	4120
q2	2554	151	151	151
q3	11360	962	974	962
q4	5525	967	954	954
q5	7956	2986	3082	2986
q6	182	126	130	126
q7	1349	864	839	839
q8	10266	2081	2085	2081
q9	7262	6488	6443	6443
q10	8213	2562	2574	2562
q11	428	224	213	213
q12	801	316	312	312
q13	17961	2944	2947	2944
q14	280	252	254	252
q15	483	444	438	438
q16	484	393	398	393
q17	957	884	901	884
q18	6704	5968	5923	5923
q19	1580	1524	1515	1515
q20	539	275	285	275
q21	7574	3655	3700	3655
q22	785	297	286	286
Total cold run time: 111422 ms
Total hot run time: 38314 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4003	4030	3970	3970
q2	321	225	226	225
q3	2933	2892	2915	2892
q4	1874	1834	1779	1779
q5	5226	5253	5222	5222
q6	203	119	119	119
q7	2266	1842	1860	1842
q8	3249	3295	3299	3295
q9	8605	8490	8585	8490
q10	6231	3757	3750	3750
q11	538	458	449	449
q12	700	528	514	514
q13	6631	2799	2759	2759
q14	275	247	251	247
q15	471	441	432	432
q16	452	412	420	412
q17	1683	1675	1651	1651
q18	7793	7580	7242	7242
q19	1646	1612	1617	1612
q20	1948	1769	1763	1763
q21	4950	4757	4815	4757
q22	533	490	480	480
Total cold run time: 62531 ms
Total hot run time: 53902 ms

doris-robot · 2024-03-05T05:00:24Z

TPC-DS: Total hot run time: 176855 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b64e2814ed384421c1d0232aab75c43d26bdc080, data reload: false
query1	914	370	336	336
query2	7304	2120	2024	2024
query3	6709	224	214	214
query4	25338	20967	20975	20967
query5	4871	517	532	517
query6	267	187	175	175
query7	4643	302	285	285
query8	240	176	165	165
query9	8500	2182	2160	2160
query10	437	256	237	237
query11	14884	14490	14277	14277
query12	145	93	85	85
query13	1659	426	426	426
query14	9194	6838	6888	6838
query15	243	191	194	191
query16	7249	258	258	258
query17	916	575	545	545
query18	1920	277	266	266
query19	198	156	154	154
query20	92	84	89	84
query21	196	124	122	122
query22	4659	4470	4411	4411
query23	31479	30476	30820	30476
query24	12160	3109	2971	2971
query25	697	386	392	386
query26	1919	159	160	159
query27	3026	369	359	359
query28	6671	1812	1800	1800
query29	1298	617	605	605
query30	313	157	148	148
query31	938	738	756	738
query32	102	68	59	59
query33	774	277	264	264
query34	1008	451	470	451
query35	939	814	799	799
query36	962	820	856	820
query37	278	67	74	67
query38	3223	3098	3143	3098
query39	1408	1403	1371	1371
query40	301	122	114	114
query41	69	64	67	64
query42	107	104	101	101
query43	448	400	395	395
query44	1091	716	711	711
query45	205	197	197	197
query46	1035	799	762	762
query47	1624	1547	1571	1547
query48	413	353	352	352
query49	1202	351	338	338
query50	776	371	368	368
query51	6615	6560	6508	6508
query52	114	95	97	95
query53	345	286	287	286
query54	339	249	249	249
query55	83	90	87	87
query56	249	226	229	226
query57	1093	1000	1000	1000
query58	240	218	226	218
query59	2483	2376	2209	2209
query60	283	253	250	250
query61	115	112	110	110
query62	657	407	406	406
query63	308	278	279	278
query64	6418	3141	3183	3141
query65	3096	3010	3000	3000
query66	1490	340	340	340
query67	15146	14628	14305	14305
query68	13428	550	595	550
query69	734	396	385	385
query70	1404	1077	1086	1077
query71	634	277	273	273
query72	9955	2585	2519	2519
query73	3380	332	331	331
query74	7225	6819	6888	6819
query75	6908	2628	2647	2628
query76	8059	1090	1176	1090
query77	918	257	271	257
query78	10146	9605	9584	9584
query79	9971	525	540	525
query80	787	404	392	392
query81	473	214	212	212
query82	260	88	86	86
query83	276	147	151	147
query84	277	75	80	75
query85	1128	334	349	334
query86	365	299	303	299
query87	3352	3214	3191	3191
query88	3052	2312	2290	2290
query89	525	374	355	355
query90	2369	182	177	177
query91	160	129	126	126
query92	61	48	48	48
query93	3076	540	506	506
query94	1586	188	184	184
query95	448	339	352	339
query96	583	263	265	263
query97	3983	3906	3871	3871
query98	230	215	201	201
query99	1090	787	756	756
Total cold run time: 310961 ms
Total hot run time: 176855 ms

doris-robot · 2024-03-05T05:05:44Z

ClickBench: Total hot run time: 31.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b64e2814ed384421c1d0232aab75c43d26bdc080, data reload: false
query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.69	0.10	0.10
query5	0.50	0.52	0.51
query6	1.27	0.67	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.52	0.51
query10	0.58	0.58	0.55
query11	0.13	0.10	0.10
query12	0.12	0.10	0.10
query13	0.57	0.57	0.56
query14	0.72	0.76	0.75
query15	0.81	0.80	0.81
query16	0.37	0.39	0.36
query17	0.97	1.00	0.98
query18	0.26	0.25	0.25
query19	1.80	1.70	1.76
query20	0.01	0.01	0.02
query21	15.43	0.62	0.63
query22	3.50	3.29	3.35
query23	17.61	1.01	1.15
query24	2.11	0.60	0.58
query25	0.27	0.07	0.06
query26	0.16	0.14	0.13
query27	0.03	0.03	0.03
query28	11.98	0.85	0.83
query29	12.53	3.21	3.23
query30	0.58	0.60	0.55
query31	2.79	0.34	0.34
query32	3.40	0.45	0.43
query33	2.95	2.89	2.87
query34	15.54	4.29	4.28
query35	4.29	4.29	4.28
query36	1.07	1.02	1.02
query37	0.07	0.05	0.06
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.43 s
Total hot run time: 31.91 s

englefly · 2024-03-10T12:10:57Z

run buildall

doris-robot · 2024-03-10T12:28:08Z

TPC-H: Total hot run time: 37885 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58d72fc945526c3e0a83d3e2150065ba04fdfb1b, data reload: false
------ Round 1 ----------------------------------
q1	17862	4050	4115	4050
q2	2341	151	142	142
q3	10558	935	932	932
q4	4655	945	949	945
q5	7595	2831	2995	2831
q6	179	126	129	126
q7	1320	828	818	818
q8	9493	2067	2040	2040
q9	7247	6467	6443	6443
q10	8238	2581	2546	2546
q11	425	217	215	215
q12	808	330	310	310
q13	18007	2914	2974	2914
q14	284	256	263	256
q15	490	449	453	449
q16	490	392	409	392
q17	933	869	885	869
q18	6714	6113	5828	5828
q19	1557	1514	1514	1514
q20	561	296	287	287
q21	7569	3759	3674	3674
q22	801	330	304	304
Total cold run time: 108127 ms
Total hot run time: 37885 ms
----- Round 2, with runtime_filter_mode=off -----
q1	3975	3988	3978	3978
q2	326	245	237	237
q3	2935	2915	2902	2902
q4	1854	1791	1802	1791
q5	5211	5221	5219	5219
q6	200	120	116	116
q7	2249	1859	1814	1814
q8	3209	3246	3271	3246
q9	8523	8512	8528	8512
q10	6151	3748	3682	3682
q11	532	435	441	435
q12	708	559	551	551
q13	14635	2779	2817	2779
q14	274	248	251	248
q15	474	443	439	439
q16	454	402	408	402
q17	1678	1629	1655	1629
q18	7823	7502	7308	7308
q19	1611	1602	1624	1602
q20	1951	1766	1733	1733
q21	4969	4840	4766	4766
q22	554	490	473	473
Total cold run time: 70296 ms
Total hot run time: 53862 ms

doris-robot · 2024-03-10T12:40:23Z

TPC-DS: Total hot run time: 187179 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58d72fc945526c3e0a83d3e2150065ba04fdfb1b, data reload: false
query1	923	369	341	341
query2	7364	2078	2089	2078
query3	6705	213	212	212
query4	27428	21208	21176	21176
query5	4287	483	4879	483
query6	268	189	186	186
query7	4634	333	303	303
query8	237	167	181	167
query9	8428	2278	2299	2278
query10	454	244	268	244
query11	15124	14603	14403	14403
query12	134	84	84	84
query13	1646	439	431	431
query14	10629	9882	9789	9789
query15	250	226	194	194
query16	7915	301	285	285
query17	1600	601	582	582
query18	1959	295	287	287
query19	205	170	166	166
query20	104	92	91	91
query21	198	134	125	125
query22	4655	4534	4462	4462
query23	31555	30689	31126	30689
query24	11879	3069	3077	3069
query25	708	392	390	390
query26	1941	171	172	171
query27	2989	372	369	369
query28	6601	1890	1879	1879
query29	1171	617	596	596
query30	311	151	150	150
query31	934	727	753	727
query32	100	64	61	61
query33	763	276	267	267
query34	974	483	509	483
query35	860	635	653	635
query36	930	814	851	814
query37	283	80	81	80
query38	3206	3107	3054	3054
query39	1406	1356	1382	1356
query40	297	123	116	116
query41	76	64	70	64
query42	108	98	106	98
query43	440	404	410	404
query44	1106	722	712	712
query45	288	263	258	258
query46	1052	799	769	769
query47	1622	1550	1542	1542
query48	413	354	357	354
query49	1246	347	357	347
query50	778	386	389	386
query51	6766	6678	6813	6678
query52	102	99	120	99
query53	358	283	295	283
query54	321	270	245	245
query55	95	84	85	84
query56	242	229	231	229
query57	1097	1007	998	998
query58	245	222	224	222
query59	2538	2472	2278	2278
query60	297	259	268	259
query61	118	115	113	113
query62	678	433	407	407
query63	310	274	278	274
query64	6082	3003	3064	3003
query65	3032	3032	2986	2986
query66	1380	325	325	325
query67	15205	14585	14556	14556
query68	11907	575	599	575
query69	761	386	387	386
query70	1397	1136	1136	1136
query71	637	283	286	283
query72	9938	2617	2473	2473
query73	1760	339	343	339
query74	6996	6935	6758	6758
query75	12193	8026	8092	8026
query76	7973	1115	1188	1115
query77	922	258	257	257
query78	10160	9647	9501	9501
query79	10684	532	528	528
query80	751	409	383	383
query81	469	209	207	207
query82	333	200	200	200
query83	217	149	151	149
query84	289	84	78	78
query85	1101	343	335	335
query86	366	307	317	307
query87	3360	3163	3227	3163
query88	3062	2372	2368	2368
query89	519	372	361	361
query90	2196	187	185	185
query91	173	163	147	147
query92	66	55	53	53
query93	3772	542	534	534
query94	1396	209	202	202
query95	466	365	360	360
query96	601	265	268	265
query97	4441	4292	4342	4292
query98	234	225	217	217
query99	1114	763	750	750
Total cold run time: 318304 ms
Total hot run time: 187179 ms

doris-robot · 2024-03-10T12:45:45Z

ClickBench: Total hot run time: 31.17 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58d72fc945526c3e0a83d3e2150065ba04fdfb1b, data reload: false
query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.05	0.05
query4	1.68	0.10	0.10
query5	0.50	0.48	0.50
query6	1.30	0.68	0.67
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.57	0.51	0.53
query10	0.60	0.57	0.57
query11	0.12	0.10	0.10
query12	0.12	0.11	0.10
query13	0.58	0.57	0.56
query14	0.73	0.74	0.75
query15	0.84	0.79	0.81
query16	0.36	0.36	0.36
query17	1.00	0.96	1.00
query18	0.26	0.26	0.25
query19	1.77	1.69	1.74
query20	0.02	0.01	0.01
query21	15.43	0.67	0.68
query22	2.98	4.06	2.84
query23	17.50	1.00	0.87
query24	2.18	0.33	0.20
query25	0.29	0.05	0.09
query26	0.17	0.15	0.14
query27	0.03	0.03	0.03
query28	12.57	0.84	0.82
query29	12.56	3.37	3.37
query30	0.60	0.57	0.62
query31	2.80	0.34	0.35
query32	3.42	0.44	0.44
query33	2.89	2.94	2.93
query34	15.48	4.30	4.30
query35	4.30	4.28	4.33
query36	1.08	1.00	1.02
query37	0.08	0.05	0.05
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.03
Total cold run time: 105.57 s
Total hot run time: 31.17 s

doris-robot · 2024-03-10T12:48:52Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 58d72fc945526c3e0a83d3e2150065ba04fdfb1b with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 17.4 seconds inserted 10000000 Rows, about 574K ops/s

englefly · 2024-03-10T14:39:36Z

run buildall

doris-robot · 2024-03-10T15:07:28Z

TPC-H: Total hot run time: 38076 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14cadba94b44c2b5a8de8b2bc338dd64265c637f, data reload: false
------ Round 1 ----------------------------------
q1	17907	4128	4126	4126
q2	2380	179	143	143
q3	11438	983	962	962
q4	4975	946	967	946
q5	7935	2883	3057	2883
q6	182	130	131	130
q7	1291	854	814	814
q8	9502	2085	2061	2061
q9	7352	6465	6417	6417
q10	8222	2561	2542	2542
q11	421	227	229	227
q12	740	312	312	312
q13	17960	2929	2949	2929
q14	275	259	265	259
q15	487	447	450	447
q16	469	393	408	393
q17	943	867	854	854
q18	6728	5912	5863	5863
q19	1586	1517	1509	1509
q20	551	292	281	281
q21	7534	3715	3673	3673
q22	804	305	314	305
Total cold run time: 109682 ms
Total hot run time: 38076 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4006	3974	3991	3974
q2	323	226	229	226
q3	2932	2927	2909	2909
q4	1866	1844	1795	1795
q5	5262	5228	5252	5228
q6	201	117	117	117
q7	2284	1829	1843	1829
q8	3193	3241	3270	3241
q9	8491	8474	8484	8474
q10	6137	3747	3717	3717
q11	529	433	437	433
q12	688	512	512	512
q13	14934	2854	2772	2772
q14	281	248	259	248
q15	475	440	442	440
q16	443	397	428	397
q17	1684	1641	1639	1639
q18	7745	7515	7353	7353
q19	1599	1601	1595	1595
q20	1937	1729	1725	1725
q21	4862	4798	4716	4716
q22	531	481	475	475
Total cold run time: 70403 ms
Total hot run time: 53815 ms

doris-robot · 2024-03-10T15:19:49Z

TPC-DS: Total hot run time: 186315 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14cadba94b44c2b5a8de8b2bc338dd64265c637f, data reload: false
query1	936	364	343	343
query2	7391	2104	2049	2049
query3	6703	218	227	218
query4	26889	21048	21063	21048
query5	4402	1222	10679	1222
query6	270	183	177	177
query7	4640	309	307	307
query8	246	200	189	189
query9	8510	2276	2276	2276
query10	454	225	246	225
query11	14982	14178	14386	14178
query12	129	112	86	86
query13	1646	439	426	426
query14	10493	9842	9643	9643
query15	247	193	193	193
query16	7962	267	277	267
query17	1734	610	601	601
query18	2038	293	291	291
query19	214	167	158	158
query20	98	90	115	90
query21	200	133	124	124
query22	4657	4497	4426	4426
query23	31699	30379	30294	30294
query24	12306	3087	3042	3042
query25	684	387	406	387
query26	1925	164	170	164
query27	3066	360	367	360
query28	6498	1888	1873	1873
query29	1105	625	611	611
query30	318	155	147	147
query31	934	720	748	720
query32	101	63	59	59
query33	762	265	257	257
query34	1012	465	477	465
query35	866	629	642	629
query36	939	807	839	807
query37	271	81	77	77
query38	3213	3044	3071	3044
query39	1407	1321	1393	1321
query40	289	120	116	116
query41	71	64	69	64
query42	101	103	102	102
query43	441	402	398	398
query44	1087	726	724	724
query45	272	263	253	253
query46	1056	780	782	780
query47	1627	1535	1557	1535
query48	414	355	351	351
query49	1257	342	353	342
query50	793	381	387	381
query51	6733	6635	6683	6635
query52	104	94	92	92
query53	355	291	285	285
query54	337	250	251	250
query55	92	81	88	81
query56	244	230	230	230
query57	1083	993	1024	993
query58	255	219	219	219
query59	2453	2392	2302	2302
query60	267	267	249	249
query61	114	116	115	115
query62	638	419	425	419
query63	308	282	278	278
query64	6212	3031	3086	3031
query65	3075	3003	2975	2975
query66	1365	324	323	323
query67	15057	14513	14326	14326
query68	11220	568	596	568
query69	708	379	375	375
query70	1335	1093	1121	1093
query71	619	267	277	267
query72	9824	2621	2490	2490
query73	1770	332	329	329
query74	7206	6862	6906	6862
query75	11456	7945	7979	7945
query76	7931	1071	1165	1071
query77	932	253	261	253
query78	10312	9539	9552	9539
query79	11337	531	515	515
query80	1376	392	385	385
query81	476	211	205	205
query82	402	197	193	193
query83	223	145	146	145
query84	282	76	78	76
query85	1127	335	327	327
query86	379	309	284	284
query87	3441	3138	3260	3138
query88	3781	2353	2350	2350
query89	532	357	361	357
query90	2213	174	171	171
query91	160	128	134	128
query92	59	51	50	50
query93	6080	534	531	531
query94	1425	186	190	186
query95	444	347	337	337
query96	593	263	261	261
query97	4413	4272	4310	4272
query98	224	217	221	217
query99	1049	772	752	752
Total cold run time: 321376 ms
Total hot run time: 186315 ms

doris-robot · 2024-03-10T15:25:09Z

ClickBench: Total hot run time: 31.34 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 14cadba94b44c2b5a8de8b2bc338dd64265c637f, data reload: false
query1	0.04	0.03	0.02
query2	0.06	0.02	0.03
query3	0.22	0.06	0.06
query4	1.67	0.10	0.10
query5	0.50	0.47	0.48
query6	1.31	0.67	0.66
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.55	0.52	0.51
query10	0.57	0.56	0.57
query11	0.14	0.10	0.10
query12	0.12	0.10	0.10
query13	0.58	0.57	0.57
query14	0.74	0.76	0.74
query15	0.83	0.80	0.80
query16	0.37	0.39	0.38
query17	0.96	1.00	0.97
query18	0.25	0.26	0.26
query19	1.80	1.69	1.73
query20	0.01	0.02	0.01
query21	15.41	0.66	0.60
query22	2.86	4.19	3.08
query23	17.44	0.99	0.98
query24	2.14	0.22	0.34
query25	0.31	0.04	0.03
query26	0.18	0.13	0.14
query27	0.04	0.03	0.03
query28	12.58	0.82	0.84
query29	12.70	3.32	3.39
query30	0.56	0.55	0.54
query31	2.80	0.34	0.33
query32	3.38	0.43	0.43
query33	2.93	2.87	2.91
query34	15.48	4.34	4.33
query35	4.32	4.34	4.30
query36	1.09	1.00	1.02
query37	0.07	0.05	0.05
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.16	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.44 s
Total hot run time: 31.34 s

doris-robot · 2024-03-10T15:28:17Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 14cadba94b44c2b5a8de8b2bc338dd64265c637f with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 17.4 seconds inserted 10000000 Rows, about 574K ops/s

github-actions · 2024-03-11T02:09:51Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-11T02:09:54Z

PR approved by anyone and no changes requested.

support multi-topn-filter

…ode (#46551) ### What problem does this PR solve? Plan node is not good to be hash map key, because two plan nodes in different tree level may be regarded as "equal". for example, in following tree, topn1.equals(topn2) may be true. Topn filter generator should distinguish them, and hence topn node is not suitable to be used as hash map key. topn1 -->some node -->topn2 -->other node Related PR: #31485

…ode (apache#46551) ### What problem does this PR solve? Plan node is not good to be hash map key, because two plan nodes in different tree level may be regarded as "equal". for example, in following tree, topn1.equals(topn2) may be true. Topn filter generator should distinguish them, and hence topn node is not suitable to be used as hash map key. topn1 -->some node -->topn2 -->other node Related PR: apache#31485 (cherry picked from commit 811f936)

…ode (#46551) (branch-3.0) (#46585) pick#46551 ### What problem does this PR solve? Plan node is not good to be hash map key, because two plan nodes in different tree level may be regarded as "equal". for example, in following tree, topn1.equals(topn2) may be true. Topn filter generator should distinguish them, and hence topn node is not suitable to be used as hash map key. topn1 -->some node -->topn2 -->other node Related PR: #31485 (cherry picked from commit 811f936)

…y instead of Topn node apache#46551 (apache#3800) pick apache#46551 ### What problem does this PR solve? Plan node is not good to be hash map key, because two plan nodes in different tree level may be regarded as "equal". for example, in following tree, topn1.equals(topn2) may be true. Topn filter generator should distinguish them, and hence topn node is not suitable to be used as hash map key. topn1 -->some node -->topn2 -->other node Related PR: apache#31485

) support multi-topn-filter

englefly changed the title ~~topn-ext~~[nereids](topn-opt) topn-extFeb 28, 2024

englefly marked this pull request as ready for review February 28, 2024 02:48

englefly changed the title ~~[nereids](topn-opt) topn-ext~~[nereids](topn-opt) WIP topn-extFeb 28, 2024

xiaokang added the dev/2.0.x label Feb 28, 2024

englefly changed the title ~~[nereids](topn-opt) WIP topn-ext~~[nereids](topn-filter) remove some shape restrictionsFeb 29, 2024

morrySnow reviewed Feb 29, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java Outdated

englefly changed the title ~~[nereids](topn-filter) remove some shape restrictions~~[nereids](topn-filter) support multi-topn filter (FE part)Mar 4, 2024

englefly marked this pull request as draft March 5, 2024 02:01

englefly force-pushed the topn-ext branch 3 times, most recently from b9d47cb to b64e281Compare March 5, 2024 03:58

englefly marked this pull request as ready for review March 5, 2024 04:00

englefly force-pushed the topn-ext branch from dc6f0a4 to 58d72fcCompare March 10, 2024 12:10

fix
14cadba

starocean999 approved these changes Mar 11, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 11, 2024

BiteTheDDDDt approved these changes Mar 11, 2024
View reviewed changes

englefly merged commit 0865f74 into apache:masterMar 11, 2024

englefly deleted the topn-ext branch March 11, 2024 02:43

yiguolei pushed a commit that referenced this pull request Mar 12, 2024
[nereids](topn-filter) support multi-topn filter (FE part) (#31485)
ab21d85
support multi-topn-filter

morrySnow removed the dev/2.0.x label Mar 25, 2024

englefly mentioned this pull request Jan 8, 2025
[fix](nereids) topN filter: use ObjectId as map key instead of Topn node #46551
Merged
16 tasks

englefly mentioned this pull request Jan 8, 2025
[fix](nereids) topN filter: use ObjectId as map key instead of Topn node (#46551) (branch-3.0) #46585
Merged
16 tasks

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[nereids](topn-filter) support multi-topn filter (FE part) (apache#31485
28146a5
) support multi-topn-filter

Uh oh!

Conversation

englefly commented Feb 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 27, 2024

Uh oh!

englefly commented Feb 28, 2024

Uh oh!

englefly commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

englefly commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

doris-robot commented Feb 28, 2024

Uh oh!

englefly commented Feb 28, 2024

Uh oh!

morrySnow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

englefly commented Feb 29, 2024

Uh oh!

englefly commented Mar 5, 2024

Uh oh!

doris-robot commented Mar 5, 2024

Uh oh!

doris-robot commented Mar 5, 2024

Uh oh!

doris-robot commented Mar 5, 2024

Uh oh!

englefly commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

englefly commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

doris-robot commented Mar 10, 2024

Uh oh!

github-actionsBot commented Mar 11, 2024

Uh oh!

github-actionsBot commented Mar 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

englefly commented Feb 27, 2024•
edited
Loading