[refactor](topn) Refactor topn filter push down by Gabriel39 · Pull Request #59005 · apache/doris

Gabriel39 · 2025-12-12T08:48:35Z

What problem does this PR solve?

TopN filter should be push down from scan operators like other predicates.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-12-12T08:48:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2025-12-12T09:16:36Z

run buildall

Gabriel39 · 2025-12-15T01:37:39Z

run buildall

doris-robot · 2025-12-15T03:10:29Z

TPC-H: Total hot run time: 35493 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b89dc18ec47b422139172c46bb4a56d7e5d31a30, data reload: false
------ Round 1 ----------------------------------
q1	17628	4274	4073	4073
q2	2011	341	236	236
q3	10190	1408	776	776
q4	10229	965	326	326
q5	7526	2192	2104	2104
q6	184	168	137	137
q7	1010	877	712	712
q8	9363	1483	1211	1211
q9	7276	5358	5384	5358
q10	6877	2426	1950	1950
q11	517	322	317	317
q12	678	740	586	586
q13	17789	3678	3053	3053
q14	288	305	284	284
q15	599	520	523	520
q16	702	672	638	638
q17	697	837	478	478
q18	7602	7154	7174	7154
q19	1104	974	650	650
q20	408	376	258	258
q21	4523	3930	3701	3701
q22	1031	1017	971	971
Total cold run time: 108232 ms
Total hot run time: 35493 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4119	4026	4073	4026
q2	324	411	312	312
q3	2154	2709	2275	2275
q4	1335	1762	1320	1320
q5	4247	4787	4707	4707
q6	232	177	135	135
q7	2064	1990	1830	1830
q8	2670	2465	2471	2465
q9	7793	7658	7615	7615
q10	3070	3266	2900	2900
q11	616	523	500	500
q12	784	737	622	622
q13	3621	4130	3449	3449
q14	303	298	280	280
q15	575	534	515	515
q16	640	666	640	640
q17	1227	1438	1410	1410
q18	7852	7800	7627	7627
q19	938	926	855	855
q20	1886	2004	1826	1826
q21	4635	4287	4151	4151
q22	1126	1059	1011	1011
Total cold run time: 52211 ms
Total hot run time: 50471 ms

doris-robot · 2025-12-15T03:21:39Z

TPC-DS: Total hot run time: 179478 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b89dc18ec47b422139172c46bb4a56d7e5d31a30, data reload: false
query5	5007	639	479	479
query6	352	230	215	215
query7	4221	464	266	266
query8	326	265	238	238
query9	8776	2554	2589	2554
query10	561	376	326	326
query11	15355	14924	14792	14792
query12	180	114	111	111
query13	1252	493	402	402
query14	6084	3243	3041	3041
query14_1	2939	2909	2899	2899
query15	213	195	185	185
query16	917	507	484	484
query17	1194	709	586	586
query18	2540	437	337	337
query19	233	232	214	214
query20	126	121	112	112
query21	216	140	112	112
query22	4125	4207	3986	3986
query23	16577	16383	16019	16019
query23_1	16203	16303	16084	16084
query24	7341	1655	1215	1215
query24_1	1220	1228	1238	1228
query25	554	463	453	453
query26	1252	275	165	165
query27	2752	469	307	307
query28	4476	2144	2125	2125
query29	798	545	442	442
query30	316	248	215	215
query31	808	708	634	634
query32	80	68	69	68
query33	541	341	283	283
query34	907	921	535	535
query35	797	815	717	717
query36	870	914	830	830
query37	123	93	77	77
query38	2919	2872	2826	2826
query39	779	740	731	731
query39_1	700	699	703	699
query40	228	133	122	122
query41	68	65	79	65
query42	111	111	105	105
query43	439	449	406	406
query44	1329	742	757	742
query45	194	190	183	183
query46	882	981	615	615
query47	1660	1723	1638	1638
query48	328	351	246	246
query49	626	454	345	345
query50	663	289	220	220
query51	3805	3842	3908	3842
query52	109	110	101	101
query53	319	358	295	295
query54	292	270	253	253
query55	82	72	71	71
query56	309	321	304	304
query57	1167	1143	1083	1083
query58	276	270	254	254
query59	2438	2473	2456	2456
query60	325	339	303	303
query61	198	194	191	191
query62	719	668	627	627
query63	332	297	298	297
query64	5080	1430	1127	1127
query65	4012	4017	3989	3989
query66	1456	471	342	342
query67	15313	15213	14812	14812
query68	7743	1025	730	730
query69	516	361	322	322
query70	1110	996	1017	996
query71	390	316	293	293
query72	6061	4947	4925	4925
query73	664	580	303	303
query74	8877	8902	8688	8688
query75	3173	3141	2764	2764
query76	3853	1145	740	740
query77	532	391	314	314
query78	9527	9750	8987	8987
query79	1710	885	617	617
query80	735	639	561	561
query81	514	267	234	234
query82	206	134	105	105
query83	261	263	242	242
query84	262	127	95	95
query85	906	505	452	452
query86	387	297	290	290
query87	3154	3080	2979	2979
query88	3202	2294	2276	2276
query89	466	426	395	395
query90	2080	157	155	155
query91	176	172	151	151
query92	79	66	70	66
query93	1180	905	569	569
query94	476	315	277	277
query95	568	335	326	326
query96	595	457	205	205
query97	2243	2308	2231	2231
query98	215	193	195	193
query99	1299	1308	1260	1260
Total cold run time: 260487 ms
Total hot run time: 179478 ms

doris-robot · 2025-12-15T03:26:41Z

ClickBench: Total hot run time: 27.47 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b89dc18ec47b422139172c46bb4a56d7e5d31a30, data reload: false
query1	0.05	0.05	0.05
query2	0.10	0.05	0.04
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.27	0.28
query6	1.17	0.63	0.64
query7	0.04	0.03	0.03
query8	0.05	0.05	0.04
query9	0.58	0.51	0.50
query10	0.55	0.57	0.56
query11	0.15	0.10	0.11
query12	0.14	0.12	0.12
query13	0.62	0.60	0.60
query14	1.00	0.98	0.99
query15	0.81	0.80	0.81
query16	0.42	0.40	0.40
query17	1.01	1.04	0.99
query18	0.23	0.21	0.22
query19	1.82	1.79	1.81
query20	0.01	0.01	0.01
query21	15.45	0.31	0.14
query22	4.72	0.05	0.05
query23	16.17	0.30	0.10
query24	0.93	0.45	0.59
query25	0.09	0.08	0.05
query26	0.14	0.13	0.14
query27	0.11	0.08	0.04
query28	4.31	1.23	1.02
query29	12.58	4.08	3.22
query30	0.27	0.13	0.12
query31	2.82	0.63	0.40
query32	3.24	0.54	0.46
query33	2.98	3.01	3.04
query34	17.03	5.20	4.56
query35	4.55	4.56	4.59
query36	0.68	0.49	0.49
query37	0.10	0.07	0.07
query38	0.07	0.05	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.51 s
Total hot run time: 27.47 s

hello-stephen · 2025-12-15T04:24:46Z

BE UT Coverage Report

Increment line coverage 36.84% (28/76) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.43% (18835/35250)
Line Coverage	39.19% (174197/444456)
Region Coverage	33.84% (134913/398717)
Branch Coverage	34.78% (58043/166903)

hello-stephen · 2025-12-15T07:48:40Z

BE Regression && UT Coverage Report

Increment line coverage 85.33% (64/75) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	63.20% (21835/34548)
Line Coverage	49.81% (221161/443967)
Region Coverage	45.63% (184096/403439)
Branch Coverage	46.30% (77688/167780)

Gabriel39 · 2025-12-15T09:47:05Z

run buildall

doris-robot · 2025-12-15T11:43:02Z

TPC-H: Total hot run time: 35151 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e8b9e60111867241f51f5322814203aa3e9104b, data reload: false
------ Round 1 ----------------------------------
q1	17613	4313	4094	4094
q2	2046	359	238	238
q3	10142	1357	753	753
q4	10215	862	310	310
q5	7520	2176	1934	1934
q6	193	178	141	141
q7	1010	891	721	721
q8	9378	1468	1161	1161
q9	7268	5339	5301	5301
q10	6842	2423	1962	1962
q11	521	326	316	316
q12	637	716	572	572
q13	17793	3637	3050	3050
q14	296	309	273	273
q15	612	530	513	513
q16	689	680	620	620
q17	699	832	464	464
q18	7480	7046	7012	7012
q19	1114	968	615	615
q20	399	373	249	249
q21	4335	3878	3878	3878
q22	1046	1000	974	974
Total cold run time: 107848 ms
Total hot run time: 35151 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4102	4064	4054	4054
q2	348	420	319	319
q3	2172	2667	2280	2280
q4	1327	1767	1318	1318
q5	4281	4631	4863	4631
q6	236	178	136	136
q7	2072	1997	1804	1804
q8	2669	2608	2585	2585
q9	7535	7637	7529	7529
q10	3142	3278	2851	2851
q11	602	517	508	508
q12	656	781	752	752
q13	3793	3881	3416	3416
q14	291	311	298	298
q15	577	517	511	511
q16	608	657	662	657
q17	1212	1567	1492	1492
q18	7811	7676	7599	7599
q19	898	887	874	874
q20	2102	2077	1938	1938
q21	4870	4246	4193	4193
q22	1097	1036	979	979
Total cold run time: 52401 ms
Total hot run time: 50724 ms

doris-robot · 2025-12-15T11:54:03Z

TPC-DS: Total hot run time: 178751 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e8b9e60111867241f51f5322814203aa3e9104b, data reload: false
query5	4422	615	491	491
query6	331	250	241	241
query7	4219	473	291	291
query8	341	262	245	245
query9	8755	2577	2572	2572
query10	534	376	343	343
query11	15510	15427	14733	14733
query12	182	117	121	117
query13	1274	496	413	413
query14	6166	3267	3128	3128
query14_1	2882	2913	2927	2913
query15	200	194	180	180
query16	771	476	448	448
query17	1115	699	571	571
query18	2514	425	329	329
query19	222	219	197	197
query20	123	112	110	110
query21	219	142	113	113
query22	3870	4223	3780	3780
query23	16476	16200	15908	15908
query23_1	16112	15935	16097	15935
query24	7403	1667	1230	1230
query24_1	1242	1262	1237	1237
query25	568	469	407	407
query26	1257	268	158	158
query27	2782	470	307	307
query28	4477	2153	2139	2139
query29	807	566	439	439
query30	315	240	224	224
query31	803	690	630	630
query32	73	67	68	67
query33	552	326	293	293
query34	910	906	546	546
query35	776	819	719	719
query36	874	904	826	826
query37	131	94	74	74
query38	2844	2877	2804	2804
query39	760	740	719	719
query39_1	708	696	712	696
query40	216	137	116	116
query41	64	61	64	61
query42	107	104	108	104
query43	413	427	400	400
query44	1327	749	753	749
query45	195	193	180	180
query46	869	976	625	625
query47	1664	1669	1631	1631
query48	314	322	246	246
query49	641	427	365	365
query50	665	292	231	231
query51	3792	3899	3773	3773
query52	109	109	100	100
query53	314	347	287	287
query54	289	262	266	262
query55	89	76	75	75
query56	292	298	292	292
query57	1140	1139	1068	1068
query58	269	253	281	253
query59	2382	2434	2344	2344
query60	307	317	296	296
query61	160	162	157	157
query62	703	663	642	642
query63	328	292	306	292
query64	4911	1281	1010	1010
query65	3991	3925	3962	3925
query66	1443	458	310	310
query67	15301	15077	14875	14875
query68	2759	1042	774	774
query69	438	346	316	316
query70	1044	986	993	986
query71	320	316	291	291
query72	6258	5200	5348	5200
query73	553	646	312	312
query74	8789	8813	8674	8674
query75	3108	3155	2793	2793
query76	2864	1141	735	735
query77	370	392	310	310
query78	9500	9795	8852	8852
query79	1117	863	710	710
query80	1343	659	584	584
query81	551	271	242	242
query82	1320	138	108	108
query83	326	251	235	235
query84	255	120	95	95
query85	961	535	459	459
query86	395	294	295	294
query87	3062	3109	2951	2951
query88	3272	2265	2259	2259
query89	471	416	395	395
query90	1967	158	155	155
query91	168	172	144	144
query92	67	71	67	67
query93	963	941	555	555
query94	534	313	282	282
query95	563	342	309	309
query96	594	462	210	210
query97	2313	2279	2212	2212
query98	217	197	192	192
query99	1304	1317	1190	1190
Total cold run time: 253348 ms
Total hot run time: 178751 ms

doris-robot · 2025-12-15T11:59:07Z

ClickBench: Total hot run time: 27.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e8b9e60111867241f51f5322814203aa3e9104b, data reload: false
query1	0.05	0.05	0.05
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.62	0.11	0.10
query5	0.28	0.26	0.25
query6	1.17	0.63	0.63
query7	0.04	0.03	0.03
query8	0.05	0.04	0.05
query9	0.56	0.50	0.51
query10	0.57	0.56	0.56
query11	0.15	0.10	0.10
query12	0.15	0.12	0.12
query13	0.62	0.59	0.60
query14	0.99	0.99	0.98
query15	0.82	0.81	0.81
query16	0.40	0.39	0.42
query17	1.03	1.03	1.04
query18	0.22	0.21	0.22
query19	1.92	1.83	1.75
query20	0.02	0.02	0.01
query21	15.47	0.30	0.15
query22	4.81	0.05	0.05
query23	16.04	0.28	0.10
query24	1.16	0.47	0.66
query25	0.11	0.08	0.05
query26	0.13	0.13	0.13
query27	0.06	0.05	0.08
query28	4.78	1.22	1.03
query29	12.60	4.10	3.33
query30	0.28	0.13	0.12
query31	2.83	0.64	0.39
query32	3.22	0.55	0.46
query33	3.08	3.07	3.06
query34	16.57	5.14	4.56
query35	4.55	4.60	4.56
query36	0.68	0.50	0.50
query37	0.10	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 97.94 s
Total hot run time: 27.65 s

doris-robot · 2025-12-15T13:07:22Z

BE UT Coverage Report

Increment line coverage 35.44% (28/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.43% (18848/35279)
Line Coverage	39.22% (174452/444820)
Region Coverage	33.86% (135077/398973)
Branch Coverage	34.79% (58101/167014)

hello-stephen · 2025-12-15T17:02:28Z

BE Regression && UT Coverage Report

Increment line coverage 87.34% (69/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.36% (25015/34572)
Line Coverage	59.11% (262588/444206)
Region Coverage	54.05% (218165/403602)
Branch Coverage	55.57% (93256/167827)

github-actions · 2025-12-16T03:42:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-12-16T03:42:38Z

PR approved by anyone and no changes requested.

yiguolei · 2025-12-16T03:50:47Z

+ int64_t index = DORIS_TRY(_contexts[target_node_id].get_field_index());
+ DCHECK(_contexts[target_node_id].predicate != nullptr);
+ assert_cast<SharedPredicate*>(_contexts[target_node_id].predicate.get())
+ ->set_column_id(cast_set<uint32_t>(index));


这里为什么要setcolumnid？

yiguolei · 2025-12-16T03:55:02Z

 _parent->cast<typename Derived::Parent>()._slot_id_to_slot_desc[slot->slot_id()];
 return _is_predicate_acting_on_slot(slot, range);
 };
+ auto topn_predicate_checker = [&](const vectorized::VExprSPtrs& children,


我们为什么在一个函数里，定义这么多lambda，而不是写多个函数呢？

yiguolei · 2025-12-16T08:22:37Z

+ slot_id_to_slot_desc[get_texpr(target_node_id).nodes[0].slot_ref.slot_id]
+ ->col_name(),
+ desc.debug_string(),
+ slot_id_to_slot_desc[get_texpr(target_node_id).nodes[0].slot_ref.slot_id]


我们这个可能不对，之前的可能是对的

TopN filter should be push down from scan operators like other predicates.

### What problem does this PR solve? pick #57397 #58283 #58290 #58282 #58832 #58905 #58960 #59005 #59088 #59098 #59126 #59187 #59581 #59625 #59775 ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

### What problem does this PR solve? pick apache#57397 apache#58283 apache#58290 apache#58282 apache#58832 apache#58905 apache#58960 apache#59005 apache#59088 apache#59098 apache#59126 apache#59187 apache#59581 apache#59625 apache#59775 ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

Gabriel39 added 3 commits December 15, 2025 09:36

[refactor](topn) Refactor topn filter push down
9114542

update
20f3d77

update
b89dc18

Gabriel39 force-pushed the dev_1212 branch from d3aea19 to b89dc18Compare December 15, 2025 01:37

update
4e8b9e6

BiteTheDDDDt approved these changes Dec 16, 2025
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 16, 2025

yiguolei reviewed Dec 16, 2025
View reviewed changes

Mryange approved these changes Dec 16, 2025
View reviewed changes

Gabriel39 merged commit 232a5a2 into apache:masterDec 16, 2025
29 of 33 checks passed

yiguolei reviewed Dec 16, 2025
View reviewed changes

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jan 13, 2026
[refactor](topn) Refactor topn filter push down (apache#59005)
370693b
TopN filter should be push down from scan operators like other predicates.

Gabriel39 mentioned this pull request Jan 13, 2026
[branch-4.0]Pick predicates-related PR #59804
Merged
16 tasks

yiguolei added the dev/4.0.3-merged label Jan 14, 2026

Gabriel39 deleted the dev_1212 branch July 30, 2026 06:26

Uh oh!

Conversation

Gabriel39 commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Dec 12, 2025

Uh oh!

Gabriel39 commented Dec 12, 2025

Uh oh!

Gabriel39 commented Dec 15, 2025

Uh oh!

doris-robot commented Dec 15, 2025

Uh oh!

doris-robot commented Dec 15, 2025

Uh oh!

doris-robot commented Dec 15, 2025

Uh oh!

hello-stephen commented Dec 15, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 15, 2025

BE Regression && UT Coverage Report

Uh oh!

Gabriel39 commented Dec 15, 2025

Uh oh!

doris-robot commented Dec 15, 2025

Uh oh!

doris-robot commented Dec 15, 2025

Uh oh!

doris-robot commented Dec 15, 2025

Uh oh!

doris-robot commented Dec 15, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 15, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Dec 16, 2025

Uh oh!

github-actionsBot commented Dec 16, 2025

Uh oh!

yiguoleiDec 16, 2025

Choose a reason for hiding this comment

Uh oh!

yiguoleiDec 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiguoleiDec 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Gabriel39 commented Dec 12, 2025•
edited
Loading