[opt](nereids) forbid topn-two-phase reading for some patterns by englefly · Pull Request #47975 · apache/doris

englefly · 2025-02-17T08:54:57Z

What problem does this PR solve?

when there is no deferred materialized slots, do not use deferred materailizion.
for example:
SELECT t1.id FROM t1 WHERE id = 1 ORDER BY id LIMIT 1;
there is no passive slots. slot "id" is the only slot need to be read by IO.

Issue Number: close #xxx

Related PR: #47661

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-02-17T08:55:09Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

englefly · 2025-02-17T08:55:12Z

run buildall

englefly · 2025-02-17T09:14:07Z

run buildall

doris-robot · 2025-02-17T09:38:42Z

TPC-H: Total hot run time: 31654 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b162672a53ab9a74da1a4b2a7abac202e8792910, data reload: false
------ Round 1 ----------------------------------
q1	17667	5186	5063	5063
q2	2044	314	178	178
q3	10379	1302	705	705
q4	10219	1002	539	539
q5	7547	2385	2370	2370
q6	189	168	134	134
q7	908	735	635	635
q8	9308	1280	972	972
q9	5020	4850	4847	4847
q10	6822	2339	1902	1902
q11	480	276	258	258
q12	342	351	213	213
q13	17748	3681	3097	3097
q14	228	231	204	204
q15	507	462	466	462
q16	611	592	564	564
q17	571	860	337	337
q18	6608	6382	6172	6172
q19	1211	951	537	537
q20	309	323	192	192
q21	2928	2158	1960	1960
q22	365	323	313	313
Total cold run time: 102011 ms
Total hot run time: 31654 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5137	5082	5251	5082
q2	238	339	234	234
q3	2165	2698	2325	2325
q4	1467	1838	1410	1410
q5	4211	4131	4179	4131
q6	203	160	121	121
q7	1845	1826	1628	1628
q8	2615	2628	2562	2562
q9	7299	7124	6993	6993
q10	2970	3176	2771	2771
q11	601	516	489	489
q12	703	765	627	627
q13	3570	3998	3216	3216
q14	294	295	269	269
q15	514	468	458	458
q16	645	674	642	642
q17	1120	1497	1390	1390
q18	7642	7277	7322	7277
q19	869	773	826	773
q20	2012	2026	1876	1876
q21	5382	4934	4836	4836
q22	634	575	547	547
Total cold run time: 52136 ms
Total hot run time: 49657 ms

doris-robot · 2025-02-17T09:50:06Z

TPC-DS: Total hot run time: 183632 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b162672a53ab9a74da1a4b2a7abac202e8792910, data reload: false
query1	981	397	373	373
query2	6542	1823	1843	1823
query3	6805	215	227	215
query4	26769	23302	23440	23302
query5	4336	661	487	487
query6	290	196	188	188
query7	4608	497	303	303
query8	294	233	213	213
query9	8620	2486	2481	2481
query10	472	315	263	263
query11	15508	15198	14882	14882
query12	159	103	103	103
query13	1645	509	388	388
query14	9721	6581	6476	6476
query15	208	197	172	172
query16	7714	616	458	458
query17	1286	725	558	558
query18	1995	402	301	301
query19	195	188	159	159
query20	117	114	114	114
query21	211	123	105	105
query22	4285	4298	4638	4298
query23	34300	33272	33030	33030
query24	7644	2363	2394	2363
query25	531	444	381	381
query26	1224	271	150	150
query27	2088	501	327	327
query28	3890	2386	2346	2346
query29	734	545	403	403
query30	225	180	151	151
query31	932	830	774	774
query32	72	60	61	60
query33	564	343	295	295
query34	805	849	487	487
query35	785	805	731	731
query36	975	965	879	879
query37	116	99	84	84
query38	4122	4150	4186	4150
query39	1438	1397	1402	1397
query40	218	116	108	108
query41	55	51	49	49
query42	119	102	96	96
query43	485	506	468	468
query44	1283	773	774	773
query45	175	167	158	158
query46	850	1023	653	653
query47	1771	1814	1724	1724
query48	380	408	304	304
query49	780	492	409	409
query50	685	730	417	417
query51	4164	4210	4113	4113
query52	108	114	105	105
query53	227	256	179	179
query54	476	476	405	405
query55	84	79	78	78
query56	257	264	256	256
query57	1129	1131	1065	1065
query58	257	234	244	234
query59	2720	2813	2697	2697
query60	286	273	267	267
query61	142	119	117	117
query62	810	721	657	657
query63	232	189	192	189
query64	4269	969	654	654
query65	3198	3123	3149	3123
query66	1056	384	302	302
query67	15674	15341	15312	15312
query68	6054	765	493	493
query69	470	293	260	260
query70	1215	1129	1100	1100
query71	395	286	262	262
query72	6081	3625	3758	3625
query73	749	743	345	345
query74	8916	9135	8962	8962
query75	3121	3171	2723	2723
query76	3254	1178	730	730
query77	479	409	285	285
query78	10011	10120	9315	9315
query79	2678	811	600	600
query80	646	517	473	473
query81	520	279	241	241
query82	494	127	94	94
query83	166	180	154	154
query84	243	95	71	71
query85	807	347	313	313
query86	382	317	283	283
query87	4401	4625	4310	4310
query88	3982	2182	2241	2182
query89	391	314	284	284
query90	1891	206	194	194
query91	136	140	109	109
query92	76	64	60	60
query93	2473	1004	573	573
query94	694	422	301	301
query95	359	264	261	261
query96	487	557	271	271
query97	2770	2858	2709	2709
query98	238	210	205	205
query99	1325	1372	1262	1262
Total cold run time: 271000 ms
Total hot run time: 183632 ms

doris-robot · 2025-02-17T09:55:25Z

ClickBench: Total hot run time: 30.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b162672a53ab9a74da1a4b2a7abac202e8792910, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.40	0.42	0.40
query6	1.18	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.52
query10	0.59	0.58	0.59
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.60	0.60
query14	2.68	2.75	2.70
query15	0.92	0.86	0.86
query16	0.35	0.38	0.38
query17	1.02	1.05	1.03
query18	0.22	0.18	0.19
query19	1.86	1.83	2.06
query20	0.01	0.01	0.02
query21	15.37	0.89	0.54
query22	0.75	1.13	0.92
query23	14.73	1.40	0.61
query24	12.03	1.03	0.31
query25	0.30	0.11	0.09
query26	0.84	0.18	0.13
query27	0.05	0.05	0.05
query28	6.16	0.78	0.44
query29	12.60	3.90	3.26
query30	0.24	0.08	0.06
query31	2.85	0.58	0.39
query32	3.23	0.55	0.46
query33	3.03	3.02	3.04
query34	15.69	5.16	4.50
query35	4.54	4.51	4.52
query36	0.65	0.49	0.50
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.3 s
Total hot run time: 30.32 s

englefly · 2025-02-18T02:04:17Z

run buildall

doris-robot · 2025-02-18T03:00:05Z

TPC-H: Total hot run time: 31523 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bba39321cbfec75c65d754e08221cff2215dda8, data reload: false
------ Round 1 ----------------------------------
q1	17627	5286	5072	5072
q2	2058	299	167	167
q3	10447	1260	760	760
q4	10203	1028	524	524
q5	7540	2311	2401	2311
q6	180	164	131	131
q7	901	735	599	599
q8	9317	1330	1137	1137
q9	4799	4717	4739	4717
q10	6812	2286	1886	1886
q11	470	273	247	247
q12	346	361	217	217
q13	17777	3668	3068	3068
q14	225	217	204	204
q15	509	472	461	461
q16	623	620	588	588
q17	574	853	338	338
q18	6869	6276	6092	6092
q19	1474	952	553	553
q20	322	319	189	189
q21	2807	2156	1964	1964
q22	366	323	298	298
Total cold run time: 102246 ms
Total hot run time: 31523 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5124	5114	5120	5114
q2	241	333	234	234
q3	2127	2639	2283	2283
q4	1449	1844	1376	1376
q5	4246	4113	4136	4113
q6	216	169	125	125
q7	1857	1804	1646	1646
q8	2615	2531	2447	2447
q9	7212	7234	7129	7129
q10	3045	3217	2764	2764
q11	570	535	516	516
q12	689	773	609	609
q13	3500	3883	3293	3293
q14	282	293	275	275
q15	512	459	453	453
q16	651	712	645	645
q17	1140	1598	1332	1332
q18	7566	7338	7251	7251
q19	793	752	820	752
q20	1992	2008	1901	1901
q21	5398	4844	4861	4844
q22	636	563	556	556
Total cold run time: 51861 ms
Total hot run time: 49658 ms

doris-robot · 2025-02-18T03:12:07Z

TPC-DS: Total hot run time: 190627 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bba39321cbfec75c65d754e08221cff2215dda8, data reload: false
query1	1309	969	966	966
query2	6236	1849	1796	1796
query3	10975	4666	4441	4441
query4	53340	24400	23478	23478
query5	4902	615	491	491
query6	333	190	175	175
query7	4873	506	295	295
query8	313	248	253	248
query9	5431	2604	2580	2580
query10	445	314	259	259
query11	15439	15070	14919	14919
query12	167	110	105	105
query13	1037	544	383	383
query14	10250	6823	6883	6823
query15	217	202	184	184
query16	7084	668	517	517
query17	1085	748	572	572
query18	1533	428	324	324
query19	206	209	167	167
query20	126	120	124	120
query21	208	130	157	130
query22	4616	4527	4444	4444
query23	34039	33447	33207	33207
query24	5720	2406	2413	2406
query25	449	458	390	390
query26	718	265	148	148
query27	1959	496	333	333
query28	2984	2484	2441	2441
query29	615	552	429	429
query30	206	190	159	159
query31	863	871	826	826
query32	73	65	63	63
query33	445	379	323	323
query34	805	882	506	506
query35	863	867	771	771
query36	946	959	891	891
query37	124	102	75	75
query38	4402	4263	4255	4255
query39	1492	1467	1455	1455
query40	206	120	111	111
query41	54	49	49	49
query42	124	111	103	103
query43	519	515	464	464
query44	1289	820	813	813
query45	180	169	177	169
query46	876	1047	663	663
query47	1876	1865	1816	1816
query48	393	417	301	301
query49	695	509	413	413
query50	723	750	410	410
query51	4241	4273	4267	4267
query52	114	114	93	93
query53	234	258	185	185
query54	482	497	423	423
query55	82	80	79	79
query56	272	283	279	279
query57	1181	1206	1125	1125
query58	238	242	239	239
query59	2677	2713	2642	2642
query60	276	280	256	256
query61	121	114	121	114
query62	727	754	688	688
query63	242	190	200	190
query64	1895	1060	680	680
query65	3296	3230	3282	3230
query66	778	391	319	319
query67	15961	15458	15316	15316
query68	5849	766	508	508
query69	526	312	262	262
query70	1181	1117	1095	1095
query71	441	294	256	256
query72	5432	3603	3720	3603
query73	1339	739	360	360
query74	9061	9202	8956	8956
query75	3163	3155	2681	2681
query76	3843	1170	746	746
query77	492	360	283	283
query78	10143	10072	9304	9304
query79	2840	787	594	594
query80	654	522	453	453
query81	513	277	240	240
query82	496	131	93	93
query83	171	165	149	149
query84	295	94	77	77
query85	791	363	312	312
query86	347	313	284	284
query87	4537	4465	4364	4364
query88	3498	2224	2211	2211
query89	442	307	278	278
query90	1651	194	186	186
query91	131	142	110	110
query92	75	62	55	55
query93	2597	1017	593	593
query94	701	398	314	314
query95	347	266	264	264
query96	475	556	282	282
query97	2843	2823	2700	2700
query98	225	199	201	199
query99	1345	1419	1283	1283
Total cold run time: 293024 ms
Total hot run time: 190627 ms

doris-robot · 2025-02-18T03:17:27Z

ClickBench: Total hot run time: 30.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bba39321cbfec75c65d754e08221cff2215dda8, data reload: false
query1	0.04	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.41	0.41	0.40
query6	1.17	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.52	0.53
query10	0.58	0.59	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.64	0.60	0.60
query14	2.70	2.75	2.69
query15	0.92	0.85	0.84
query16	0.38	0.37	0.38
query17	1.06	1.04	1.04
query18	0.22	0.20	0.20
query19	1.91	1.83	2.00
query20	0.01	0.01	0.02
query21	15.38	0.88	0.56
query22	0.76	1.07	0.69
query23	14.97	1.37	0.66
query24	10.69	0.98	0.43
query25	0.50	0.20	0.11
query26	0.79	0.18	0.15
query27	0.06	0.06	0.05
query28	7.43	0.81	0.43
query29	12.64	3.92	3.26
query30	0.25	0.08	0.06
query31	2.83	0.57	0.36
query32	3.22	0.55	0.47
query33	3.00	3.00	3.09
query34	15.74	5.05	4.57
query35	4.53	4.54	4.57
query36	0.66	0.51	0.49
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.86 s
Total hot run time: 30.37 s

github-actions · 2025-02-19T08:11:43Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-19T08:11:46Z

PR approved by anyone and no changes requested.

…e#47975) ### What problem does this PR solve? when there is no deferred materialized slots, do not use deferred materailizion. for example: SELECT t1.id FROM t1 WHERE id = 1 ORDER BY id LIMIT 1; there is no passive slots. slot "id" is the only slot need to be read by IO.

pick from #47661 #47975 #48747

…#52078) pick from #47661 #47975 #48747

pick from apache#47661 apache#47975 apache#48747

…#52522) pick from #47661 #47975 #48747

opt
b162672

englefly force-pushed the topn-two-phase branch from b8d7c3e to b162672Compare February 17, 2025 09:13

shape refresh
3bba393

morrySnow approved these changes Feb 19, 2025
View reviewed changes

morrySnow requested a review from starocean999 February 19, 2025 08:11

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 19, 2025

github-actionsBot added the reviewed label Feb 19, 2025

starocean999 approved these changes Feb 19, 2025
View reviewed changes

englefly merged commit b88c850 into apache:masterFeb 19, 2025

englefly deleted the topn-two-phase branch February 19, 2025 08:13

morrySnow added a commit that referenced this pull request Jun 20, 2025
branch-3.1: [opt](Nereids) support defer materialization with project
a214d49
pick from #47661 #47975 #48747

morrySnow mentioned this pull request Jun 20, 2025
branch-3.1: [opt](Nereids) support defer materialization with project #52078
Merged

morrySnow added a commit that referenced this pull request Jun 20, 2025
branch-3.1: [opt](Nereids) support defer materialization with project
97eda6a
pick from #47661 #47975 #48747

morrySnow added a commit that referenced this pull request Jun 20, 2025
branch-3.1: [opt](Nereids) support defer materialization with project
9371d86
pick from #47661 #47975 #48747

morrySnow added a commit that referenced this pull request Jun 20, 2025
branch-3.1: [opt](Nereids) support defer materialization with project (…
171c067
…#52078) pick from #47661 #47975 #48747

morrySnow added the dev/3.1.0-merged label Jun 20, 2025

morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jun 30, 2025
branch-3.0: [opt](Nereids) support defer materialization with project
fedaeea
pick from apache#47661 apache#47975 apache#48747

morrySnow mentioned this pull request Jun 30, 2025
branch-3.0: [opt](Nereids) support defer materialization with project #52522
Merged

dataroaring pushed a commit to morrySnow/incubator-doris that referenced this pull request Jul 9, 2025
branch-3.0: [opt](Nereids) support defer materialization with project
81f12f5
pick from apache#47661 apache#47975 apache#48747

dataroaring pushed a commit that referenced this pull request Jul 10, 2025
branch-3.0: [opt](Nereids) support defer materialization with project (…
1357b68
…#52522) pick from #47661 #47975 #48747

dataroaring added the dev/3.0.7-merged label Jul 10, 2025

Navigation Menu

Uh oh!

[opt](nereids) forbid topn-two-phase reading for some patterns - #47975

[opt](nereids) forbid topn-two-phase reading for some patterns#47975
englefly merged 2 commits into
apache:masterfrom
englefly:topn-two-phase

englefly commented Feb 17, 2025•
edited by morrySnow
Loading

Uh oh!

hello-stephen commented Feb 17, 2025

Uh oh!

englefly commented Feb 17, 2025

Uh oh!

englefly commented Feb 17, 2025

Uh oh!

doris-robot commented Feb 17, 2025

Uh oh!

doris-robot commented Feb 17, 2025

Uh oh!

doris-robot commented Feb 17, 2025

Uh oh!

englefly commented Feb 18, 2025

Uh oh!

doris-robot commented Feb 18, 2025

Uh oh!

doris-robot commented Feb 18, 2025

Uh oh!

doris-robot commented Feb 18, 2025

Uh oh!

github-actionsBot commented Feb 19, 2025

Uh oh!

github-actionsBot commented Feb 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

englefly commented Feb 17, 2025• edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Feb 17, 2025

Uh oh!

englefly commented Feb 17, 2025

Uh oh!

englefly commented Feb 17, 2025

Uh oh!

doris-robot commented Feb 17, 2025

Uh oh!

doris-robot commented Feb 17, 2025

Uh oh!

doris-robot commented Feb 17, 2025

Uh oh!

englefly commented Feb 18, 2025

Uh oh!

doris-robot commented Feb 18, 2025

Uh oh!

doris-robot commented Feb 18, 2025

Uh oh!

doris-robot commented Feb 18, 2025

Uh oh!

github-actionsBot commented Feb 19, 2025

Uh oh!

github-actionsBot commented Feb 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

englefly commented Feb 17, 2025•
edited by morrySnow
Loading