Skip to content

[opt](Nereids) support defer materialization with project - #47661

Merged
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:defer_support_project
Feb 14, 2025
Merged

[opt](Nereids) support defer materialization with project#47661
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:defer_support_project

Conversation

@morrySnow

@morrySnowmorrySnow commented Feb 8, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

  1. Because defer materialization in BE do not support project on scan node, so we use a little trick to do projection on last fragment's OutputExprs.

  2. fix core when table use partial column row store, introduced by [Feature](Row store) support column group with store row format for partial columns of table #34089

thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Feb 8, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

morrySnow commented Feb 8, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morrySnowmorrySnow changed the title [opt](Nereids) support defer materialized with project[opt](Nereids) support defer materialization with projectFeb 8, 2025
@doris-robot

doris-robot commented Feb 8, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31476 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3cfaed7992548826b6ec4329676cbfddb30809d, data reload: false
------ Round 1 ----------------------------------
q1	17580	5207	5027	5027
q2	2040	288	169	169
q3	10437	1241	743	743
q4	10287	974	530	530
q5	8453	2397	2258	2258
q6	194	174	137	137
q7	889	752	602	602
q8	9508	1326	1127	1127
q9	4941	4695	4864	4695
q10	6831	2312	1884	1884
q11	492	268	260	260
q12	345	354	220	220
q13	17788	3662	3080	3080
q14	224	220	211	211
q15	522	469	447	447
q16	625	628	578	578
q17	559	842	352	352
q18	6629	6167	6192	6167
q19	1205	965	540	540
q20	290	324	189	189
q21	2642	2193	1955	1955
q22	357	322	305	305
Total cold run time: 102838 ms
Total hot run time: 31476 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5051	5037	5100	5037
q2	232	324	219	219
q3	2148	2718	2285	2285
q4	1408	1797	1377	1377
q5	4227	4134	4124	4124
q6	207	168	128	128
q7	1828	1839	1645	1645
q8	2574	2596	2508	2508
q9	7291	7096	7232	7096
q10	3019	3172	2767	2767
q11	570	507	495	495
q12	669	762	592	592
q13	3572	3865	3257	3257
q14	283	317	276	276
q15	496	463	475	463
q16	641	678	648	648
q17	1150	1548	1378	1378
q18	7722	7393	7442	7393
q19	741	759	854	759
q20	1974	2002	1877	1877
q21	5407	5044	4776	4776
q22	661	598	566	566
Total cold run time: 51871 ms
Total hot run time: 49666 ms

@doris-robot

doris-robot commented Feb 8, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 189911 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3cfaed7992548826b6ec4329676cbfddb30809d, data reload: false
query1	1327	944	924	924
query2	6102	1851	1832	1832
query3	10951	4440	4504	4440
query4	53998	25778	23131	23131
query5	5121	553	482	482
query6	346	201	187	187
query7	4894	507	297	297
query8	312	249	238	238
query9	5651	2533	2487	2487
query10	417	308	273	273
query11	15271	15016	14969	14969
query12	163	115	113	113
query13	1049	532	406	406
query14	10626	6379	6846	6379
query15	195	195	200	195
query16	6813	665	509	509
query17	1074	717	565	565
query18	1081	383	298	298
query19	192	192	170	170
query20	133	125	124	124
query21	212	125	117	117
query22	4280	4422	4226	4226
query23	33883	33414	33238	33238
query24	5837	2492	2493	2492
query25	466	463	407	407
query26	657	272	162	162
query27	1629	492	343	343
query28	2809	2463	2409	2409
query29	541	543	449	449
query30	215	183	155	155
query31	904	884	813	813
query32	73	61	59	59
query33	455	359	290	290
query34	793	860	513	513
query35	782	839	760	760
query36	971	987	915	915
query37	119	106	85	85
query38	4272	4296	4367	4296
query39	1532	1456	1446	1446
query40	214	121	100	100
query41	51	51	49	49
query42	122	113	107	107
query43	487	526	474	474
query44	1329	821	812	812
query45	175	178	168	168
query46	890	1066	679	679
query47	1839	1885	1771	1771
query48	383	430	307	307
query49	713	525	426	426
query50	726	752	436	436
query51	4298	4291	4176	4176
query52	113	109	99	99
query53	251	270	190	190
query54	499	496	424	424
query55	84	85	79	79
query56	277	294	258	258
query57	1151	1206	1109	1109
query58	252	242	258	242
query59	2664	2930	2719	2719
query60	292	278	261	261
query61	120	121	120	120
query62	762	752	692	692
query63	235	196	206	196
query64	1434	1086	714	714
query65	3354	3158	3207	3158
query66	730	408	309	309
query67	15745	15624	15381	15381
query68	3668	829	542	542
query69	485	314	263	263
query70	1218	1115	1089	1089
query71	389	291	268	268
query72	5458	3608	3852	3608
query73	762	746	353	353
query74	8860	9047	9041	9041
query75	3189	3134	2669	2669
query76	2724	1165	757	757
query77	553	376	300	300
query78	10014	10138	9313	9313
query79	2984	804	608	608
query80	859	515	439	439
query81	530	273	240	240
query82	511	146	117	117
query83	188	170	153	153
query84	283	91	73	73
query85	811	346	300	300
query86	454	298	284	284
query87	4581	4623	4414	4414
query88	3734	2201	2200	2200
query89	476	309	278	278
query90	1768	193	190	190
query91	131	136	111	111
query92	75	61	53	53
query93	2699	1022	588	588
query94	704	401	278	278
query95	341	284	254	254
query96	479	571	268	268
query97	2810	2861	2720	2720
query98	226	206	195	195
query99	1351	1410	1295	1295
Total cold run time: 288651 ms
Total hot run time: 189911 ms

@morrySnow

morrySnow commented Feb 11, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morrySnow

morrySnow commented Feb 11, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 11, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32074 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9c0511a81c36118149aa416110b0312272003fcb, data reload: false
------ Round 1 ----------------------------------
q1	17582	5327	5198	5198
q2	2047	305	169	169
q3	10405	1298	727	727
q4	10216	1034	536	536
q5	7513	2486	2380	2380
q6	193	167	137	137
q7	918	750	617	617
q8	9319	1302	1150	1150
q9	5062	4835	4730	4730
q10	6869	2328	1913	1913
q11	488	284	265	265
q12	353	362	222	222
q13	17764	3707	3146	3146
q14	227	243	212	212
q15	513	456	458	456
q16	632	612	568	568
q17	590	877	345	345
q18	6837	6154	6279	6154
q19	1075	945	581	581
q20	321	348	199	199
q21	3176	2271	2045	2045
q22	364	340	324	324
Total cold run time: 102464 ms
Total hot run time: 32074 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5232	5269	5240	5240
q2	250	332	235	235
q3	2176	2700	2344	2344
q4	1483	1860	1369	1369
q5	4256	4167	4177	4167
q6	208	162	123	123
q7	1909	1847	1815	1815
q8	2584	2662	2599	2599
q9	7123	7049	7074	7049
q10	3048	3187	2774	2774
q11	585	509	490	490
q12	692	754	662	662
q13	3560	3878	3186	3186
q14	296	298	281	281
q15	518	477	464	464
q16	646	690	618	618
q17	1194	1586	1338	1338
q18	7640	7206	7235	7206
q19	847	827	900	827
q20	1968	2013	1878	1878
q21	5460	5072	4826	4826
q22	658	627	549	549
Total cold run time: 52333 ms
Total hot run time: 50040 ms

@doris-robot

doris-robot commented Feb 11, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 183167 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9c0511a81c36118149aa416110b0312272003fcb, data reload: false
query1	975	370	378	370
query2	6524	1807	1836	1807
query3	6805	208	207	207
query4	26845	23947	22917	22917
query5	4379	670	479	479
query6	308	205	223	205
query7	4617	525	302	302
query8	302	255	251	251
query9	8625	2492	2497	2492
query10	467	319	265	265
query11	15664	15138	14856	14856
query12	159	108	103	103
query13	1653	508	402	402
query14	9605	6300	6252	6252
query15	216	185	173	173
query16	7317	618	462	462
query17	1187	700	539	539
query18	1940	388	313	313
query19	192	181	158	158
query20	125	123	118	118
query21	205	117	105	105
query22	4279	4063	4464	4063
query23	34421	33442	33089	33089
query24	7721	2387	2369	2369
query25	528	479	414	414
query26	1226	278	157	157
query27	2102	515	338	338
query28	3910	2418	2361	2361
query29	743	569	439	439
query30	225	199	159	159
query31	930	850	798	798
query32	75	64	63	63
query33	569	364	315	315
query34	794	857	507	507
query35	801	824	779	779
query36	1014	988	899	899
query37	123	118	81	81
query38	4091	4126	4143	4126
query39	1451	1584	1385	1385
query40	210	121	110	110
query41	58	56	54	54
query42	125	105	106	105
query43	505	501	472	472
query44	1313	798	796	796
query45	186	171	168	168
query46	870	1056	650	650
query47	1741	1821	1706	1706
query48	376	411	315	315
query49	841	494	419	419
query50	686	720	436	436
query51	4170	4098	4084	4084
query52	110	111	93	93
query53	232	250	185	185
query54	483	486	409	409
query55	77	77	78	77
query56	273	275	257	257
query57	1149	1142	1089	1089
query58	249	247	239	239
query59	2488	2744	2603	2603
query60	276	272	256	256
query61	126	178	118	118
query62	810	722	657	657
query63	220	191	192	191
query64	4281	976	647	647
query65	3201	3162	3107	3107
query66	1131	406	340	340
query67	15969	15463	15373	15373
query68	7810	755	512	512
query69	480	296	258	258
query70	1133	1126	1048	1048
query71	407	295	266	266
query72	6059	3607	3707	3607
query73	749	735	353	353
query74	9168	8994	8774	8774
query75	3168	3174	2712	2712
query76	3175	1165	752	752
query77	458	375	283	283
query78	9774	10062	9365	9365
query79	2504	810	595	595
query80	656	521	454	454
query81	519	274	237	237
query82	498	152	116	116
query83	171	180	155	155
query84	244	96	75	75
query85	784	354	310	310
query86	451	280	307	280
query87	4384	4492	4540	4492
query88	3996	2216	2223	2216
query89	395	316	294	294
query90	1879	196	188	188
query91	141	140	110	110
query92	70	60	53	53
query93	2199	1027	592	592
query94	704	397	293	293
query95	362	264	252	252
query96	486	559	268	268
query97	2838	2833	2733	2733
query98	235	202	196	196
query99	1352	1407	1254	1254
Total cold run time: 272518 ms
Total hot run time: 183167 ms

@morrySnow
morrySnow marked this pull request as draft February 11, 2025 12:35
@morrySnow
morrySnow marked this pull request as draft February 11, 2025 12:35
@morrySnow

morrySnow commented Feb 12, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 12, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31361 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7bf52dc596a4f604c91ece8db48b7ac7b000525e, data reload: false
------ Round 1 ----------------------------------
q1	17682	5358	5107	5107
q2	2058	294	169	169
q3	10489	1212	732	732
q4	10196	997	524	524
q5	7517	2302	2349	2302
q6	182	167	131	131
q7	887	748	595	595
q8	9306	1273	1047	1047
q9	4811	4726	4529	4529
q10	6819	2323	1893	1893
q11	480	275	255	255
q12	346	353	224	224
q13	17783	3679	3173	3173
q14	241	237	208	208
q15	528	464	470	464
q16	635	622	575	575
q17	551	872	333	333
q18	6791	6253	6106	6106
q19	1208	944	551	551
q20	305	328	195	195
q21	2728	2285	1939	1939
q22	363	328	309	309
Total cold run time: 101906 ms
Total hot run time: 31361 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5144	5171	5162	5162
q2	237	325	228	228
q3	2149	2710	2276	2276
q4	1444	1819	1394	1394
q5	4219	4143	4152	4143
q6	214	165	125	125
q7	1895	1848	1655	1655
q8	2609	2594	2584	2584
q9	7164	7155	7063	7063
q10	2999	3204	2772	2772
q11	580	524	510	510
q12	695	798	599	599
q13	3601	3779	3306	3306
q14	304	302	273	273
q15	503	462	456	456
q16	639	724	648	648
q17	1129	1607	1341	1341
q18	7455	7281	7348	7281
q19	859	788	842	788
q20	1956	2052	1870	1870
q21	5473	5138	4969	4969
q22	597	566	567	566
Total cold run time: 51865 ms
Total hot run time: 50009 ms

@doris-robot

doris-robot commented Feb 12, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 189728 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7bf52dc596a4f604c91ece8db48b7ac7b000525e, data reload: false
query1	1292	971	918	918
query2	6206	1869	1847	1847
query3	11103	4579	4603	4579
query4	54464	26030	23255	23255
query5	5246	530	489	489
query6	397	191	198	191
query7	5274	513	309	309
query8	336	244	235	235
query9	7215	2563	2555	2555
query10	427	312	251	251
query11	15364	15162	14761	14761
query12	159	105	107	105
query13	1258	516	394	394
query14	10862	6780	6188	6188
query15	203	209	176	176
query16	7006	667	460	460
query17	1049	721	561	561
query18	1521	412	318	318
query19	206	201	167	167
query20	135	120	133	120
query21	213	119	99	99
query22	4471	4364	4523	4364
query23	33927	33483	33381	33381
query24	5692	2432	2443	2432
query25	471	482	446	446
query26	637	279	163	163
query27	1703	501	334	334
query28	2762	2468	2429	2429
query29	612	589	446	446
query30	217	192	156	156
query31	898	882	831	831
query32	80	66	60	60
query33	463	405	294	294
query34	741	848	509	509
query35	790	831	748	748
query36	966	1012	904	904
query37	117	90	71	71
query38	4323	4434	4296	4296
query39	1515	1424	1421	1421
query40	207	113	103	103
query41	56	53	56	53
query42	117	106	101	101
query43	497	520	482	482
query44	1288	800	805	800
query45	181	167	168	167
query46	866	1064	684	684
query47	1825	1912	1789	1789
query48	412	413	313	313
query49	721	504	409	409
query50	704	744	429	429
query51	4279	4330	4291	4291
query52	113	102	98	98
query53	221	260	187	187
query54	484	499	432	432
query55	81	78	84	78
query56	268	260	283	260
query57	1131	1201	1120	1120
query58	243	242	256	242
query59	2711	3033	2731	2731
query60	296	266	278	266
query61	119	115	116	115
query62	754	778	682	682
query63	229	192	200	192
query64	1428	1031	672	672
query65	3256	3151	3118	3118
query66	816	393	297	297
query67	15847	15440	15548	15440
query68	5578	766	502	502
query69	531	296	260	260
query70	1211	1063	1145	1063
query71	444	280	257	257
query72	6020	3572	3739	3572
query73	1012	757	351	351
query74	9064	9218	8856	8856
query75	3284	3144	2700	2700
query76	3986	1163	731	731
query77	544	361	280	280
query78	10098	10100	9267	9267
query79	2356	817	594	594
query80	651	522	442	442
query81	517	271	240	240
query82	304	122	94	94
query83	174	163	152	152
query84	294	92	79	79
query85	732	350	306	306
query86	348	270	291	270
query87	4480	4464	4471	4464
query88	3572	2176	2161	2161
query89	397	310	282	282
query90	1812	190	193	190
query91	137	133	113	113
query92	77	58	53	53
query93	2616	1017	568	568
query94	673	415	308	308
query95	352	264	255	255
query96	483	556	266	266
query97	2754	2786	2764	2764
query98	220	224	194	194
query99	1308	1388	1280	1280
Total cold run time: 295500 ms
Total hot run time: 189728 ms

@morrySnow

morrySnow commented Feb 12, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 12, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31309 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc5d98ae36a00250782715ee25964253f213f57e, data reload: false
------ Round 1 ----------------------------------
q1	17578	5185	5071	5071
q2	2045	307	182	182
q3	10440	1209	773	773
q4	10195	995	508	508
q5	7526	2398	2249	2249
q6	194	168	134	134
q7	884	760	599	599
q8	9308	1274	987	987
q9	4875	4629	4637	4629
q10	6846	2299	1876	1876
q11	477	277	253	253
q12	346	362	208	208
q13	17779	3648	3075	3075
q14	240	236	217	217
q15	509	465	470	465
q16	649	633	596	596
q17	564	855	330	330
q18	6595	6255	6176	6176
q19	1532	965	555	555
q20	314	328	188	188
q21	2648	2134	1941	1941
q22	366	320	297	297
Total cold run time: 101910 ms
Total hot run time: 31309 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5124	5139	5138	5138
q2	236	321	233	233
q3	2155	2638	2297	2297
q4	1428	1802	1352	1352
q5	4197	4139	4133	4133
q6	212	164	131	131
q7	1874	1812	1651	1651
q8	2539	2584	2616	2584
q9	7239	7170	7107	7107
q10	2997	3192	2765	2765
q11	571	525	502	502
q12	668	796	626	626
q13	3595	3812	3305	3305
q14	301	291	267	267
q15	519	461	470	461
q16	644	685	621	621
q17	1132	1605	1338	1338
q18	7502	7366	7310	7310
q19	776	779	892	779
q20	2081	2020	1872	1872
q21	5392	5022	4796	4796
q22	614	581	567	567
Total cold run time: 51796 ms
Total hot run time: 49835 ms

@doris-robot

doris-robot commented Feb 12, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 190083 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc5d98ae36a00250782715ee25964253f213f57e, data reload: false
query1	1312	955	939	939
query2	6217	1831	1800	1800
query3	10925	4533	4475	4475
query4	54587	25773	23236	23236
query5	5093	581	488	488
query6	335	203	201	201
query7	4906	515	286	286
query8	316	240	235	235
query9	5635	2563	2535	2535
query10	408	306	267	267
query11	15105	15066	14765	14765
query12	163	113	104	104
query13	1048	520	403	403
query14	10919	7004	6878	6878
query15	208	231	175	175
query16	7132	631	463	463
query17	1085	698	559	559
query18	1530	416	332	332
query19	200	184	159	159
query20	123	118	117	117
query21	210	128	110	110
query22	4530	4570	4380	4380
query23	34102	33213	33250	33213
query24	5666	2424	2399	2399
query25	476	451	380	380
query26	675	284	152	152
query27	1761	500	329	329
query28	2729	2438	2405	2405
query29	580	545	419	419
query30	211	190	154	154
query31	907	831	802	802
query32	72	62	60	60
query33	447	352	303	303
query34	764	879	497	497
query35	809	835	779	779
query36	958	1008	912	912
query37	115	102	74	74
query38	4435	4257	4306	4257
query39	1515	1554	1461	1461
query40	209	117	108	108
query41	52	50	50	50
query42	124	103	105	103
query43	509	512	476	476
query44	1287	798	782	782
query45	176	176	163	163
query46	883	1057	657	657
query47	1895	1891	1791	1791
query48	393	420	306	306
query49	687	491	410	410
query50	712	762	424	424
query51	4213	4277	4233	4233
query52	100	103	93	93
query53	243	251	182	182
query54	473	475	427	427
query55	79	78	83	78
query56	268	267	269	267
query57	1195	1226	1128	1128
query58	251	235	247	235
query59	2813	2796	2700	2700
query60	284	291	267	267
query61	116	117	115	115
query62	751	738	676	676
query63	225	189	185	185
query64	1518	1030	674	674
query65	3252	3118	3153	3118
query66	725	456	301	301
query67	15812	15705	15346	15346
query68	3394	786	509	509
query69	477	310	272	272
query70	1179	1078	1037	1037
query71	386	283	266	266
query72	6244	3664	3750	3664
query73	626	729	354	354
query74	9016	9184	8814	8814
query75	3163	3140	2718	2718
query76	2310	1157	732	732
query77	547	362	281	281
query78	10005	10187	9429	9429
query79	986	875	592	592
query80	855	532	475	475
query81	535	273	239	239
query82	269	128	99	99
query83	209	168	155	155
query84	294	99	70	70
query85	749	351	295	295
query86	391	306	288	288
query87	4440	4684	4509	4509
query88	2930	2200	2164	2164
query89	394	310	309	309
query90	1534	191	193	191
query91	131	137	106	106
query92	68	57	59	57
query93	1583	978	637	637
query94	670	411	300	300
query95	354	266	260	260
query96	486	558	267	267
query97	2813	2855	2704	2704
query98	217	219	199	199
query99	1275	1397	1293	1293
Total cold run time: 285984 ms
Total hot run time: 190083 ms

@morrySnow
morrySnowforce-pushed the defer_support_project branch 2 times, most recently from f8d47fc to cd7fa56CompareFebruary 13, 2025 03:23
@morrySnow

morrySnow commented Feb 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Feb 13, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31683 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd7fa56a1fafda32bde94dae465391f81d5c9443, data reload: false
------ Round 1 ----------------------------------
q1	17593	5221	5077	5077
q2	2061	304	170	170
q3	10482	1356	729	729
q4	10271	1020	548	548
q5	8485	2387	2365	2365
q6	186	168	138	138
q7	939	785	612	612
q8	9310	1312	1079	1079
q9	5781	4914	4650	4650
q10	6831	2341	1871	1871
q11	475	275	270	270
q12	349	363	218	218
q13	17800	3710	3088	3088
q14	238	224	214	214
q15	508	473	476	473
q16	621	605	585	585
q17	577	888	343	343
q18	6710	6332	6145	6145
q19	1209	949	556	556
q20	306	343	196	196
q21	2952	2226	2055	2055
q22	378	321	301	301
Total cold run time: 104062 ms
Total hot run time: 31683 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5169	5138	5136	5136
q2	239	325	229	229
q3	2157	2669	2347	2347
q4	1535	1908	1441	1441
q5	4276	4151	4183	4151
q6	203	162	122	122
q7	1850	1946	1853	1853
q8	2660	2687	2585	2585
q9	7223	7081	7150	7081
q10	3028	3214	2752	2752
q11	566	504	485	485
q12	666	735	664	664
q13	3532	3925	3294	3294
q14	280	296	275	275
q15	517	467	460	460
q16	620	664	639	639
q17	1151	1538	1378	1378
q18	7583	7405	7265	7265
q19	804	837	947	837
q20	1978	2016	1917	1917
q21	5494	5038	4917	4917
q22	658	588	548	548
Total cold run time: 52189 ms
Total hot run time: 50376 ms

@doris-robot

doris-robot commented Feb 13, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 183529 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd7fa56a1fafda32bde94dae465391f81d5c9443, data reload: false
query1	969	381	395	381
query2	6541	1847	1816	1816
query3	6791	212	208	208
query4	26279	24110	22932	22932
query5	4784	681	502	502
query6	296	209	188	188
query7	4622	520	299	299
query8	294	239	238	238
query9	8627	2576	2567	2567
query10	486	319	267	267
query11	15757	15106	15033	15033
query12	158	108	109	108
query13	1668	536	393	393
query14	9716	6225	6248	6225
query15	222	203	178	178
query16	7572	647	493	493
query17	1151	717	575	575
query18	2049	400	295	295
query19	194	192	164	164
query20	118	114	115	114
query21	212	122	107	107
query22	4245	4100	4178	4100
query23	34021	32885	32906	32885
query24	7806	2411	2399	2399
query25	532	461	388	388
query26	1228	266	152	152
query27	2634	466	330	330
query28	4354	2399	2415	2399
query29	717	565	425	425
query30	231	186	162	162
query31	938	826	807	807
query32	77	63	62	62
query33	555	360	296	296
query34	784	830	508	508
query35	818	810	762	762
query36	976	968	894	894
query37	124	104	72	72
query38	4183	4234	4074	4074
query39	1463	1404	1424	1404
query40	208	115	107	107
query41	54	52	55	52
query42	123	109	104	104
query43	501	503	486	486
query44	1304	790	785	785
query45	185	168	162	162
query46	871	1024	639	639
query47	1757	1784	1723	1723
query48	388	412	293	293
query49	800	505	407	407
query50	726	747	435	435
query51	4191	4199	4098	4098
query52	114	110	98	98
query53	235	258	184	184
query54	485	480	406	406
query55	81	79	82	79
query56	305	277	239	239
query57	1139	1160	1059	1059
query58	239	253	233	233
query59	2727	2818	2607	2607
query60	282	301	260	260
query61	120	116	120	116
query62	822	712	671	671
query63	227	189	186	186
query64	4263	996	652	652
query65	3195	3123	3147	3123
query66	1071	418	308	308
query67	15801	15594	15447	15447
query68	6855	775	507	507
query69	464	300	276	276
query70	1196	1143	1049	1049
query71	429	308	280	280
query72	5735	3636	3821	3636
query73	748	763	407	407
query74	9232	9197	9050	9050
query75	3153	3204	2684	2684
query76	3222	1236	752	752
query77	465	369	281	281
query78	10068	10105	9356	9356
query79	2492	807	586	586
query80	623	543	447	447
query81	503	280	240	240
query82	477	131	103	103
query83	181	179	156	156
query84	251	91	80	80
query85	768	353	299	299
query86	377	326	285	285
query87	4431	4569	4502	4502
query88	3853	2193	2184	2184
query89	384	322	282	282
query90	1932	199	192	192
query91	130	141	110	110
query92	73	61	55	55
query93	2010	1012	579	579
query94	678	414	301	301
query95	356	272	256	256
query96	495	567	266	266
query97	2788	2831	2706	2706
query98	231	211	204	204
query99	1313	1416	1257	1257
Total cold run time: 272319 ms
Total hot run time: 183529 ms

@doris-robot

doris-robot commented Feb 13, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd7fa56a1fafda32bde94dae465391f81d5c9443, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.02
query3	0.24	0.06	0.07
query4	1.67	0.10	0.10
query5	0.41	0.40	0.40
query6	1.17	0.66	0.67
query7	0.02	0.01	0.02
query8	0.03	0.03	0.02
query9	0.59	0.52	0.52
query10	0.57	0.58	0.58
query11	0.15	0.10	0.10
query12	0.15	0.11	0.12
query13	0.63	0.60	0.61
query14	2.80	2.73	2.68
query15	0.91	0.86	0.84
query16	0.36	0.38	0.37
query17	1.02	1.06	1.00
query18	0.21	0.19	0.19
query19	1.92	1.77	1.91
query20	0.01	0.01	0.01
query21	15.37	0.89	0.56
query22	0.73	1.21	0.71
query23	14.88	1.42	0.59
query24	7.06	1.29	1.32
query25	0.52	0.28	0.08
query26	0.58	0.17	0.16
query27	0.04	0.05	0.06
query28	9.97	0.86	0.44
query29	12.55	3.88	3.27
query30	0.26	0.09	0.06
query31	2.82	0.57	0.39
query32	3.23	0.54	0.47
query33	3.02	3.02	3.02
query34	15.74	5.16	4.53
query35	4.60	4.58	4.59
query36	0.66	0.50	0.50
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.49 s
Total hot run time: 31.11 s

@morrySnowmorrySnow mentioned this pull request Feb 19, 2025
Merged
16 tasks
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@morrySnow@lzyy2024
[opt](Nereids) support defer materialization with project (apache#47661)
18989c5
### What problem does this PR solve?
Problem Summary:
1. Because defer materialization in BE do not support project on scan
node, so we use a little trick to do projection on last fragment's
OutputExprs.
2. fix core when table use partial column row store, introduced by apache#34089
thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start
@morrySnowmorrySnow mentioned this pull request Mar 6, 2025
Merged
16 tasks
morrySnow added a commit that referenced this pull request Mar 20, 2025
@morrySnow
[opt](Nereids) support more pattern in DeferMaterializeTopNResult (#4…
947463d
…8747)
### What problem does this PR solve?
Related PR: #47661, #22487
Problem Summary:
after this PR we support:
1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morrySnowkoarz
[opt](Nereids) support defer materialization with project (apache#47661)
6ba6d96
### What problem does this PR solve?
Problem Summary:
1. Because defer materialization in BE do not support project on scan
node, so we use a little trick to do projection on last fragment's
OutputExprs.
2. fix core when table use partial column row store, introduced by apache#34089
thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morrySnowkoarz
[opt](Nereids) support more pattern in DeferMaterializeTopNResult (ap…
bd2a11f
…ache#48747)
### What problem does this PR solve?
Related PR: apache#47661, apache#22487
Problem Summary:
after this PR we support:
1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project
a214d49
pick from #47661#47975#48747
@morrySnowmorrySnow mentioned this pull request Jun 20, 2025
Merged
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project
97eda6a
pick from #47661#47975#48747
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project
9371d86
pick from #47661#47975#48747
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project (…
171c067
…#52078)
pick from #47661#47975#48747
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jun 30, 2025
@morrySnow
branch-3.0: [opt](Nereids) support defer materialization with project
fedaeea
pick from apache#47661apache#47975apache#48747
@morrySnowmorrySnow mentioned this pull request Jun 30, 2025
Merged
dataroaring pushed a commit to morrySnow/incubator-doris that referenced this pull request Jul 9, 2025
@morrySnow@dataroaring
branch-3.0: [opt](Nereids) support defer materialization with project
81f12f5
pick from apache#47661apache#47975apache#48747
dataroaring pushed a commit that referenced this pull request Jul 10, 2025
@morrySnow
branch-3.0: [opt](Nereids) support defer materialization with project (…
1357b68
…#52522)
pick from #47661#47975#48747
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@engleflyengleflyenglefly approved these changes
@924060929924060929924060929 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.7-mergeddev/3.1.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morrySnow@Thearas@doris-robot@englefly@924060929@dataroaring