[fix](planner) retain groupingSlotIds as materialized for aggregate by nextdreamblue · Pull Request #33060 · apache/doris

nextdreamblue · 2024-03-30T00:19:03Z

Proposed changes

pr #33033 need retain groupingSlotIds, these slots must be materialized

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Signed-off-by: nextdreamblue <zxw520blue1@163.com>

doris-robot · 2024-03-30T00:19:08Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

nextdreamblue · 2024-03-30T00:19:11Z

run buildall

doris-robot · 2024-03-30T00:56:59Z

TPC-H: Total hot run time: 38079 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efd79f319c4852ee182c5e2dbed45ea7f067792f, data reload: false
------ Round 1 ----------------------------------
q1	17620	4373	4209	4209
q2	2082	198	192	192
q3	10430	1066	1122	1066
q4	10192	790	741	741
q5	7451	2798	2702	2702
q6	214	134	131	131
q7	1012	605	632	605
q8	9303	2018	2013	2013
q9	7590	6528	6498	6498
q10	8599	3489	3479	3479
q11	448	241	240	240
q12	457	219	220	219
q13	18761	2962	2997	2962
q14	261	246	245	245
q15	528	474	474	474
q16	540	391	382	382
q17	954	569	635	569
q18	7306	6576	6613	6576
q19	5132	1417	1443	1417
q20	687	318	313	313
q21	3545	2802	2743	2743
q22	365	311	303	303
Total cold run time: 113477 ms
Total hot run time: 38079 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4271	4200	4181	4181
q2	373	279	274	274
q3	2968	2715	2673	2673
q4	1896	1502	1480	1480
q5	5262	5193	5196	5193
q6	206	124	125	124
q7	2237	1808	1866	1808
q8	3164	3282	3264	3264
q9	8525	8557	8564	8557
q10	3979	3857	3926	3857
q11	572	482	510	482
q12	867	618	634	618
q13	18049	3255	3230	3230
q14	337	337	293	293
q15	511	476	486	476
q16	495	440	447	440
q17	1781	1477	1496	1477
q18	8202	7890	7963	7890
q19	1650	1638	1583	1583
q20	2056	1760	1795	1760
q21	10576	4940	4869	4869
q22	543	471	487	471
Total cold run time: 78520 ms
Total hot run time: 55000 ms

doris-robot · 2024-03-30T01:08:46Z

TPC-DS: Total hot run time: 182425 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit efd79f319c4852ee182c5e2dbed45ea7f067792f, data reload: false
query1	876	377	358	358
query2	6321	2091	2093	2091
query3	6658	213	213	213
query4	25186	21306	21505	21306
query5	4221	415	402	402
query6	280	188	180	180
query7	4612	294	290	290
query8	223	175	175	175
query9	8570	2276	2284	2276
query10	414	244	238	238
query11	16471	14354	14263	14263
query12	134	91	90	90
query13	1649	374	366	366
query14	10542	7220	7013	7013
query15	204	174	170	170
query16	6852	269	276	269
query17	1703	605	566	566
query18	1409	288	281	281
query19	207	160	157	157
query20	92	88	87	87
query21	200	137	136	136
query22	4952	4827	4857	4827
query23	33275	32645	32706	32645
query24	10765	2911	2930	2911
query25	598	390	375	375
query26	786	152	160	152
query27	2594	367	362	362
query28	6126	1895	1886	1886
query29	881	645	630	630
query30	302	155	154	154
query31	1009	779	768	768
query32	93	63	54	54
query33	597	244	245	244
query34	954	506	527	506
query35	868	726	727	726
query36	1057	933	890	890
query37	108	65	74	65
query38	3708	3614	3646	3614
query39	1639	1600	1611	1600
query40	194	139	146	139
query41	54	51	46	46
query42	110	97	104	97
query43	490	463	472	463
query44	1102	748	746	746
query45	303	284	274	274
query46	1099	740	746	740
query47	2037	1951	1984	1951
query48	407	313	315	313
query49	810	386	378	378
query50	817	413	410	410
query51	6962	6817	6882	6817
query52	106	94	96	94
query53	372	298	298	298
query54	323	247	246	246
query55	86	80	80	80
query56	269	238	238	238
query57	1289	1199	1216	1199
query58	239	278	234	234
query59	2934	3127	2667	2667
query60	266	246	242	242
query61	114	111	109	109
query62	607	431	448	431
query63	308	291	288	288
query64	4662	4088	4079	4079
query65	3093	3047	3040	3040
query66	821	345	334	334
query67	15446	15063	14761	14761
query68	9434	546	553	546
query69	598	322	324	322
query70	1391	1204	1157	1157
query71	456	276	286	276
query72	7127	2621	2466	2466
query73	1680	324	322	322
query74	7019	6309	6311	6309
query75	3770	2325	2272	2272
query76	5023	934	918	918
query77	620	260	249	249
query78	10974	10199	10136	10136
query79	12792	540	539	539
query80	2583	421	422	421
query81	541	220	217	217
query82	540	92	91	91
query83	213	166	169	166
query84	258	82	80	80
query85	1153	282	274	274
query86	363	288	252	252
query87	3779	3561	3546	3546
query88	5619	2405	2384	2384
query89	514	373	372	372
query90	2055	187	176	176
query91	136	96	99	96
query92	62	49	46	46
query93	7127	510	505	505
query94	1368	179	179	179
query95	398	312	312	312
query96	632	272	266	266
query97	2621	2453	2497	2453
query98	245	216	219	216
query99	1099	833	811	811
Total cold run time: 304646 ms
Total hot run time: 182425 ms

doris-robot · 2024-03-30T01:14:07Z

ClickBench: Total hot run time: 30.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit efd79f319c4852ee182c5e2dbed45ea7f067792f, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.09	0.10
query5	0.51	0.49	0.49
query6	1.14	0.66	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.54	0.55	0.55
query11	0.17	0.10	0.12
query12	0.14	0.12	0.11
query13	0.59	0.58	0.59
query14	0.76	0.76	0.78
query15	0.82	0.80	0.82
query16	0.35	0.38	0.38
query17	0.96	1.01	1.02
query18	0.20	0.25	0.24
query19	1.81	1.83	1.79
query20	0.02	0.01	0.01
query21	15.52	0.63	0.63
query22	2.64	2.19	2.29
query23	17.34	1.02	0.80
query24	1.11	0.25	0.20
query25	0.10	0.07	0.09
query26	0.23	0.17	0.16
query27	0.08	0.08	0.08
query28	14.04	0.94	0.93
query29	12.51	3.26	3.21
query30	0.28	0.09	0.08
query31	2.81	0.38	0.39
query32	3.27	0.46	0.45
query33	2.80	2.81	2.86
query34	17.18	4.45	4.42
query35	4.53	4.46	4.48
query36	0.60	0.47	0.47
query37	0.17	0.15	0.14
query38	0.14	0.14	0.14
query39	0.05	0.03	0.04
query40	0.17	0.15	0.14
query41	0.08	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.41 s
Total hot run time: 30.15 s

doris-robot · 2024-03-30T01:17:13Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit efd79f319c4852ee182c5e2dbed45ea7f067792f with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.4 seconds inserted 10000000 Rows, about 746K ops/s

BiteTheDDDDt · 2024-03-30T01:37:58Z

run buildall

nextdreamblue · 2024-03-30T02:02:27Z

run buildall

nextdreamblue · 2024-03-30T02:02:59Z

run buildall

nextdreamblue · 2024-03-30T02:11:19Z

run feut

doris-robot · 2024-03-30T02:20:17Z

TPC-H: Total hot run time: 38142 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed5bbeae8aea4d735ceba5f18f29f5bc9586c8d0, data reload: false
------ Round 1 ----------------------------------
q1	17606	4333	4199	4199
q2	2072	194	188	188
q3	10421	1252	1163	1163
q4	10200	846	748	748
q5	7444	2720	2744	2720
q6	214	130	134	130
q7	1035	613	587	587
q8	9220	2019	1996	1996
q9	7436	6531	6475	6475
q10	8517	3500	3515	3500
q11	480	245	240	240
q12	449	222	217	217
q13	17787	3006	2976	2976
q14	267	231	232	231
q15	516	461	467	461
q16	533	384	391	384
q17	948	537	555	537
q18	7322	6649	6715	6649
q19	7648	1426	1416	1416
q20	659	308	303	303
q21	3578	2715	2765	2715
q22	388	307	317	307
Total cold run time: 114740 ms
Total hot run time: 38142 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4347	4219	4164	4164
q2	365	282	261	261
q3	2962	2686	2689	2686
q4	1875	1486	1538	1486
q5	5249	5264	5273	5264
q6	212	124	127	124
q7	2210	1853	1856	1853
q8	3201	3310	3307	3307
q9	8564	8529	8599	8529
q10	4026	3886	3931	3886
q11	620	521	512	512
q12	816	612	633	612
q13	17785	3235	3172	3172
q14	311	305	288	288
q15	517	468	471	468
q16	511	465	475	465
q17	1847	1457	1484	1457
q18	8128	7984	7892	7892
q19	1625	1511	1507	1507
q20	1996	1862	1787	1787
q21	8561	4978	4894	4894
q22	521	461	459	459
Total cold run time: 76249 ms
Total hot run time: 55073 ms

doris-robot · 2024-03-30T02:32:06Z

TPC-DS: Total hot run time: 182067 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed5bbeae8aea4d735ceba5f18f29f5bc9586c8d0, data reload: false
query1	881	376	1118	376
query2	6205	1976	2044	1976
query3	6657	204	202	202
query4	24679	21306	21381	21306
query5	4249	412	410	410
query6	285	178	174	174
query7	4597	299	288	288
query8	237	166	161	161
query9	8551	2291	2287	2287
query10	417	234	272	234
query11	15120	14213	14313	14213
query12	141	92	86	86
query13	1667	373	367	367
query14	8686	6868	7000	6868
query15	202	173	172	172
query16	7217	266	259	259
query17	1427	595	569	569
query18	1498	297	278	278
query19	208	163	165	163
query20	94	97	89	89
query21	199	133	132	132
query22	5064	4833	4854	4833
query23	33290	32761	32725	32725
query24	11508	2958	2953	2953
query25	655	411	420	411
query26	1497	169	163	163
query27	3272	367	368	367
query28	7621	1898	1881	1881
query29	902	628	617	617
query30	1096	170	165	165
query31	1020	740	754	740
query32	83	64	56	56
query33	686	264	254	254
query34	1254	514	513	513
query35	871	734	702	702
query36	1093	965	933	933
query37	129	72	72	72
query38	3851	3661	3606	3606
query39	1668	1561	1596	1561
query40	187	152	133	133
query41	46	44	45	44
query42	107	105	101	101
query43	489	457	451	451
query44	1162	717	728	717
query45	297	270	252	252
query46	1098	739	726	726
query47	2003	1940	1933	1933
query48	387	302	308	302
query49	890	389	394	389
query50	860	416	418	416
query51	6896	6802	6781	6781
query52	111	104	107	104
query53	365	303	289	289
query54	303	248	240	240
query55	84	79	77	77
query56	250	245	245	245
query57	1207	1106	1114	1106
query58	240	222	223	222
query59	2893	2685	2644	2644
query60	277	251	243	243
query61	113	110	108	108
query62	602	437	435	435
query63	308	281	281	281
query64	5411	4250	4244	4244
query65	3078	3030	3037	3030
query66	781	362	354	354
query67	15480	15370	14832	14832
query68	9373	531	532	531
query69	583	321	315	315
query70	1257	1135	1171	1135
query71	531	277	275	275
query72	7009	2616	2436	2436
query73	830	317	317	317
query74	7138	6361	6483	6361
query75	3782	2325	2322	2322
query76	5218	1039	977	977
query77	607	247	254	247
query78	10970	10197	10110	10110
query79	12838	520	512	512
query80	1859	424	411	411
query81	502	216	1077	216
query82	802	96	85	85
query83	206	169	159	159
query84	261	78	81	78
query85	1362	265	258	258
query86	396	284	303	284
query87	3680	3525	3486	3486
query88	5831	2316	2292	2292
query89	500	362	369	362
query90	1997	171	172	171
query91	121	93	93	93
query92	58	46	46	46
query93	6897	491	490	490
query94	1210	176	173	173
query95	404	314	311	311
query96	603	269	266	266
query97	2660	2463	2461	2461
query98	246	221	207	207
query99	1126	846	819	819
Total cold run time: 305559 ms
Total hot run time: 182067 ms

doris-robot · 2024-03-30T02:37:25Z

ClickBench: Total hot run time: 29.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed5bbeae8aea4d735ceba5f18f29f5bc9586c8d0, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.09	0.09
query5	0.50	0.49	0.48
query6	1.13	0.65	0.65
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.51
query10	0.56	0.56	0.55
query11	0.16	0.11	0.11
query12	0.14	0.12	0.10
query13	0.59	0.59	0.58
query14	0.77	0.78	0.77
query15	0.82	0.80	0.81
query16	0.37	0.37	0.37
query17	0.93	0.94	0.95
query18	0.23	0.23	0.25
query19	1.79	1.74	1.67
query20	0.01	0.01	0.02
query21	15.59	0.65	0.64
query22	2.38	2.91	1.72
query23	17.20	0.90	0.82
query24	1.17	0.23	0.25
query25	0.09	0.08	0.08
query26	0.21	0.16	0.15
query27	0.07	0.07	0.08
query28	14.02	0.92	0.93
query29	12.53	3.27	3.23
query30	0.28	0.08	0.08
query31	2.83	0.39	0.38
query32	3.29	0.45	0.45
query33	2.78	2.87	2.87
query34	16.78	4.41	4.47
query35	4.52	4.47	4.52
query36	0.58	0.46	0.46
query37	0.18	0.14	0.14
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.18	0.14	0.14
query41	0.10	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.05
Total cold run time: 105.69 s
Total hot run time: 29.58 s

doris-robot · 2024-03-30T02:40:32Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ed5bbeae8aea4d735ceba5f18f29f5bc9586c8d0 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.3 seconds inserted 10000000 Rows, about 699K ops/s

nextdreamblue · 2024-03-30T03:23:45Z

run pipelinex_p0

github-actions · 2024-03-30T04:07:30Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-30T04:07:32Z

PR approved by anyone and no changes requested.

lide-reed

LGTM

…33060)

…pache#33060)

[fix](planner) retain groupingSlotIds as materialized for aggregate
efd79f3
Signed-off-by: nextdreamblue <zxw520blue1@163.com>

wm1581066 requested review from 924060929 and starocean999 March 30, 2024 00:25

Merge branch 'master' into fix_slot_again_again
0bd63a9

Merge branch 'master' into fix_slot_again_again
ed5bbea

starocean999 approved these changes Mar 30, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2024

github-actionsBot added the reviewed label Mar 30, 2024

lide-reed approved these changes Mar 30, 2024
View reviewed changes

lide-reed merged commit 2a7876b into apache:masterMar 30, 2024

nextdreamblue mentioned this pull request Apr 1, 2024
[fix](planner) retain groupingSlotIds as materialized for aggregate #33064
Merged

yiguolei pushed a commit that referenced this pull request Apr 1, 2024
[fix](planner) retain groupingSlotIds as materialized for aggregate (#…
cddf133
…33060)

xiaokang added the dev/2.0.8-merged label Apr 2, 2024

yiguolei pushed a commit that referenced this pull request Apr 10, 2024
[fix](planner) retain groupingSlotIds as materialized for aggregate (#…
fb910e5
…33060)

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](planner) retain groupingSlotIds as materialized for aggregate (a…
01acd0f
…pache#33060)

Uh oh!

Conversation

nextdreamblue commented Mar 30, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

nextdreamblue commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

BiteTheDDDDt commented Mar 30, 2024

Uh oh!

nextdreamblue commented Mar 30, 2024

Uh oh!

nextdreamblue commented Mar 30, 2024

Uh oh!

nextdreamblue commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

doris-robot commented Mar 30, 2024

Uh oh!

nextdreamblue commented Mar 30, 2024

Uh oh!

github-actionsBot commented Mar 30, 2024

Uh oh!

github-actionsBot commented Mar 30, 2024

Uh oh!

lide-reed left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants