[enhancement](nereids)only push having as agg's parent if having just use slots from agg's output by starocean999 · Pull Request #32414 · apache/doris

starocean999 · 2024-03-18T11:21:15Z

Proposed changes

only push having as agg's parent if having just use slots from agg's output
show user friendly error message when item in select list but not in aggregate node's output

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-18T11:21:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

starocean999 · 2024-03-18T11:21:23Z

run buildall

starocean999 · 2024-03-19T02:11:34Z

run buildall

starocean999 · 2024-03-20T01:44:14Z

run buildall

starocean999 · 2024-03-20T06:55:39Z

run buildall

starocean999 · 2024-03-21T10:07:40Z

run buildall

doris-robot · 2024-03-21T10:36:32Z

TPC-H: Total hot run time: 38372 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eeb5805d9e13559a732c6bf1d6e3dcbe5f8fed96, data reload: false
------ Round 1 ----------------------------------
q1	17625	4457	4264	4264
q2	2023	153	140	140
q3	10705	1169	1254	1169
q4	10651	765	809	765
q5	7768	3034	2998	2998
q6	205	126	124	124
q7	1043	590	582	582
q8	9489	2097	2082	2082
q9	7325	6645	6521	6521
q10	8439	3398	3511	3398
q11	454	235	228	228
q12	383	213	198	198
q13	17910	2856	2888	2856
q14	228	214	215	214
q15	523	469	465	465
q16	461	364	364	364
q17	991	542	652	542
q18	7580	6576	6620	6576
q19	4853	1397	1452	1397
q20	526	251	258	251
q21	3666	2939	2997	2939
q22	347	327	299	299
Total cold run time: 113195 ms
Total hot run time: 38372 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4248	4154	4182	4154
q2	316	225	225	225
q3	2996	2842	2861	2842
q4	1893	1566	1597	1566
q5	5243	5262	5297	5262
q6	199	118	122	118
q7	2265	1822	1823	1822
q8	3196	3356	3335	3335
q9	8655	8606	8617	8606
q10	3749	3904	3753	3753
q11	547	444	444	444
q12	728	552	563	552
q13	16917	2861	2847	2847
q14	278	251	256	251
q15	504	458	453	453
q16	458	411	430	411
q17	1826	1501	1499	1499
q18	7853	7299	7037	7037
q19	1634	1550	1497	1497
q20	1946	1744	1742	1742
q21	4953	4903	4644	4644
q22	541	463	461	461
Total cold run time: 70945 ms
Total hot run time: 53521 ms

doris-robot · 2024-03-21T10:48:28Z

TPC-DS: Total hot run time: 180924 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eeb5805d9e13559a732c6bf1d6e3dcbe5f8fed96, data reload: false
query1	921	372	355	355
query2	7330	2073	2005	2005
query3	6708	207	207	207
query4	31088	20788	20690	20690
query5	4331	396	409	396
query6	264	175	178	175
query7	4635	285	296	285
query8	234	177	171	171
query9	9958	2304	2276	2276
query10	463	247	254	247
query11	17178	14092	14094	14092
query12	144	94	91	91
query13	1651	414	414	414
query14	12115	10895	10420	10420
query15	255	203	202	202
query16	8171	272	277	272
query17	1892	582	540	540
query18	2108	286	283	283
query19	299	151	161	151
query20	94	86	86	86
query21	195	129	134	129
query22	4627	4470	4475	4470
query23	31839	31321	31112	31112
query24	10721	2796	2806	2796
query25	616	386	393	386
query26	1159	155	158	155
query27	2729	346	346	346
query28	7880	1876	1896	1876
query29	863	659	629	629
query30	302	146	142	142
query31	954	739	712	712
query32	97	58	55	55
query33	773	271	257	257
query34	1053	489	466	466
query35	832	609	614	609
query36	979	875	910	875
query37	122	78	77	77
query38	3562	3438	3422	3422
query39	1422	1362	1425	1362
query40	218	113	114	113
query41	54	47	46	46
query42	105	95	100	95
query43	481	441	449	441
query44	1185	714	707	707
query45	265	261	260	260
query46	1087	667	669	667
query47	1658	1602	1577	1577
query48	445	346	355	346
query49	1140	342	337	337
query50	767	373	380	373
query51	6684	6466	6605	6466
query52	114	92	93	92
query53	346	277	273	273
query54	310	248	255	248
query55	88	83	82	82
query56	266	247	240	240
query57	1081	1025	1020	1020
query58	251	219	213	213
query59	2740	2575	2657	2575
query60	271	257	264	257
query61	116	119	120	119
query62	612	403	401	401
query63	306	278	279	278
query64	5727	3902	3925	3902
query65	3162	3066	2998	2998
query66	1421	343	335	335
query67	14963	14335	14333	14333
query68	9246	517	530	517
query69	650	375	390	375
query70	1394	1148	1169	1148
query71	533	288	275	275
query72	6676	2678	2489	2489
query73	1635	316	317	316
query74	7224	6600	6595	6595
query75	4532	2849	2809	2809
query76	5657	909	986	909
query77	664	269	269	269
query78	10430	9622	9544	9544
query79	11428	520	519	519
query80	1791	391	381	381
query81	517	206	213	206
query82	458	196	194	194
query83	209	144	140	140
query84	284	82	76	76
query85	1185	333	316	316
query86	348	286	314	286
query87	3748	3494	3555	3494
query88	4777	2306	2277	2277
query89	482	359	359	359
query90	2098	177	176	176
query91	160	132	134	132
query92	60	48	47	47
query93	6963	503	492	492
query94	1319	176	174	174
query95	431	340	320	320
query96	617	264	272	264
query97	3039	2854	2871	2854
query98	235	215	197	197
query99	1036	746	760	746
Total cold run time: 316253 ms
Total hot run time: 180924 ms

doris-robot · 2024-03-21T10:51:41Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit eeb5805d9e13559a732c6bf1d6e3dcbe5f8fed96 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.2 seconds inserted 10000000 Rows, about 495K ops/s

…elect list but not in aggregate node's output

starocean999 · 2024-03-22T10:32:53Z

run buildall

doris-robot · 2024-03-22T10:50:55Z

TPC-H: Total hot run time: 38251 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 216114ad3998622f5002b23f4d0fa7a749bf99ee, data reload: false
------ Round 1 ----------------------------------
q1	17660	4467	4254	4254
q2	3213	164	154	154
q3	11093	1157	1209	1157
q4	10858	792	725	725
q5	7892	3097	3052	3052
q6	211	126	122	122
q7	1049	591	586	586
q8	9463	1995	1999	1995
q9	7239	6635	6571	6571
q10	9651	3458	3487	3458
q11	443	230	217	217
q12	389	202	198	198
q13	17830	2895	2883	2883
q14	244	201	202	201
q15	520	461	469	461
q16	496	377	361	361
q17	965	568	576	568
q18	7236	6533	6462	6462
q19	1570	1512	1430	1430
q20	565	250	246	246
q21	3526	2864	2967	2864
q22	344	286	299	286
Total cold run time: 112457 ms
Total hot run time: 38251 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4132	4116	4085	4085
q2	328	234	233	233
q3	2982	2841	2828	2828
q4	1870	1558	1534	1534
q5	5239	5259	5269	5259
q6	195	116	116	116
q7	2235	1888	1824	1824
q8	3178	3279	3341	3279
q9	8602	8587	8581	8581
q10	3694	3675	3714	3675
q11	547	449	441	441
q12	727	554	550	550
q13	16911	2885	2851	2851
q14	274	267	267	267
q15	502	463	469	463
q16	473	425	433	425
q17	1754	1495	1455	1455
q18	7514	7260	7220	7220
q19	1910	1556	1468	1468
q20	1924	1735	1697	1697
q21	4724	4726	4606	4606
q22	531	450	434	434
Total cold run time: 70246 ms
Total hot run time: 53291 ms

doris-robot · 2024-03-22T11:02:40Z

TPC-DS: Total hot run time: 181722 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 216114ad3998622f5002b23f4d0fa7a749bf99ee, data reload: false
query1	944	367	360	360
query2	7386	1991	1946	1946
query3	6702	215	211	211
query4	31278	20567	20813	20567
query5	4362	416	405	405
query6	272	179	177	177
query7	4638	302	297	297
query8	223	166	174	166
query9	9300	2346	2339	2339
query10	557	247	254	247
query11	15416	14360	14250	14250
query12	135	95	83	83
query13	1616	414	408	408
query14	11834	11540	10972	10972
query15	263	210	194	194
query16	8184	263	265	263
query17	1976	560	550	550
query18	2098	298	282	282
query19	351	161	157	157
query20	97	86	89	86
query21	204	125	125	125
query22	4609	4428	4476	4428
query23	31905	31152	30923	30923
query24	10637	2869	2850	2850
query25	606	380	377	377
query26	1183	160	156	156
query27	2917	349	357	349
query28	7890	1897	1917	1897
query29	889	625	619	619
query30	306	147	145	145
query31	957	728	722	722
query32	91	57	60	57
query33	777	263	263	263
query34	1044	464	471	464
query35	819	608	615	608
query36	971	867	884	867
query37	128	80	77	77
query38	3575	3448	3455	3448
query39	1435	1384	1380	1380
query40	214	121	117	117
query41	53	50	48	48
query42	105	94	101	94
query43	482	474	472	472
query44	1256	751	716	716
query45	273	225	248	225
query46	1098	691	679	679
query47	1678	1579	1588	1579
query48	457	371	361	361
query49	1109	322	335	322
query50	759	372	365	365
query51	6651	6597	6507	6507
query52	108	95	93	93
query53	342	272	276	272
query54	300	251	260	251
query55	85	80	76	76
query56	267	237	229	229
query57	1074	997	1009	997
query58	236	208	211	208
query59	2758	2658	2541	2541
query60	280	241	248	241
query61	99	98	95	95
query62	585	419	402	402
query63	310	277	279	277
query64	5711	3886	3942	3886
query65	3102	3017	2979	2979
query66	1458	347	340	340
query67	14942	14347	14312	14312
query68	5845	525	529	525
query69	596	374	378	374
query70	1221	1208	1137	1137
query71	456	288	284	284
query72	6437	2709	2655	2655
query73	720	322	322	322
query74	6961	6551	6549	6549
query75	3823	2848	2856	2848
query76	4126	902	897	897
query77	672	295	260	260
query78	10444	9698	9592	9592
query79	8287	523	514	514
query80	1305	392	404	392
query81	557	218	218	218
query82	1411	213	204	204
query83	212	150	152	150
query84	299	82	83	82
query85	1567	382	360	360
query86	462	280	298	280
query87	3761	3536	3574	3536
query88	5152	2353	2355	2353
query89	535	360	371	360
query90	2019	171	174	171
query91	165	143	139	139
query92	59	47	49	47
query93	6356	520	493	493
query94	1180	182	180	180
query95	430	335	333	333
query96	597	272	265	265
query97	3060	2878	2854	2854
query98	219	218	207	207
query99	1220	742	756	742
Total cold run time: 304521 ms
Total hot run time: 181722 ms

doris-robot · 2024-03-22T11:05:53Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 216114ad3998622f5002b23f4d0fa7a749bf99ee with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s

github-actions · 2024-03-22T13:22:54Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-22T13:22:56Z

PR approved by anyone and no changes requested.

starocean999 · 2024-03-22T14:27:45Z

run buildall

doris-robot · 2024-03-22T15:29:02Z

TPC-H: Total hot run time: 38225 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2935f643f5962578164555a3d1b8caafacaa1e6a, data reload: false
------ Round 1 ----------------------------------
q1	17672	4304	4164	4164
q2	2101	159	162	159
q3	10570	1158	1179	1158
q4	10232	799	761	761
q5	7471	2982	2984	2982
q6	207	123	123	123
q7	1037	599	562	562
q8	9342	2039	1949	1949
q9	7343	6631	6575	6575
q10	8466	3492	3592	3492
q11	424	220	220	220
q12	425	201	204	201
q13	17794	2868	2869	2868
q14	252	202	203	202
q15	500	455	461	455
q16	491	369	377	369
q17	958	623	517	517
q18	7241	6632	6582	6582
q19	4659	1422	1392	1392
q20	546	259	244	244
q21	3684	2991	2956	2956
q22	361	294	305	294
Total cold run time: 111776 ms
Total hot run time: 38225 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4174	4092	4093	4092
q2	322	232	234	232
q3	2954	2823	2828	2823
q4	1883	1603	1535	1535
q5	5279	5345	5357	5345
q6	191	118	115	115
q7	2236	1874	1880	1874
q8	3148	3281	3299	3281
q9	8682	8682	8711	8682
q10	3836	3809	3775	3775
q11	554	449	469	449
q12	723	553	579	553
q13	16925	2860	2868	2860
q14	280	249	259	249
q15	488	451	455	451
q16	477	432	420	420
q17	1751	1510	1483	1483
q18	7402	7247	7045	7045
q19	1642	1518	1527	1518
q20	1901	1729	1744	1729
q21	4836	4565	4723	4565
q22	511	456	452	452
Total cold run time: 70195 ms
Total hot run time: 53528 ms

doris-robot · 2024-03-22T15:41:06Z

TPC-DS: Total hot run time: 186219 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2935f643f5962578164555a3d1b8caafacaa1e6a, data reload: false
query1	952	373	365	365
query2	7437	2009	1883	1883
query3	6702	211	209	209
query4	31977	21184	21520	21184
query5	4530	412	402	402
query6	284	177	177	177
query7	4617	290	278	278
query8	232	184	175	175
query9	9115	2277	2254	2254
query10	582	238	266	238
query11	14875	14512	14484	14484
query12	132	90	89	89
query13	1622	420	416	416
query14	12344	10935	11580	10935
query15	295	218	192	192
query16	8186	259	248	248
query17	1991	550	520	520
query18	2100	276	293	276
query19	347	158	154	154
query20	94	92	92	92
query21	213	127	130	127
query22	5015	4784	4775	4775
query23	33664	33104	32856	32856
query24	10648	2903	2905	2903
query25	601	384	382	382
query26	1170	159	154	154
query27	2396	348	361	348
query28	7251	1900	1870	1870
query29	905	654	618	618
query30	302	149	147	147
query31	945	723	734	723
query32	87	58	55	55
query33	762	364	261	261
query34	926	478	488	478
query35	833	607	624	607
query36	995	856	859	856
query37	126	76	76	76
query38	3560	3439	3464	3439
query39	1461	1474	1443	1443
query40	224	118	116	116
query41	48	47	48	47
query42	104	96	93	93
query43	473	452	470	452
query44	1204	752	719	719
query45	287	258	238	238
query46	1120	718	719	718
query47	1933	1877	1853	1853
query48	438	351	356	351
query49	1108	341	337	337
query50	758	375	380	375
query51	6846	6774	6641	6641
query52	111	88	90	88
query53	343	278	272	272
query54	312	256	257	256
query55	92	78	79	78
query56	243	237	233	233
query57	1208	1137	1150	1137
query58	238	214	217	214
query59	2826	2519	2519	2519
query60	278	242	254	242
query61	130	114	123	114
query62	641	460	451	451
query63	311	275	270	270
query64	5380	4073	3982	3982
query65	3058	3016	3002	3002
query66	884	393	376	376
query67	15372	14815	14687	14687
query68	8692	551	528	528
query69	632	371	380	371
query70	1223	1172	1133	1133
query71	517	282	280	280
query72	6316	2721	2558	2558
query73	748	310	311	310
query74	7143	6613	6653	6613
query75	4547	2813	2773	2773
query76	5439	909	888	888
query77	677	262	255	255
query78	10853	10086	10199	10086
query79	10140	517	529	517
query80	1889	381	386	381
query81	533	214	217	214
query82	329	201	201	201
query83	217	143	143	143
query84	287	79	84	79
query85	1132	315	317	315
query86	373	280	299	280
query87	3759	3527	3574	3527
query88	4856	2281	2269	2269
query89	476	372	372	372
query90	2105	175	172	172
query91	173	132	140	132
query92	60	48	46	46
query93	6679	488	478	478
query94	1325	174	170	170
query95	436	335	321	321
query96	614	270	263	263
query97	3072	2894	2898	2894
query98	233	211	200	200
query99	1158	946	900	900
Total cold run time: 312877 ms
Total hot run time: 186219 ms

doris-robot · 2024-03-22T15:44:16Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2935f643f5962578164555a3d1b8caafacaa1e6a with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.0 seconds inserted 10000000 Rows, about 500K ops/s

github-actions · 2024-03-26T08:55:26Z

PR approved by at least one committer and no changes requested.

…t use slots from agg's output (#32847) pick from master #32414

…t use slots from agg's output (#32414) 1. only push having as agg's parent if having just use slots from agg's output 2. show user friendly error message when item in select list but not in aggregate node's output

…t use slots from agg's output (apache#32847) pick from master apache#32414

…t use slots from agg's output (apache#32414) 1. only push having as agg's parent if having just use slots from agg's output 2. show user friendly error message when item in select list but not in aggregate node's output

starocean999 force-pushed the nereids_1026 branch from 1731c94 to 0036799Compare March 19, 2024 02:11

starocean999 force-pushed the nereids_1026 branch from 0036799 to 9b7db6cCompare March 20, 2024 01:43

starocean999 force-pushed the nereids_1026 branch 2 times, most recently from 1d4d0b1 to c54d4a3Compare March 20, 2024 06:55

starocean999 marked this pull request as ready for review March 21, 2024 08:20

morrySnow reviewed Mar 21, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java Outdated

starocean999 force-pushed the nereids_1026 branch from c54d4a3 to eeb5805Compare March 21, 2024 10:07

starocean999 added 3 commits March 22, 2024 18:03

[enhancement](nereids)show user friendly error message when item in s…
873e55d
…elect list but not in aggregate node's output

update code
2272bb0

fix bug
216114a

starocean999 force-pushed the nereids_1026 branch from eeb5805 to 216114aCompare March 22, 2024 10:30

starocean999 changed the title ~~[enhancement](nereids)show user friendly error message when item in select list but not in aggregate node's output~~[enhancement](nereids)only push having as agg's parent if having just use slots from agg's outputMar 22, 2024

morrySnow previously approved these changes Mar 22, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024

github-actionsBot added the reviewed label Mar 22, 2024

fix case
2935f64

starocean999 dismissed morrySnow’s stale review via 2935f64March 22, 2024 14:27

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Mar 22, 2024

morrySnow approved these changes Mar 26, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2024

924060929 approved these changes Mar 26, 2024
View reviewed changes

morrySnow merged commit 0dd27dc into apache:masterMar 26, 2024

morrySnow added the dev/2.0.x label Mar 26, 2024

starocean999 mentioned this pull request Mar 26, 2024
[enhancement](nereids)only push having as agg's parent if having just use slots from agg's output #32847
Merged

morrySnow pushed a commit that referenced this pull request Mar 26, 2024
[enhancement](nereids) only push having as agg's parent if having jus…
a902716
…t use slots from agg's output (#32847) pick from master #32414

morrySnow added dev/2.0.8-merged and removed dev/2.0.x labels Mar 26, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[enhancement](nereids) only push having as agg's parent if having jus…
ceedfc1
…t use slots from agg's output (apache#32847) pick from master apache#32414

Uh oh!

Conversation

starocean999 commented Mar 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 18, 2024

Uh oh!

starocean999 commented Mar 18, 2024

Uh oh!

starocean999 commented Mar 19, 2024

Uh oh!

starocean999 commented Mar 20, 2024

Uh oh!

starocean999 commented Mar 20, 2024

Uh oh!

Uh oh!

Uh oh!

starocean999 commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

starocean999 commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

github-actionsBot commented Mar 22, 2024

Uh oh!

github-actionsBot commented Mar 22, 2024

Uh oh!

starocean999 commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

github-actionsBot commented Mar 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

starocean999 commented Mar 18, 2024•
edited
Loading