[fix](Nereids) should push project through limit after eliminate union node by morrySnow · Pull Request #39640 · apache/doris

morrySnow · 2024-08-20T11:51:28Z

other wise:

push limit through union could generate plan:

limit
+-- union
|-- limit
| +-- empty relation
+-- limit
+-- project

and then eliminate union will generate plan:

+-- limit
+- project
+-- limit
+-- project

it could not be processed by tranlator correctly

Proposed changes

Issue Number: close #xxx

doris-robot · 2024-08-20T11:51:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morrySnow · 2024-08-20T11:51:35Z

run buildall

morrySnow · 2024-08-20T12:04:30Z

run buildall

doris-robot · 2024-08-20T14:44:24Z

TPC-H: Total hot run time: 37687 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1e599b4487d8ebfcafe2028a051c59f0be4d518, data reload: false
------ Round 1 ----------------------------------
q1	17885	4407	4301	4301
q2	2063	206	203	203
q3	11623	1065	1045	1045
q4	10538	810	722	722
q5	7773	2864	2818	2818
q6	266	158	156	156
q7	1006	654	652	652
q8	9602	2108	2158	2108
q9	8726	6512	6551	6512
q10	6988	1774	1805	1774
q11	484	277	279	277
q12	430	261	248	248
q13	18856	3018	3009	3009
q14	289	251	254	251
q15	550	527	522	522
q16	516	412	404	404
q17	976	648	699	648
q18	7428	6702	6804	6702
q19	6525	1071	1124	1071
q20	681	361	359	359
q21	3955	2909	2888	2888
q22	1106	1022	1017	1017
Total cold run time: 118266 ms
Total hot run time: 37687 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4547	4284	4297	4284
q2	401	308	283	283
q3	2780	2558	2640	2558
q4	1996	1720	1663	1663
q5	5611	5652	5914	5652
q6	248	150	149	149
q7	2194	1795	1765	1765
q8	3284	3517	3473	3473
q9	8764	8718	8769	8718
q10	3070	2838	2830	2830
q11	647	549	519	519
q12	858	632	647	632
q13	16977	3171	3215	3171
q14	315	286	294	286
q15	574	522	532	522
q16	496	454	453	453
q17	1824	1567	1513	1513
q18	8200	7996	7612	7612
q19	9729	1564	1551	1551
q20	2211	1890	1858	1858
q21	13927	5261	5196	5196
q22	1200	1066	1074	1066
Total cold run time: 89853 ms
Total hot run time: 55754 ms

morrySnow · 2024-08-21T03:05:28Z

run buildall

other wise: push limit through union could generate plan: ``` limit +-- union |-- limit | +-- empty relation +-- limit +-- project ``` and then eliminate union will generate plan: ``` +-- limit +- project +-- limit +-- project ``` it could not be processed by tranlator correctly

morrySnow · 2024-08-21T03:13:32Z

run buildall

doris-robot · 2024-08-21T03:35:01Z

TPC-H: Total hot run time: 38371 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7ae49a269829bb3928f4bc391b1d9301b625d79, data reload: false
------ Round 1 ----------------------------------
q1	18187	4443	4481	4443
q2	2161	218	224	218
q3	10394	1195	1167	1167
q4	10177	688	818	688
q5	7771	2825	2850	2825
q6	261	158	160	158
q7	1013	667	656	656
q8	9393	2104	2106	2104
q9	7211	6554	6543	6543
q10	7047	2284	2196	2196
q11	516	279	275	275
q12	437	261	260	260
q13	18057	3004	3013	3004
q14	307	249	260	249
q15	567	528	537	528
q16	510	414	411	411
q17	977	699	680	680
q18	7448	6803	6793	6793
q19	7197	1133	1094	1094
q20	700	376	355	355
q21	3955	2916	2669	2669
q22	1134	1070	1055	1055
Total cold run time: 115420 ms
Total hot run time: 38371 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4509	4334	4296	4296
q2	423	323	320	320
q3	2869	2641	2579	2579
q4	1925	1652	1689	1652
q5	5383	5447	5365	5365
q6	241	152	151	151
q7	2060	1709	1742	1709
q8	3209	3338	3397	3338
q9	8403	8323	8367	8323
q10	3430	3199	3195	3195
q11	632	530	529	529
q12	825	636	644	636
q13	16651	3008	3006	3006
q14	304	296	290	290
q15	552	508	514	508
q16	492	441	454	441
q17	1791	1503	1516	1503
q18	7701	8735	8019	8019
q19	1713	1683	1568	1568
q20	2078	1855	1850	1850
q21	10744	4963	5188	4963
q22	1122	1025	1028	1025
Total cold run time: 77057 ms
Total hot run time: 55266 ms

doris-robot · 2024-08-21T03:47:15Z

TPC-DS: Total hot run time: 190607 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7ae49a269829bb3928f4bc391b1d9301b625d79, data reload: false
query1	994	411	385	385
query2	6733	1996	1919	1919
query3	6657	234	248	234
query4	34130	23169	23126	23126
query5	4430	713	680	680
query6	310	206	208	206
query7	4615	333	325	325
query8	468	430	429	429
query9	8610	2512	2499	2499
query10	513	370	339	339
query11	16269	15055	14979	14979
query12	193	137	134	134
query13	1715	478	456	456
query14	9418	7595	7388	7388
query15	260	205	188	188
query16	7625	497	519	497
query17	1613	587	583	583
query18	1936	332	344	332
query19	219	172	170	170
query20	141	134	135	134
query21	243	141	137	137
query22	4126	4002	3912	3912
query23	34019	33273	33097	33097
query24	11605	2937	2870	2870
query25	653	424	426	424
query26	1129	175	180	175
query27	2618	308	305	305
query28	7036	2129	2087	2087
query29	782	461	443	443
query30	343	187	182	182
query31	1054	831	822	822
query32	122	78	80	78
query33	808	329	378	329
query34	916	518	496	496
query35	913	791	739	739
query36	1122	945	974	945
query37	163	101	103	101
query38	3945	3886	3897	3886
query39	1508	1470	1463	1463
query40	250	151	152	151
query41	137	133	132	132
query42	132	118	118	118
query43	533	499	515	499
query44	1225	780	796	780
query45	220	194	209	194
query46	1138	788	794	788
query47	1932	1822	1807	1807
query48	418	341	335	335
query49	1219	573	578	573
query50	861	474	469	469
query51	7272	7038	7075	7038
query52	122	107	109	107
query53	298	226	226	226
query54	984	515	506	506
query55	92	89	89	89
query56	327	308	301	301
query57	1229	1094	1123	1094
query58	306	296	325	296
query59	3022	3002	2731	2731
query60	344	363	365	363
query61	191	184	142	142
query62	877	703	706	703
query63	257	224	229	224
query64	5357	2404	1809	1809
query65	3218	3344	3216	3216
query66	1279	664	663	663
query67	15513	14883	15089	14883
query68	4697	578	582	578
query69	699	376	323	323
query70	1165	1197	1196	1196
query71	483	312	324	312
query72	7507	2299	2053	2053
query73	813	358	364	358
query74	9350	8801	8800	8800
query75	4204	2668	2730	2668
query76	3464	1081	1057	1057
query77	871	436	451	436
query78	10027	9081	9022	9022
query79	8439	548	552	548
query80	1376	635	613	613
query81	626	261	262	261
query82	1343	164	161	161
query83	364	213	212	212
query84	294	99	98	98
query85	1523	362	349	349
query86	473	297	325	297
query87	4469	4202	4274	4202
query88	4958	2540	2525	2525
query89	575	330	326	326
query90	1963	233	236	233
query91	159	127	125	125
query92	89	76	76	76
query93	6645	573	562	562
query94	1011	332	320	320
query95	390	298	292	292
query96	630	284	284	284
query97	3279	3045	3040	3040
query98	255	232	236	232
query99	1644	1317	1321	1317
Total cold run time: 312492 ms
Total hot run time: 190607 ms

doris-robot · 2024-08-21T03:52:46Z

ClickBench: Total hot run time: 31.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7ae49a269829bb3928f4bc391b1d9301b625d79, data reload: false
query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.07	0.08
query5	0.51	0.49	0.51
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.05	0.04
query9	0.57	0.50	0.49
query10	0.56	0.54	0.54
query11	0.17	0.14	0.12
query12	0.15	0.13	0.14
query13	0.61	0.59	0.59
query14	0.77	0.78	0.79
query15	0.83	0.82	0.82
query16	0.38	0.38	0.38
query17	1.00	1.06	1.08
query18	0.22	0.20	0.20
query19	1.86	1.74	1.76
query20	0.02	0.01	0.01
query21	15.43	0.67	0.66
query22	4.05	6.51	2.16
query23	18.28	1.41	1.27
query24	2.16	0.23	0.22
query25	0.15	0.09	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.09
query28	13.26	1.02	1.01
query29	12.58	3.39	3.37
query30	0.42	0.25	0.25
query31	2.81	0.39	0.38
query32	3.27	0.49	0.48
query33	2.97	2.97	2.94
query34	16.92	4.42	4.38
query35	4.43	4.38	4.42
query36	0.69	0.49	0.52
query37	0.21	0.18	0.17
query38	0.17	0.17	0.17
query39	0.07	0.06	0.07
query40	0.18	0.14	0.14
query41	0.11	0.06	0.06
query42	0.08	0.06	0.06
query43	0.06	0.05	0.05
Total cold run time: 109.54 s
Total hot run time: 31.35 s

morrySnow · 2024-08-21T08:48:29Z

run cloud_p1

morrySnow · 2024-08-21T08:49:59Z

run feut

morrySnow · 2024-08-21T08:50:10Z

run external

morrySnow · 2024-08-21T11:29:53Z

run buildall

doris-robot · 2024-08-21T12:11:17Z

TPC-H: Total hot run time: 38040 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 334a3a39221c7a97a29aef6cdf58bfff97495efa, data reload: false
------ Round 1 ----------------------------------
q1	17842	4477	4312	4312
q2	2047	208	205	205
q3	11818	1002	1149	1002
q4	10517	809	779	779
q5	7769	2853	2828	2828
q6	264	164	162	162
q7	1047	669	656	656
q8	9596	2101	2163	2101
q9	8699	6544	6591	6544
q10	7092	2267	2215	2215
q11	483	293	283	283
q12	441	267	277	267
q13	17778	2988	3006	2988
q14	304	267	268	267
q15	573	540	545	540
q16	535	430	409	409
q17	984	701	671	671
q18	7378	6751	6720	6720
q19	6311	1074	1099	1074
q20	686	359	348	348
q21	3892	2662	2897	2662
q22	1121	1014	1007	1007
Total cold run time: 117177 ms
Total hot run time: 38040 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4541	4344	4305	4305
q2	395	290	301	290
q3	2863	2655	2667	2655
q4	1999	1729	1702	1702
q5	5687	5715	5721	5715
q6	251	155	154	154
q7	2196	1848	1795	1795
q8	3297	3498	3473	3473
q9	8809	8803	8770	8770
q10	3623	3365	3327	3327
q11	637	527	527	527
q12	850	693	668	668
q13	16996	3204	3178	3178
q14	329	291	303	291
q15	560	523	510	510
q16	498	459	467	459
q17	1849	1549	1511	1511
q18	8733	7935	7777	7777
q19	10272	1561	1666	1561
q20	2153	1866	1899	1866
q21	14144	5407	5252	5252
q22	1195	1075	1086	1075
Total cold run time: 91877 ms
Total hot run time: 56861 ms

doris-robot · 2024-08-21T12:24:04Z

TPC-DS: Total hot run time: 196786 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 334a3a39221c7a97a29aef6cdf58bfff97495efa, data reload: false
query1	1353	903	882	882
query2	6583	1971	1956	1956
query3	10748	3860	3757	3757
query4	57325	24446	23321	23321
query5	5823	722	701	701
query6	496	212	209	209
query7	5913	329	340	329
query8	525	444	443	443
query9	9154	2558	2533	2533
query10	577	353	334	334
query11	18402	15014	15280	15014
query12	203	143	146	143
query13	1647	425	424	424
query14	11231	7924	7265	7265
query15	304	199	198	198
query16	7565	548	542	542
query17	1308	621	623	621
query18	1644	345	346	345
query19	313	168	170	168
query20	147	139	136	136
query21	251	148	143	143
query22	4689	4315	4477	4315
query23	34371	34364	34065	34065
query24	6116	2944	2960	2944
query25	562	431	457	431
query26	696	179	189	179
query27	1757	305	311	305
query28	3680	2193	2178	2178
query29	688	462	452	452
query30	250	197	198	197
query31	1068	857	860	857
query32	102	81	79	79
query33	510	351	346	346
query34	937	522	523	522
query35	895	786	789	786
query36	1088	965	943	943
query37	153	103	103	103
query38	3981	3825	3791	3791
query39	1518	1479	1495	1479
query40	241	157	155	155
query41	141	138	137	137
query42	131	116	118	116
query43	535	506	517	506
query44	1118	774	784	774
query45	224	199	201	199
query46	1141	753	796	753
query47	1937	1846	1885	1846
query48	406	325	329	325
query49	926	580	581	580
query50	870	460	469	460
query51	7190	7160	7174	7160
query52	126	109	112	109
query53	296	223	224	223
query54	599	509	499	499
query55	88	91	91	91
query56	315	308	312	308
query57	1192	1140	1151	1140
query58	294	303	297	297
query59	2986	2852	2688	2688
query60	358	319	325	319
query61	152	150	145	145
query62	790	689	699	689
query63	253	226	223	223
query64	3346	1870	1838	1838
query65	3262	3184	3195	3184
query66	1016	696	675	675
query67	15455	15192	15140	15140
query68	6559	583	587	583
query69	743	401	323	323
query70	1215	1113	1120	1113
query71	534	325	316	316
query72	7212	2348	2053	2053
query73	818	357	355	355
query74	9291	8930	8905	8905
query75	4282	2755	2813	2755
query76	4028	1064	986	986
query77	914	432	440	432
query78	10118	9102	9011	9011
query79	6849	566	570	566
query80	1172	600	596	596
query81	607	264	263	263
query82	855	158	159	158
query83	432	215	215	215
query84	286	101	94	94
query85	995	360	351	351
query86	408	341	339	339
query87	4485	4152	4258	4152
query88	4676	2472	2435	2435
query89	449	326	326	326
query90	2110	225	226	225
query91	151	127	125	125
query92	84	78	76	76
query93	5934	551	549	549
query94	777	314	326	314
query95	389	300	297	297
query96	628	291	286	286
query97	3236	3070	3059	3059
query98	255	225	224	224
query99	1677	1303	1289	1289
Total cold run time: 332130 ms
Total hot run time: 196786 ms

doris-robot · 2024-08-21T12:29:39Z

ClickBench: Total hot run time: 32.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 334a3a39221c7a97a29aef6cdf58bfff97495efa, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.07
query5	0.52	0.49	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.54	0.50	0.49
query10	0.55	0.54	0.55
query11	0.15	0.12	0.12
query12	0.15	0.13	0.14
query13	0.62	0.60	0.60
query14	0.78	0.80	0.81
query15	0.85	0.84	0.83
query16	0.37	0.38	0.39
query17	1.06	1.00	0.97
query18	0.23	0.21	0.21
query19	1.92	1.77	1.79
query20	0.01	0.01	0.02
query21	15.44	0.68	0.67
query22	4.21	6.85	2.88
query23	18.30	1.47	1.32
query24	2.12	0.25	0.23
query25	0.16	0.08	0.08
query26	0.28	0.19	0.19
query27	0.09	0.08	0.08
query28	13.16	1.04	1.02
query29	12.64	3.33	3.31
query30	0.44	0.25	0.24
query31	2.79	0.41	0.41
query32	3.24	0.49	0.49
query33	2.98	3.01	2.99
query34	16.90	4.41	4.40
query35	4.47	4.46	4.42
query36	0.68	0.49	0.51
query37	0.22	0.17	0.17
query38	0.18	0.17	0.17
query39	0.06	0.06	0.06
query40	0.18	0.15	0.16
query41	0.11	0.07	0.08
query42	0.08	0.08	0.07
query43	0.07	0.07	0.06
Total cold run time: 109.77 s
Total hot run time: 32.3 s

github-actions · 2024-08-22T03:02:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-22T03:02:36Z

PR approved by anyone and no changes requested.

…n node (#39640) pick from master #39640 otherwise: push limit through union could generate plan: limit +-- union |-- limit | +-- empty relation +-- limit +-- project and then eliminate union will generate plan: +-- limit +- project +-- limit +-- project it could not be processed by tranlator correctly

…n node (apache#39640) pick from master apache#39640 otherwise: push limit through union could generate plan: limit +-- union |-- limit | +-- empty relation +-- limit +-- project and then eliminate union will generate plan: +-- limit +- project +-- limit +-- project it could not be processed by tranlator correctly

…n node (#39640) (#39761) pick from master #39640 otherwise: push limit through union could generate plan: limit +-- union |-- limit | +-- empty relation +-- limit +-- project and then eliminate union will generate plan: +-- limit +- project +-- limit +-- project it could not be processed by tranlator correctly

…n node (#39640) (#39755) pick from master #39640 otherwise: push limit through union could generate plan: limit +-- union |-- limit | +-- empty relation +-- limit +-- project and then eliminate union will generate plan: +-- limit +- project +-- limit +-- project it could not be processed by tranlator correctly

…n node (#39640) otherwise: push limit through union could generate plan: limit +-- union |-- limit | +-- empty relation +-- limit +-- project and then eliminate union will generate plan: +-- limit +- project +-- limit +-- project it could not be processed by tranlator correctly

…n node (apache#39640) (apache#39761) pick from master apache#39640 otherwise: push limit through union could generate plan: limit +-- union |-- limit | +-- empty relation +-- limit +-- project and then eliminate union will generate plan: +-- limit +- project +-- limit +-- project it could not be processed by tranlator correctly

…n node (apache#39640) otherwise: push limit through union could generate plan: limit +-- union |-- limit | +-- empty relation +-- limit +-- project and then eliminate union will generate plan: +-- limit +- project +-- limit +-- project it could not be processed by tranlator correctly

morrySnow added dev/2.0.x dev/2.1.x usercase Important user case type label labels Aug 20, 2024

morrySnow force-pushed the fix_push_limit_conflict_with_union_elimiante branch from 9399a78 to a1e599bCompare August 20, 2024 12:04

morrySnow force-pushed the fix_push_limit_conflict_with_union_elimiante branch from a1e599b to f00331cCompare August 21, 2024 03:05

morrySnow force-pushed the fix_push_limit_conflict_with_union_elimiante branch from f00331c to c7ae49aCompare August 21, 2024 03:13

morrySnow changed the title ~~[fix](Nereids) eliminate union should execute before limit push down~~[fix](Nereids) should push project through limit after eliminate union nodeAug 21, 2024

morrySnow marked this pull request as ready for review August 21, 2024 08:51

Merge branch 'master' into fix_push_limit_conflict_with_union_elimiante
334a3a3

924060929 approved these changes Aug 22, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2024

github-actionsBot added the reviewed label Aug 22, 2024

starocean999 approved these changes Aug 22, 2024
View reviewed changes

morrySnow merged commit ed1a32b into apache:masterAug 22, 2024

morrySnow deleted the fix_push_limit_conflict_with_union_elimiante branch August 22, 2024 03:21

morrySnow mentioned this pull request Aug 22, 2024
[fix](Nereids) should push project through limit after eliminate union node (#39640) #39755
Merged

morrySnow mentioned this pull request Aug 22, 2024
[fix](Nereids) should push project through limit after eliminate union node (#39640) #39761
Merged

morrySnow added dev/2.1.6-merged dev/2.0.15-merged and removed dev/2.0.x dev/2.1.x labels Aug 22, 2024

dataroaring added dev/3.0.x dev/3.0.2-merged and removed dev/3.0.x labels Aug 26, 2024

yiguolei mentioned this pull request Sep 5, 2024
Release Note 2.1.6 #40443
Closed
3 tasks

Uh oh!

Conversation

morrySnow commented Aug 20, 2024

Proposed changes

Uh oh!

doris-robot commented Aug 20, 2024

Uh oh!

morrySnow commented Aug 20, 2024

Uh oh!

morrySnow commented Aug 20, 2024

Uh oh!

doris-robot commented Aug 20, 2024

Uh oh!

morrySnow commented Aug 21, 2024

Uh oh!

morrySnow commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

morrySnow commented Aug 21, 2024

Uh oh!

morrySnow commented Aug 21, 2024

Uh oh!

morrySnow commented Aug 21, 2024

Uh oh!

morrySnow commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

github-actionsBot commented Aug 22, 2024

Uh oh!

github-actionsBot commented Aug 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants