Skip to content

[Fix](nereids) fix condition function partition prune - #39298

Merged
morrySnow merged 2 commits into
apache:masterfrom
feiniaofeiafei:fix_condition_func_partition_prune
Aug 14, 2024
Merged

[Fix](nereids) fix condition function partition prune#39298
morrySnow merged 2 commits into
apache:masterfrom
feiniaofeiafei:fix_condition_func_partition_prune

Conversation

@feiniaofeiafei

@feiniaofeiafeifeiniaofeiafei commented Aug 13, 2024•
edited
Loading

Copy link
Copy Markdown
Collaborator

Current partition prune logic cannot correctly deal with condition function in predicates. For example, where if(substring('abc',1,2) <> "case", a > 0, b > 0), if the first parameter is not fold by FoldConstantRule, then our current logic will also intersect the partition range of b with b>0. If one of the partitions is empty, the partition will be trimmed. However, substring('abc',1,2) <> "case" is true, the predicate b>0 is not useful and should not be involved in partition trimming.
This pr remove the logic that make the result to "Boolean.False" when considering the child ranges and finding an empty range.

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@feiniaofeiafei

feiniaofeiafei commented Aug 13, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40150 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e40384d43a517f93fc290da5319b8f4b4b580410, data reload: false
------ Round 1 ----------------------------------
q1	18810	4345	4280	4280
q2	2007	178	175	175
q3	12067	1275	1162	1162
q4	10532	717	735	717
q5	7817	2914	2850	2850
q6	231	148	153	148
q7	1008	625	649	625
q8	10287	2078	2091	2078
q9	10604	6745	6635	6635
q10	7133	2169	2186	2169
q11	478	240	252	240
q12	386	230	223	223
q13	17990	3069	3038	3038
q14	285	233	243	233
q15	540	483	496	483
q16	510	411	394	394
q17	969	726	667	667
q18	7987	7270	7339	7270
q19	3901	1088	1067	1067
q20	751	327	346	327
q21	5380	4361	4500	4361
q22	1123	1020	1008	1008
Total cold run time: 120796 ms
Total hot run time: 40150 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4550	4298	4248	4248
q2	373	285	271	271
q3	2976	2760	2714	2714
q4	2020	1730	1658	1658
q5	5610	5523	5560	5523
q6	229	133	137	133
q7	2118	1758	1777	1758
q8	3262	3421	3364	3364
q9	8711	8832	8745	8745
q10	3420	3158	3320	3158
q11	593	505	497	497
q12	806	660	638	638
q13	17125	3140	3135	3135
q14	319	280	286	280
q15	529	502	480	480
q16	489	434	430	430
q17	1815	1540	1523	1523
q18	8219	8059	7833	7833
q19	1734	1602	1475	1475
q20	2175	1875	1911	1875
q21	8349	5139	5427	5139
q22	1118	1077	1011	1011
Total cold run time: 76540 ms
Total hot run time: 55888 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 190103 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e40384d43a517f93fc290da5319b8f4b4b580410, data reload: false
query1	1269	881	850	850
query2	6336	1898	1860	1860
query3	10610	3911	3559	3559
query4	58946	24939	23196	23196
query5	5878	519	498	498
query6	540	199	158	158
query7	6494	289	293	289
query8	301	186	195	186
query9	9148	2433	2416	2416
query10	512	263	256	256
query11	15883	15159	15103	15103
query12	161	98	99	98
query13	1618	387	385	385
query14	12198	7803	7269	7269
query15	234	181	181	181
query16	7204	513	495	495
query17	1155	588	588	588
query18	1869	323	307	307
query19	272	158	159	158
query20	117	115	113	113
query21	212	99	99	99
query22	4682	4440	4395	4395
query23	34236	33871	33479	33479
query24	5613	2846	2815	2815
query25	550	397	408	397
query26	691	164	159	159
query27	1776	282	278	278
query28	3644	2073	2034	2034
query29	672	423	434	423
query30	235	153	148	148
query31	930	742	740	740
query32	83	52	57	52
query33	486	291	286	286
query34	843	463	481	463
query35	798	725	719	719
query36	1085	932	913	913
query37	144	83	82	82
query38	3890	3788	3786	3786
query39	1446	1413	1386	1386
query40	197	129	117	117
query41	49	46	49	46
query42	131	99	95	95
query43	506	458	473	458
query44	1067	726	751	726
query45	203	164	160	160
query46	1085	775	743	743
query47	1835	1768	1754	1754
query48	379	296	300	296
query49	770	427	450	427
query50	801	399	411	399
query51	6835	6797	6694	6694
query52	101	96	92	92
query53	254	184	183	183
query54	587	455	454	454
query55	78	76	75	75
query56	272	259	254	254
query57	1127	1030	1067	1030
query58	339	227	228	227
query59	2802	2799	2565	2565
query60	296	260	263	260
query61	96	95	98	95
query62	734	645	631	631
query63	236	177	177	177
query64	3243	1718	1680	1680
query65	3192	3116	3153	3116
query66	702	325	325	325
query67	15348	15105	14959	14959
query68	6247	560	551	551
query69	633	392	290	290
query70	1132	1112	1121	1112
query71	501	273	273	273
query72	7161	2204	2088	2088
query73	781	323	325	323
query74	9244	8960	8723	8723
query75	3757	2698	2734	2698
query76	3485	968	987	968
query77	732	325	312	312
query78	9740	9010	8997	8997
query79	7775	523	527	523
query80	2346	493	490	490
query81	574	221	224	221
query82	1076	134	138	134
query83	287	147	145	145
query84	273	120	77	77
query85	1327	276	268	268
query86	393	295	274	274
query87	4331	4148	4217	4148
query88	4652	2405	2401	2401
query89	551	288	294	288
query90	1969	202	202	202
query91	126	97	93	93
query92	63	52	51	51
query93	6568	539	538	538
query94	719	288	292	288
query95	340	255	265	255
query96	611	270	269	269
query97	3220	3002	3063	3002
query98	211	202	206	202
query99	1544	1259	1242	1242
Total cold run time: 328395 ms
Total hot run time: 190103 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e40384d43a517f93fc290da5319b8f4b4b580410, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.22	0.04	0.05
query4	1.68	0.09	0.07
query5	0.47	0.50	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.61	0.58
query14	0.78	0.77	0.79
query15	0.85	0.83	0.81
query16	0.35	0.37	0.37
query17	0.98	1.05	0.97
query18	0.23	0.22	0.22
query19	1.92	1.66	1.74
query20	0.01	0.01	0.00
query21	15.42	0.74	0.65
query22	3.71	7.11	2.19
query23	18.31	1.38	1.23
query24	2.07	0.22	0.22
query25	0.14	0.08	0.08
query26	0.29	0.22	0.22
query27	0.46	0.23	0.22
query28	13.32	1.01	1.00
query29	12.60	3.31	3.28
query30	0.24	0.05	0.05
query31	2.90	0.40	0.39
query32	3.26	0.49	0.47
query33	2.92	2.94	2.98
query34	17.09	4.32	4.32
query35	4.47	4.39	4.39
query36	0.65	0.48	0.48
query37	0.19	0.15	0.16
query38	0.15	0.16	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.42 s
Total hot run time: 30.79 s

morrySnow
morrySnow previously approved these changes Aug 13, 2024
View reviewed changes
@morrySnowmorrySnow added usercase Important user case type label dev/2.0.x dev/2.1.x and removed doing labels Aug 13, 2024
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 13, 2024
@github-actions

github-actionsBot commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@feiniaofeiafei

feiniaofeiafei commented Aug 13, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run cloud_p0

@feiniaofeiafei

feiniaofeiafei commented Aug 13, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run cloud_p1

@github-actionsgithub-actionsBot added doing and removed approved Indicates a PR has been approved by one committer. labels Aug 13, 2024
@feiniaofeiafei

feiniaofeiafei commented Aug 13, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39574 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed0ccadc5fd5c1bd74d34158797eab78c7afa6d6, data reload: false
------ Round 1 ----------------------------------
q1	17625	4331	4236	4236
q2	2019	182	177	177
q3	11810	974	1044	974
q4	10535	718	737	718
q5	7761	2836	2734	2734
q6	222	134	134	134
q7	941	592	589	589
q8	9518	2025	2038	2025
q9	10210	6543	6559	6543
q10	6999	2262	2186	2186
q11	473	247	249	247
q12	391	225	216	216
q13	17780	3057	2984	2984
q14	269	235	238	235
q15	521	479	468	468
q16	470	397	388	388
q17	976	669	773	669
q18	7987	7460	7435	7435
q19	7201	1032	969	969
q20	687	340	321	321
q21	5381	4361	4410	4361
q22	1110	993	965	965
Total cold run time: 120886 ms
Total hot run time: 39574 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4449	4239	4259	4239
q2	364	273	264	264
q3	2970	2765	2857	2765
q4	2035	1722	1714	1714
q5	5578	5558	5609	5558
q6	233	137	132	132
q7	2158	1771	1776	1771
q8	3317	3437	3407	3407
q9	8709	8871	8841	8841
q10	3404	3198	3324	3198
q11	616	521	514	514
q12	823	649	638	638
q13	17049	3210	3233	3210
q14	325	292	308	292
q15	538	488	495	488
q16	509	436	424	424
q17	1819	1523	1502	1502
q18	8156	8012	7923	7923
q19	3857	1567	1545	1545
q20	2987	1918	1881	1881
q21	5584	5299	5266	5266
q22	1119	1076	1056	1056
Total cold run time: 76599 ms
Total hot run time: 56628 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 190215 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed0ccadc5fd5c1bd74d34158797eab78c7afa6d6, data reload: false
query1	1261	898	867	867
query2	6369	1919	1862	1862
query3	10595	3887	3564	3564
query4	58468	23714	23295	23295
query5	5760	497	482	482
query6	491	161	166	161
query7	6468	289	294	289
query8	326	208	206	206
query9	8984	2435	2432	2432
query10	495	266	265	265
query11	17712	15021	15089	15021
query12	166	103	102	102
query13	1570	392	401	392
query14	11802	7850	7968	7850
query15	304	175	174	174
query16	7699	502	484	484
query17	1175	594	589	589
query18	2080	316	314	314
query19	285	156	169	156
query20	127	117	113	113
query21	214	110	112	110
query22	4467	4197	4161	4161
query23	34219	33423	33018	33018
query24	5607	2867	2927	2867
query25	547	405	410	405
query26	685	159	162	159
query27	1797	273	277	273
query28	3859	2056	2059	2056
query29	712	424	428	424
query30	355	152	154	152
query31	922	721	755	721
query32	90	59	58	58
query33	444	296	285	285
query34	858	462	467	462
query35	838	725	733	725
query36	1054	962	927	927
query37	134	84	84	84
query38	3910	3809	3768	3768
query39	1437	1377	1393	1377
query40	208	122	117	117
query41	47	45	48	45
query42	127	98	97	97
query43	486	469	474	469
query44	1054	733	737	733
query45	197	166	166	166
query46	1114	772	751	751
query47	1839	1744	1767	1744
query48	372	308	296	296
query49	771	425	455	425
query50	813	412	402	402
query51	6754	6740	6751	6740
query52	101	95	89	89
query53	264	186	183	183
query54	571	456	545	456
query55	76	71	74	71
query56	285	249	253	249
query57	1138	1076	1017	1017
query58	228	219	224	219
query59	2986	2691	2734	2691
query60	308	263	263	263
query61	101	95	95	95
query62	764	636	659	636
query63	207	189	192	189
query64	3173	1724	1733	1724
query65	3195	3149	3157	3149
query66	673	332	330	330
query67	15160	14648	14756	14648
query68	5147	539	540	539
query69	559	263	280	263
query70	1162	1178	1141	1141
query71	527	305	274	274
query72	7881	2292	2016	2016
query73	782	330	325	325
query74	9289	8778	8795	8778
query75	4832	2708	2716	2708
query76	3443	1050	929	929
query77	703	314	310	310
query78	9788	9190	9013	9013
query79	4572	524	534	524
query80	2256	489	497	489
query81	573	227	227	227
query82	909	137	138	137
query83	296	146	150	146
query84	277	82	78	78
query85	1714	319	271	271
query86	440	306	299	299
query87	4420	4265	4252	4252
query88	4597	2427	2389	2389
query89	416	295	297	295
query90	2035	199	196	196
query91	121	97	95	95
query92	59	49	50	49
query93	5359	549	534	534
query94	1001	292	307	292
query95	352	263	268	263
query96	602	273	277	273
query97	3182	3000	3075	3000
query98	231	223	210	210
query99	1515	1256	1307	1256
Total cold run time: 326338 ms
Total hot run time: 190215 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed0ccadc5fd5c1bd74d34158797eab78c7afa6d6, data reload: false
query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.09
query5	0.50	0.49	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.05
query9	0.55	0.50	0.48
query10	0.54	0.52	0.53
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.62	0.58
query14	0.76	0.80	0.78
query15	0.85	0.82	0.81
query16	0.37	0.37	0.37
query17	0.98	1.05	1.05
query18	0.23	0.23	0.22
query19	1.93	1.84	1.77
query20	0.01	0.01	0.01
query21	15.39	0.73	0.65
query22	4.00	7.59	2.14
query23	18.28	1.35	1.29
query24	2.12	0.23	0.21
query25	0.15	0.08	0.08
query26	0.29	0.22	0.22
query27	0.45	0.23	0.23
query28	13.26	1.03	1.00
query29	12.66	3.35	3.32
query30	0.24	0.05	0.05
query31	2.89	0.40	0.40
query32	3.26	0.49	0.49
query33	2.96	3.02	3.02
query34	16.78	4.40	4.34
query35	4.41	4.40	4.42
query36	0.66	0.46	0.47
query37	0.18	0.17	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.12
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.37 s
Total hot run time: 31.16 s

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 14, 2024
@github-actions

github-actionsBot commented Aug 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow
morrySnow merged commit 1430449 into apache:masterAug 14, 2024
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Aug 14, 2024
@feiniaofeiafei
[Fix](nereids) fix condition function partition prune (apache#39298)
93568d0
Current partition prune logic cannot correctly deal with condition
function in predicates. For example, where if(substring('abc',1,2) <>
"case", a > 0, b > 0), if the first parameter is not fold by
FoldConstantRule, then our current logic will also intersect the
partition range of b with b>0. If one of the partitions is empty, the
partition will be trimmed. However, substring('abc',1,2) <> "case" is
true, the predicate b>0 is not useful and should not be involved in
partition trimming.
This pr remove the logic that make the result to "Boolean.False" when
considering the child ranges and finding an empty range.
@feiniaofeiafeifeiniaofeiafei mentioned this pull request Aug 14, 2024
Merged
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Aug 14, 2024
@feiniaofeiafei
[Fix](nereids) fix condition function partition prune (apache#39298)
240149f
Current partition prune logic cannot correctly deal with condition
function in predicates. For example, where if(substring('abc',1,2) <>
"case", a > 0, b > 0), if the first parameter is not fold by
FoldConstantRule, then our current logic will also intersect the
partition range of b with b>0. If one of the partitions is empty, the
partition will be trimmed. However, substring('abc',1,2) <> "case" is
true, the predicate b>0 is not useful and should not be involved in
partition trimming.
This pr remove the logic that make the result to "Boolean.False" when
considering the child ranges and finding an empty range.
@feiniaofeiafeifeiniaofeiafei mentioned this pull request Aug 14, 2024
Merged
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@feiniaofeiafei@wyxxxcat
[Fix](nereids) fix condition function partition prune (apache#39298)
79dfd07
Current partition prune logic cannot correctly deal with condition
function in predicates. For example, where if(substring('abc',1,2) <>
"case", a > 0, b > 0), if the first parameter is not fold by
FoldConstantRule, then our current logic will also intersect the
partition range of b with b>0. If one of the partitions is empty, the
partition will be trimmed. However, substring('abc',1,2) <> "case" is
true, the predicate b>0 is not useful and should not be involved in
partition trimming.
This pr remove the logic that make the result to "Boolean.False" when
considering the child ranges and finding an empty range.
yiguolei pushed a commit that referenced this pull request Aug 14, 2024
@feiniaofeiafei
[Fix](nereids) fix condition function partition prune (#39298) (#39332)
701e23b
cherry-pick #39298 to branch-2.1
morrySnow pushed a commit that referenced this pull request Aug 16, 2024
@feiniaofeiafei
[Fix](nereids) fix condition function partition prune (#39298) (#39331)
787cf16
cherry-pick from master #39298
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@feiniaofeiafei@dataroaring
[Fix](nereids) fix condition function partition prune (#39298)
3da9d7d
Current partition prune logic cannot correctly deal with condition
function in predicates. For example, where if(substring('abc',1,2) <>
"case", a > 0, b > 0), if the first parameter is not fold by
FoldConstantRule, then our current logic will also intersect the
partition range of b with b>0. If one of the partitions is empty, the
partition will be trimmed. However, substring('abc',1,2) <> "case" is
true, the predicate b>0 is not useful and should not be involved in
partition trimming.
This pr remove the logic that make the result to "Boolean.False" when
considering the child ranges and finding an empty range.
@yiguoleiyiguolei mentioned this pull request Sep 5, 2024
Closed
3 tasks
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@feiniaofeiafei@Baymine
[Fix](nereids) fix condition function partition prune (apache#39298) (a…
5818ea8
…pache#39331)
cherry-pick from master apache#39298
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@feiniaofeiafei@HappenLee
[Fix](nereids) fix condition function partition prune (apache#39298)
72041c9
Current partition prune logic cannot correctly deal with condition
function in predicates. For example, where if(substring('abc',1,2) <>
"case", a > 0, b > 0), if the first parameter is not fold by
FoldConstantRule, then our current logic will also intersect the
partition range of b with b>0. If one of the partitions is empty, the
partition will be trimmed. However, substring('abc',1,2) <> "case" is
true, the predicate b>0 is not useful and should not be involved in
partition trimming.
This pr remove the logic that make the result to "Boolean.False" when
considering the child ranges and finding an empty range.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929924060929924060929 approved these changes
@morrySnowmorrySnowmorrySnow approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.15-mergeddev/2.1.6-mergeddev/3.0.2-mergeddoingreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@feiniaofeiafei@doris-robot@924060929@morrySnow@yiguolei@dataroaring