[feat](nereids) simplify conflict compound by yujun777 · Pull Request #47385 · apache/doris

yujun777 · 2025-01-24T02:28:34Z

What problem does this PR solve?

Add an expression rewrite rule, it will simplify like:

x and not x => falseOrNull(x)
x or not x => trueOrNull(x)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-01-24T02:28:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

yujun777 · 2025-01-24T02:30:24Z

run buildall

yujun777 · 2025-01-24T02:59:15Z

run buildall

doris-robot · 2025-01-24T03:20:16Z

TPC-H: Total hot run time: 32353 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 62162e4e6974491a82afa0a1aac921204189b822, data reload: false
------ Round 1 ----------------------------------
q1	17582	5483	5369	5369
q2	2049	309	166	166
q3	10452	1230	772	772
q4	10525	958	551	551
q5	9394	2412	2165	2165
q6	203	162	133	133
q7	903	757	604	604
q8	9235	1342	1176	1176
q9	5235	4932	4893	4893
q10	6854	2353	1916	1916
q11	475	281	259	259
q12	346	360	225	225
q13	17764	3712	3105	3105
q14	229	241	201	201
q15	531	464	477	464
q16	650	615	582	582
q17	566	866	338	338
q18	6913	6351	6616	6351
q19	1635	952	542	542
q20	310	327	196	196
q21	2780	2246	2033	2033
q22	364	328	312	312
Total cold run time: 104995 ms
Total hot run time: 32353 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5495	5548	6150	5548
q2	246	333	238	238
q3	2227	2679	2318	2318
q4	1442	1822	1388	1388
q5	4332	4789	4783	4783
q6	177	171	131	131
q7	2040	1994	1844	1844
q8	2660	2882	2750	2750
q9	7343	7272	7356	7272
q10	3038	3328	2747	2747
q11	566	503	492	492
q12	651	765	625	625
q13	3613	3967	3316	3316
q14	300	286	284	284
q15	518	480	467	467
q16	648	690	671	671
q17	1208	1752	1286	1286
q18	7714	7635	7281	7281
q19	805	1198	1062	1062
q20	2023	2072	1934	1934
q21	5846	5144	4992	4992
q22	611	604	574	574
Total cold run time: 53503 ms
Total hot run time: 52003 ms

doris-robot · 2025-01-24T03:32:09Z

TPC-DS: Total hot run time: 192929 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 62162e4e6974491a82afa0a1aac921204189b822, data reload: false
query1	1326	956	955	955
query2	6373	1988	2065	1988
query3	11122	4582	4582	4582
query4	32298	23395	23153	23153
query5	3644	625	466	466
query6	303	211	193	193
query7	3985	488	321	321
query8	295	256	232	232
query9	9460	2580	2572	2572
query10	448	320	261	261
query11	17828	15438	15085	15085
query12	164	110	106	106
query13	1570	532	395	395
query14	9612	7536	7020	7020
query15	261	210	186	186
query16	7998	644	479	479
query17	1597	744	563	563
query18	2165	414	317	317
query19	190	191	159	159
query20	124	121	110	110
query21	213	126	104	104
query22	4721	4467	4466	4466
query23	34633	34340	33696	33696
query24	6470	2340	2311	2311
query25	477	453	410	410
query26	721	250	159	159
query27	1987	481	346	346
query28	5763	2478	2420	2420
query29	571	540	459	459
query30	214	189	167	167
query31	999	884	838	838
query32	94	66	59	59
query33	503	359	303	303
query34	758	890	517	517
query35	843	839	761	761
query36	1028	1059	944	944
query37	131	103	81	81
query38	4405	4258	4303	4258
query39	1491	1483	1463	1463
query40	203	123	109	109
query41	59	60	55	55
query42	131	112	112	112
query43	534	525	509	509
query44	1346	850	867	850
query45	184	179	166	166
query46	892	1050	658	658
query47	1908	1919	1855	1855
query48	394	415	323	323
query49	729	508	398	398
query50	664	682	402	402
query51	4315	4305	4258	4258
query52	109	107	99	99
query53	237	262	200	200
query54	499	518	427	427
query55	85	77	80	77
query56	262	250	264	250
query57	1199	1201	1129	1129
query58	268	231	236	231
query59	3179	3240	3170	3170
query60	276	280	276	276
query61	117	129	123	123
query62	779	733	693	693
query63	242	196	196	196
query64	3719	1042	665	665
query65	3425	3275	3369	3275
query66	793	412	316	316
query67	16289	15866	15688	15688
query68	1979	784	558	558
query69	464	318	271	271
query70	1300	1110	1102	1102
query71	348	284	259	259
query72	5336	3790	3880	3790
query73	660	749	364	364
query74	9947	8931	8792	8792
query75	3219	3138	2656	2656
query76	2865	1198	794	794
query77	506	358	278	278
query78	10068	10246	9496	9496
query79	1005	881	587	587
query80	1203	537	465	465
query81	527	280	236	236
query82	655	198	122	122
query83	256	167	152	152
query84	236	96	79	79
query85	761	343	304	304
query86	397	321	308	308
query87	4468	4509	4404	4404
query88	3713	2206	2278	2206
query89	412	336	306	306
query90	1577	193	193	193
query91	130	133	104	104
query92	59	55	54	54
query93	975	871	543	543
query94	636	416	298	298
query95	334	264	264	264
query96	492	614	292	292
query97	2805	2915	2776	2776
query98	226	229	195	195
query99	1291	1426	1266	1266
Total cold run time: 278472 ms
Total hot run time: 192929 ms

doris-robot · 2025-01-24T03:37:28Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 62162e4e6974491a82afa0a1aac921204189b822, data reload: false
query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.11
query5	0.43	0.42	0.41
query6	1.17	0.65	0.65
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.51	0.50
query10	0.57	0.59	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.63	0.60	0.59
query14	2.74	2.83	2.75
query15	0.90	0.83	0.84
query16	0.38	0.38	0.38
query17	1.01	1.08	1.07
query18	0.23	0.21	0.22
query19	1.93	1.84	2.01
query20	0.02	0.01	0.02
query21	15.37	0.97	0.60
query22	0.76	0.76	0.63
query23	15.32	1.41	0.57
query24	3.05	0.65	1.36
query25	0.23	0.07	0.21
query26	0.39	0.15	0.14
query27	0.04	0.06	0.05
query28	13.71	1.06	0.45
query29	12.56	3.88	3.24
query30	0.25	0.10	0.07
query31	2.82	0.60	0.38
query32	3.23	0.55	0.46
query33	3.04	3.17	3.01
query34	16.61	5.23	4.57
query35	4.61	4.56	4.57
query36	0.63	0.50	0.48
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.05 s
Total hot run time: 30.55 s

yujun777 · 2025-01-24T08:02:57Z

run p0

yujun777 · 2025-01-26T02:15:46Z

run p0

yujun777 · 2025-01-26T03:04:51Z

run p0

github-actions · 2025-02-14T06:59:14Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-14T06:59:16Z

PR approved by anyone and no changes requested.

…ly to join (#47985) example sql: `SELECT * FROM t1 WHERE t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);` the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be replaced as `FALSE` by SimplifyConflictCompound rule. So we need check if correlated filter exists before converting apply to join Related PR: (#47385)

Add an expression rewrite rule, it will simplify like: ``` x and not x => falseOrNull(x) x or not x => trueOrNull(x) ```

…ly to join (apache#47985) example sql: `SELECT * FROM t1 WHERE t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);` the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be replaced as `FALSE` by SimplifyConflictCompound rule. So we need check if correlated filter exists before converting apply to join Related PR: (apache#47385)

Add an expression rewrite rule, it will simplify like: ``` x and not x => falseOrNull(x) x or not x => trueOrNull(x) ```

…ly to join (apache#47985) example sql: `SELECT * FROM t1 WHERE t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);` the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be replaced as `FALSE` by SimplifyConflictCompound rule. So we need check if correlated filter exists before converting apply to join Related PR: (apache#47385)

pick #41222 #46468 #44574 #48034 #50265 #47385

yujun777 added 2 commits January 23, 2025 22:03

add expression simplify conflict predicate rule
26c4380

rename to SimplifyConflictCompound
9839d58

update
1a2580f

update
62162e4

morrySnow approved these changes Feb 14, 2025
View reviewed changes

morrySnow requested a review from starocean999 February 14, 2025 06:58

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 14, 2025

github-actionsBot added the reviewed label Feb 14, 2025

starocean999 approved these changes Feb 14, 2025
View reviewed changes

starocean999 merged commit 46f12ee into apache:masterFeb 14, 2025

starocean999 mentioned this pull request Feb 19, 2025
[fix](nereids)check if correlated filter exists before converting apply to join #47985
Merged
16 tasks

lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
[feat](nereids) simplify conflict compound (apache#47385)
8c7b7b2
Add an expression rewrite rule, it will simplify like: ``` x and not x => falseOrNull(x) x or not x => trueOrNull(x) ```

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
[feat](nereids) simplify conflict compound (apache#47385)
2499436
Add an expression rewrite rule, it will simplify like: ``` x and not x => falseOrNull(x) x or not x => trueOrNull(x) ```

englefly mentioned this pull request Jun 20, 2025
branch-3.1: [opt](nereids) infer in-predicate from or-predicate (#46468) #51869
Merged

morrySnow pushed a commit that referenced this pull request Jun 20, 2025
branch-3.1: [opt](nereids) infer in-predicate from or-predicate (#51869)
622e21e
pick #41222 #46468 #44574 #48034 #50265 #47385

morrySnow added the dev/3.1.0-merged label Jun 20, 2025

Navigation Menu

Uh oh!

[feat](nereids) simplify conflict compound - #47385

[feat](nereids) simplify conflict compound#47385
starocean999 merged 4 commits into
apache:masterfrom
yujun777:conflict-predicate-2

yujun777 commented Jan 24, 2025•
edited
Loading

Uh oh!

Thearas commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 26, 2025

Uh oh!

yujun777 commented Jan 26, 2025

Uh oh!

github-actionsBot commented Feb 14, 2025

Uh oh!

github-actionsBot commented Feb 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

yujun777 commented Jan 24, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 24, 2025

Uh oh!

yujun777 commented Jan 26, 2025

Uh oh!

yujun777 commented Jan 26, 2025

Uh oh!

github-actionsBot commented Feb 14, 2025

Uh oh!

github-actionsBot commented Feb 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yujun777 commented Jan 24, 2025•
edited
Loading