[clean](planner)row policy removes dependency on the old optimizer by zddr · Pull Request #51122 · apache/doris

zddr · 2025-05-21T10:00:05Z

What problem does this PR solve?

Change "Expr" to "Expression" in RowPolicy
Remove the logic of rewriting using row policy in the old optimizer
Remove the logic for creating row policies based on CreatePolicyStmt

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

row policy removes dependency on the old optimizer

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-05-21T10:00:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zddr · 2025-05-21T10:00:25Z

run buildall

doris-robot · 2025-05-21T11:26:17Z

TPC-H: Total hot run time: 33963 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86e4534269800373001a8e3cb7645355adcba1c7, data reload: false
------ Round 1 ----------------------------------
q1	26345	5072	5105	5072
q2	2075	292	198	198
q3	10513	1288	697	697
q4	10248	985	525	525
q5	7837	2479	2336	2336
q6	188	164	137	137
q7	948	730	613	613
q8	9333	1336	1092	1092
q9	6825	5173	5127	5127
q10	6815	2310	1890	1890
q11	497	280	263	263
q12	351	352	218	218
q13	17776	3681	3096	3096
q14	230	222	213	213
q15	525	491	476	476
q16	424	430	377	377
q17	614	854	364	364
q18	7801	7218	7162	7162
q19	1732	971	546	546
q20	322	336	220	220
q21	3706	3155	2361	2361
q22	1043	1016	980	980
Total cold run time: 116148 ms
Total hot run time: 33963 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5176	5048	5120	5048
q2	244	329	234	234
q3	2184	2686	2305	2305
q4	1317	1916	1479	1479
q5	4577	4433	4369	4369
q6	216	171	126	126
q7	1962	1905	1760	1760
q8	2618	2561	2506	2506
q9	7207	7183	7165	7165
q10	2997	3176	2753	2753
q11	578	512	496	496
q12	695	801	610	610
q13	3538	3897	3273	3273
q14	294	302	280	280
q15	511	472	466	466
q16	447	491	464	464
q17	1145	1464	1412	1412
q18	7768	7661	7466	7466
q19	809	821	862	821
q20	1979	2013	1792	1792
q21	4763	4355	4169	4169
q22	1040	1017	964	964
Total cold run time: 52065 ms
Total hot run time: 49958 ms

doris-robot · 2025-05-21T11:37:48Z

TPC-DS: Total hot run time: 186337 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86e4534269800373001a8e3cb7645355adcba1c7, data reload: false
query1	996	480	511	480
query2	6585	1810	1816	1810
query3	6743	226	222	222
query4	26376	23462	23662	23462
query5	4324	630	440	440
query6	293	212	199	199
query7	4623	492	286	286
query8	282	235	228	228
query9	8592	2629	2639	2629
query10	505	340	290	290
query11	15395	15235	14855	14855
query12	156	116	102	102
query13	1645	516	405	405
query14	8842	6178	6167	6167
query15	205	195	210	195
query16	7137	611	506	506
query17	1180	704	556	556
query18	1974	414	294	294
query19	192	188	153	153
query20	119	114	121	114
query21	211	122	107	107
query22	4110	4057	3995	3995
query23	34068	33140	33224	33140
query24	8380	2373	2349	2349
query25	531	451	415	415
query26	1233	272	157	157
query27	2756	504	347	347
query28	4344	2124	2094	2094
query29	799	565	437	437
query30	282	216	183	183
query31	947	854	765	765
query32	71	66	63	63
query33	555	379	313	313
query34	831	898	534	534
query35	775	799	740	740
query36	967	1020	890	890
query37	114	110	77	77
query38	4199	4182	4118	4118
query39	1521	1422	1416	1416
query40	213	136	114	114
query41	63	58	61	58
query42	125	119	113	113
query43	507	498	493	493
query44	1332	841	842	841
query45	185	177	175	175
query46	861	1048	659	659
query47	1781	1797	1723	1723
query48	402	454	346	346
query49	805	543	459	459
query50	680	685	430	430
query51	4118	4055	4121	4055
query52	108	104	104	104
query53	222	251	194	194
query54	588	585	519	519
query55	88	86	85	85
query56	319	314	285	285
query57	1116	1158	1080	1080
query58	272	260	255	255
query59	2579	2604	2528	2528
query60	342	320	320	320
query61	127	158	124	124
query62	793	743	705	705
query63	233	188	194	188
query64	4398	1021	700	700
query65	4317	4278	4228	4228
query66	1134	415	317	317
query67	15664	15618	15424	15424
query68	7774	880	518	518
query69	484	305	324	305
query70	1189	1135	1079	1079
query71	421	332	300	300
query72	5600	4750	4858	4750
query73	680	646	354	354
query74	8905	9108	8806	8806
query75	3207	3210	2694	2694
query76	3206	1197	765	765
query77	536	366	293	293
query78	10077	10309	9264	9264
query79	1734	871	585	585
query80	630	518	440	440
query81	491	246	220	220
query82	233	139	99	99
query83	253	255	233	233
query84	248	108	87	87
query85	785	350	309	309
query86	372	333	281	281
query87	4414	4437	4338	4338
query88	2911	2337	2299	2299
query89	382	326	279	279
query90	1895	209	202	202
query91	152	145	111	111
query92	73	61	59	59
query93	1418	953	565	565
query94	653	433	289	289
query95	369	309	297	297
query96	505	571	285	285
query97	2727	2797	2660	2660
query98	227	217	206	206
query99	1317	1385	1310	1310
Total cold run time: 270098 ms
Total hot run time: 186337 ms

doris-robot · 2025-05-21T11:43:00Z

ClickBench: Total hot run time: 29.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86e4534269800373001a8e3cb7645355adcba1c7, data reload: false
query1	0.04	0.04	0.03
query2	0.13	0.11	0.12
query3	0.26	0.20	0.19
query4	1.61	0.20	0.11
query5	0.44	0.42	0.43
query6	1.14	0.66	0.66
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.58	0.52	0.52
query10	0.57	0.58	0.58
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.61	0.61	0.60
query14	0.79	0.80	0.82
query15	0.88	0.85	0.85
query16	0.37	0.37	0.39
query17	1.04	1.02	1.02
query18	0.22	0.22	0.21
query19	2.06	1.79	1.84
query20	0.01	0.01	0.02
query21	15.41	0.88	0.54
query22	0.77	1.20	0.96
query23	14.71	1.38	0.62
query24	7.13	1.53	1.26
query25	0.46	0.15	0.08
query26	0.54	0.17	0.13
query27	0.05	0.06	0.05
query28	10.12	0.90	0.44
query29	12.57	4.09	3.38
query30	0.25	0.09	0.06
query31	2.82	0.57	0.39
query32	3.23	0.57	0.48
query33	3.04	3.07	3.07
query34	15.68	5.06	4.48
query35	4.51	4.48	4.49
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.58 s
Total hot run time: 29.76 s

Copilot

Pull Request Overview

This PR removes the dependency on the old optimizer for row policy processing and renames the Expr types to Expression. Key changes include:

Removing CreatePolicyStmt-based logic and replacing it with NereidsParser and CreatePolicyCommand.
Updating row policy classes to use the new Expression type instead of Expr.
Removing legacy row policy rewriting and merging logic from various components.

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java	Replaced CreatePolicyStmt usage with NereidsParser-based CreatePolicyCommand.
fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java	Updated Expression type usage for row policy predicates.
fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java	Transitioned row policy creation logic to use NereidsParser.
fe/fe-core/src/main/java/org/apache/doris/qe/DdlExecutor.java	Removed legacy CreatePolicyStmt handling.
fe/fe-core/src/main/java/org/apache/doris/policy/RowPolicy.java	Modified to use Expression type and updated getter logic for wherePredicate.
fe/fe-core/src/main/java/org/apache/doris/policy/PolicyMgr.java	Removed legacy row policy merging code.
fe/fe-core/src/main/java/org/apache/doris/policy/Policy.java	Removed dependency on CreatePolicyStmt for policy creation.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CreatePolicyCommand.java	Updated to no longer translate legacy expressions.
fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java	Eliminated row policy rewriting logic for the old planner.
fe/fe-core/src/main/java/org/apache/doris/analysis/CreatePolicyStmt.java	Removed legacy properties and analysis logic; toSql now returns an empty string.

zddr · 2025-05-22T03:32:19Z

run buildall

doris-robot · 2025-05-22T04:08:59Z

TPC-H: Total hot run time: 34515 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17eaca8fb2680eef31f1d11414b7179c635f805e, data reload: false
------ Round 1 ----------------------------------
q1	26179	5125	5049	5049
q2	2082	281	190	190
q3	10382	1240	711	711
q4	10231	987	542	542
q5	7512	2425	2329	2329
q6	187	168	138	138
q7	933	745	628	628
q8	9328	1314	1123	1123
q9	6787	5019	5061	5019
q10	6871	2331	1919	1919
q11	488	297	276	276
q12	358	361	219	219
q13	17760	3677	3087	3087
q14	240	228	207	207
q15	529	487	491	487
q16	419	433	379	379
q17	613	860	382	382
q18	7478	7300	6973	6973
q19	1335	963	553	553
q20	345	328	236	236
q21	3998	3304	3059	3059
q22	1083	1034	1009	1009
Total cold run time: 115138 ms
Total hot run time: 34515 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5167	5120	5118	5118
q2	239	323	228	228
q3	2159	2736	2323	2323
q4	1361	1838	1413	1413
q5	4451	4408	4385	4385
q6	225	170	135	135
q7	2048	1946	1779	1779
q8	2608	2870	2527	2527
q9	7168	7120	7201	7120
q10	3061	3229	2788	2788
q11	574	500	505	500
q12	706	776	647	647
q13	3496	3902	3338	3338
q14	271	305	267	267
q15	539	480	471	471
q16	444	489	466	466
q17	1198	1573	1346	1346
q18	7640	7451	7373	7373
q19	869	929	1171	929
q20	1905	2004	1895	1895
q21	4943	4516	4500	4500
q22	1074	1038	1019	1019
Total cold run time: 52146 ms
Total hot run time: 50567 ms

doris-robot · 2025-05-22T04:21:11Z

TPC-DS: Total hot run time: 192831 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17eaca8fb2680eef31f1d11414b7179c635f805e, data reload: false
query1	1354	1097	1060	1060
query2	6111	1821	1814	1814
query3	10985	4501	4528	4501
query4	53204	25399	23253	23253
query5	4909	539	451	451
query6	322	203	200	200
query7	4881	513	290	290
query8	316	273	236	236
query9	5451	2642	2637	2637
query10	401	352	277	277
query11	14995	14966	14827	14827
query12	163	114	106	106
query13	1039	529	411	411
query14	10102	6365	6358	6358
query15	209	207	184	184
query16	7062	677	510	510
query17	1092	771	612	612
query18	1559	420	326	326
query19	200	203	178	178
query20	128	125	121	121
query21	212	136	107	107
query22	4379	4458	4257	4257
query23	34372	33494	33616	33494
query24	6722	2413	2433	2413
query25	475	469	424	424
query26	722	291	160	160
query27	2257	512	342	342
query28	3168	2174	2164	2164
query29	567	567	454	454
query30	277	221	188	188
query31	879	850	768	768
query32	76	65	67	65
query33	449	370	318	318
query34	774	876	536	536
query35	809	853	735	735
query36	948	1014	909	909
query37	125	109	80	80
query38	4297	4227	4278	4227
query39	1491	1455	1449	1449
query40	228	126	113	113
query41	56	53	53	53
query42	130	114	107	107
query43	508	518	464	464
query44	1323	830	835	830
query45	180	180	167	167
query46	846	1035	643	643
query47	1868	1910	1817	1817
query48	393	450	322	322
query49	682	508	438	438
query50	706	706	417	417
query51	4203	4241	4194	4194
query52	120	110	96	96
query53	229	266	193	193
query54	609	604	513	513
query55	87	83	81	81
query56	310	302	305	302
query57	1198	1224	1176	1176
query58	272	258	254	254
query59	2699	2851	2716	2716
query60	340	350	339	339
query61	131	132	132	132
query62	770	773	664	664
query63	234	195	192	192
query64	1933	1040	706	706
query65	4321	4195	4253	4195
query66	738	409	310	310
query67	15840	15681	15551	15551
query68	7433	882	533	533
query69	550	306	275	275
query70	1211	1209	1143	1143
query71	503	311	296	296
query72	5462	4560	4978	4560
query73	1467	616	350	350
query74	9013	9119	9041	9041
query75	3889	3156	2706	2706
query76	4201	1190	754	754
query77	623	380	303	303
query78	10236	10121	9310	9310
query79	2670	831	569	569
query80	650	509	457	457
query81	491	256	218	218
query82	483	130	103	103
query83	362	253	232	232
query84	304	104	90	90
query85	794	363	312	312
query86	384	310	293	293
query87	4359	4410	4346	4346
query88	3467	2248	2240	2240
query89	402	314	286	286
query90	1811	203	206	203
query91	145	146	111	111
query92	80	59	58	58
query93	1950	953	584	584
query94	660	403	299	299
query95	375	299	285	285
query96	505	569	281	281
query97	2746	2790	2674	2674
query98	228	200	203	200
query99	1414	1407	1292	1292
Total cold run time: 297304 ms
Total hot run time: 192831 ms

doris-robot · 2025-05-22T04:26:29Z

ClickBench: Total hot run time: 28.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17eaca8fb2680eef31f1d11414b7179c635f805e, data reload: false
query1	0.04	0.03	0.03
query2	0.13	0.10	0.11
query3	0.26	0.19	0.19
query4	1.59	0.18	0.19
query5	0.46	0.43	0.44
query6	1.49	0.66	0.67
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.52	0.51
query10	0.56	0.59	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.59	0.60
query14	0.79	0.82	0.79
query15	0.87	0.85	0.84
query16	0.39	0.38	0.38
query17	1.03	1.07	1.07
query18	0.23	0.21	0.20
query19	1.88	1.84	1.83
query20	0.02	0.01	0.01
query21	15.41	0.90	0.55
query22	0.76	1.27	0.70
query23	14.80	1.38	0.61
query24	7.17	2.02	0.35
query25	0.31	0.13	0.08
query26	0.59	0.17	0.13
query27	0.05	0.04	0.05
query28	9.12	0.93	0.45
query29	12.53	4.00	3.27
query30	0.25	0.10	0.07
query31	2.81	0.58	0.39
query32	3.23	0.54	0.48
query33	3.09	3.04	3.09
query34	15.78	5.06	4.51
query35	4.47	4.54	4.52
query36	0.66	0.50	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 102.83 s
Total hot run time: 28.64 s

zddr · 2025-05-22T07:43:26Z

run cloud_p0

github-actions · 2025-05-22T10:37:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-22T10:37:04Z

PR approved by anyone and no changes requested.

…pache#51122) - Change "Expr" to "Expression" in RowPolicy - Remove the logic of rewriting using row policy in the old optimizer - Remove the logic for creating row policies based on CreatePolicyStmt

1
86e4534

zddr marked this pull request as draft May 21, 2025 10:00

zddr changed the title ~~[draft]policy~~[enhance](nereids)row policy removes dependency on the old optimizerMay 22, 2025

zddr marked this pull request as ready for review May 22, 2025 02:52

morrySnow requested a review from Copilot May 22, 2025 03:11

CopilotAI reviewed May 22, 2025
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/CreatePolicyStmt.java Outdated

1
17eaca8

starocean999 approved these changes May 22, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 22, 2025

github-actionsBot added the reviewed label May 22, 2025

morrySnow approved these changes May 22, 2025
View reviewed changes

morrySnow merged commit 3970183 into apache:masterMay 22, 2025

zddr changed the title ~~[enhance](nereids)row policy removes dependency on the old optimizer~~[clean](planner)row policy removes dependency on the old optimizerMay 22, 2025

ngbinh mentioned this pull request Apr 23, 2026
[Bug] CREATE ROW POLICY with correlated EXISTS subquery silently succeeds but policy is never enforced #62729
Open

Uh oh!

Conversation

zddr commented May 21, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented May 21, 2025

Uh oh!

zddr commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

zddr commented May 22, 2025

Uh oh!

doris-robot commented May 22, 2025

Uh oh!

doris-robot commented May 22, 2025

Uh oh!

doris-robot commented May 22, 2025

Uh oh!

zddr commented May 22, 2025

Uh oh!

github-actionsBot commented May 22, 2025

Uh oh!

github-actionsBot commented May 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zddr commented May 21, 2025•
edited
Loading