Skip to content

[improvement](cascades) Fuse project pushdown into join reorder - #67541

Draft
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:opt-cascades
Draft

[improvement](cascades) Fuse project pushdown into join reorder#67541
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:opt-cascades

Conversation

@morrySnow

@morrySnowmorrySnow commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

AddProjectForJoin inserts a LogicalProject above every Join before classic Cascades exploration. Most of these Projects contain only Slots, but the standalone PushDownProjectThroughSemiJoin and PushDownProjectThroughInnerOuterJoin exploration rules still expand child groups and assemble temporary Plans before their predicates reject the match. This creates substantial CPU and allocation overhead as the Memo grows.

This change:

  • extracts the existing Project normalization logic into a shared helper;
  • invokes that helper only when a Project-aware Join reorder rule is ready to produce an alternative;
  • covers inner associate/asscom/exchange, outer associate/asscom, and semi-join transpose paths;
  • removes the two standalone PushDown rule factories from classic OTHER_REORDER_RULES;
  • keeps the standalone rules and their AFTER_DPHYP_REORDER_RULES registration unchanged for DPHyp.

Local FE-only validation:

  • 24 targeted FE unit tests passed, including helper edge cases, all Project-aware reorder families, the existing standalone rule tests, and a complete AddProjectForJoin -> classic optimizer path.
  • TPC-H 22, TPC-DS 99, and one DPHyp smoke query produced identical optimizer mode, physical Plan fingerprint, root cost, and limit state between the registered baseline and candidate.
  • The candidate removed 162 redundant Memo expressions across six workload queries without changing the selected Plan.
  • Across 12 STAR/DENSE × SLOT/COMPLEX × 3/8/16-table JMH pairs, candidate mean planning time and allocation/op were lower in every configuration. The geometric baseline/candidate ratios were 1.066x/1.114x for Slot Projects and 1.698x/1.617x for Complex Projects.
  • In a 9-table JFR workload, candidate main-thread allocation fell 16.2%, matcher allocation 34.4%, Plan-assembly allocation 39.4%, and withGroupExprLogicalPropChildren allocation 40.7%.

All temporary instrumentation, JMH/JFR harnesses, and local analysis documents were removed from this commit.

Release note

Reduce Nereids classic Cascades planning overhead for queries with multiple joins.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes. Classic Cascades now normalizes Complex Projects while producing Join reorder alternatives instead of through standalone exploration rules. DPHyp behavior is unchanged.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@morrySnow
[improvement](fe) Fuse project pushdown into join reorder
ce87278
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: AddProjectForJoin inserts a Project above every Join before classic Cascades exploration. The standalone PushDownProjectThroughSemiJoin and PushDownProjectThroughInnerOuterJoin rules therefore expand child groups and assemble temporary plans for many slot-only Projects before rejecting them. Reuse a shared Project normalization helper from the Project-aware join reorder rules and remove the standalone rules from classic registration, while preserving their after-DPHyp registration. Local plan-only comparison kept the physical plan fingerprint and root cost unchanged for 122 TPC-H, TPC-DS, and DPHyp smoke queries. Across 12 JMH configurations, candidate mean planning time and allocation per operation were lower in every configuration; a 9-table JFR workload reduced matcher allocation by 34.4% and plan-assembly allocation by 39.4%.
### Release note
Reduce Nereids classic Cascades planning overhead for queries with multiple joins.
### Check List (For Author)
- Test: Unit Test
- 24 targeted FE unit tests passed
- FE Checkstyle passed
- Behavior changed: Yes. Classic Cascades performs complex Project normalization only while producing join reorder alternatives; after-DPHyp behavior is unchanged.
- Does this need documentation: No
@hello-stephen

hello-stephen commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnowmorrySnow changed the title [improvement](fe) Fuse project pushdown into join reorder[improvement](cascades) Fuse project pushdown into join reorderSep 4, 2026
@morrySnow

morrySnow commented Sep 4, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Sep 4, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16958 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce872786edfb8d6fa346a1bf17265666b5843e49, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17554	3108	3085	3085
q2	2084	258	216	216
q3	10244	852	513	513
q4	4674	248	212	212
q5	7672	589	388	388
q6	135	115	93	93
q7	515	487	381	381
q8	9228	868	974	868
q9	3411	2363	2346	2346
q10	6497	861	699	699
q11	391	202	183	183
q12	612	258	195	195
q13	18145	1537	1165	1165
q14	156	148	138	138
q15	q16	436	397	373	373
q17	1364	897	797	797
q18	3083	2241	2228	2228
q19	1264	882	802	802
q20	374	303	208	208
q21	5613	1838	1860	1838
q22	330	273	230	230
Total cold run time: 93782 ms
Total hot run time: 16958 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3501	3396	3405	3396
q2	504	400	378	378
q3	2225	2662	2215	2215
q4	1195	1189	896	896
q5	2226	2161	2122	2122
q6	164	118	88	88
q7	1004	914	853	853
q8	1611	1406	1418	1406
q9	3143	3122	3111	3111
q10	1877	1813	1630	1630
q11	366	272	252	252
q12	458	435	355	355
q13	1476	1570	1131	1131
q14	172	182	154	154
q15	q16	396	391	349	349
q17	3595	3260	3263	3260
q18	4778	4364	4751	4364
q19	926	886	865	865
q20	987	950	834	834
q21	3755	3082	3246	3082
q22	404	353	314	314
Total cold run time: 34763 ms
Total hot run time: 31055 ms

@hello-stephen

hello-stephen commented Sep 4, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81958 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce872786edfb8d6fa346a1bf17265666b5843e49, data reload: false
query5	4242	417	339	339
query6	394	134	142	134
query7	4929	416	231	231
query8	289	129	116	116
query9	8684	2929	2894	2894
query10	386	209	174	174
query11	5383	1068	915	915
query12	116	72	69	69
query13	1181	447	316	316
query14	6144	2199	2118	2118
query14_1	1994	1985	1984	1984
query15	173	117	109	109
query16	910	327	355	327
query17	814	441	354	354
query18	2321	332	238	238
query19	165	138	109	109
query20	73	68	67	67
query21	199	100	86	86
query22	5430	5289	5295	5289
query23	6664	6267	5826	5826
query23_1	5937	5965	5898	5898
query24	7281	1115	795	795
query24_1	784	778	782	778
query25	426	305	252	252
query26	1227	234	128	128
query27	2780	413	251	251
query28	4695	1554	1524	1524
query29	934	461	352	352
query30	265	153	133	133
query31	820	400	339	339
query32	127	75	77	75
query33	470	229	179	179
query34	976	803	478	478
query35	412	416	354	354
query36	595	552	529	529
query37	120	80	71	71
query38	1001	840	829	829
query39	497	491	469	469
query39_1	463	476	457	457
query40	209	91	80	80
query41	60	58	55	55
query42	75	78	75	75
query43	248	246	213	213
query44	1009	546	561	546
query45	111	109	110	109
query46	771	845	536	536
query47	750	762	715	715
query48	322	326	234	234
query49	551	241	222	222
query50	776	264	188	188
query51	8288	8515	8192	8192
query52	68	67	58	58
query53	203	200	147	147
query54	221	266	153	153
query55	72	62	56	56
query56	180	168	147	147
query57	728	700	656	656
query58	192	162	158	158
query59	1220	1246	1123	1123
query60	246	184	190	184
query61	120	116	108	108
query62	390	212	193	193
query63	170	135	137	135
query64	2727	689	688	688
query65	1592	1557	1585	1557
query66	1943	261	222	222
query67	9904	9754	9546	9546
query68	2757	1233	773	773
query69	341	232	195	195
query70	683	615	616	615
query71	252	179	172	172
query72	2238	1689	1483	1483
query73	643	566	336	336
query74	1566	1211	1132	1132
query75	1161	1101	954	954
query76	2283	723	554	554
query77	249	266	214	214
query78	3869	3722	3209	3209
query79	2394	874	598	598
query80	1569	327	282	282
query81	485	158	141	141
query82	643	122	100	100
query83	293	222	214	214
query84	309	111	90	90
query85	766	333	276	276
query86	395	178	169	169
query87	1020	980	900	900
query88	2773	2112	2106	2106
query89	287	189	176	176
query90	1977	126	126	126
query91	128	118	102	102
query92	81	73	66	66
query93	1472	1085	688	688
query94	623	248	223	223
query95	520	325	230	230
query96	773	571	274	274
query97	1032	1072	1025	1025
query98	151	143	131	131
query99	428	336	311	311
Total cold run time: 177026 ms
Total hot run time: 81958 ms

@hello-stephen

hello-stephen commented Sep 4, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce872786edfb8d6fa346a1bf17265666b5843e49, data reload: false
query1	0.01	0.01	0.01
query2	0.07	0.04	0.05
query3	0.24	0.10	0.11
query4	1.61	0.09	0.10
query5	0.18	0.16	0.16
query6	1.26	0.65	0.68
query7	0.03	0.01	0.01
query8	0.05	0.03	0.04
query9	0.28	0.22	0.21
query10	0.35	0.35	0.34
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.31	0.30	0.30
query14	0.44	0.45	0.46
query15	0.36	0.35	0.33
query16	0.21	0.24	0.23
query17	0.72	0.64	0.75
query18	0.17	0.15	0.17
query19	1.11	1.14	1.12
query20	0.02	0.01	0.01
query21	15.44	0.17	0.11
query22	5.06	0.04	0.05
query23	16.16	0.26	0.10
query24	2.99	0.31	0.25
query25	0.12	0.06	0.03
query26	0.82	0.16	0.12
query27	0.04	0.03	0.03
query28	3.68	0.56	0.29
query29	12.42	3.19	2.61
query30	0.25	0.12	0.12
query31	2.76	0.39	0.18
query32	3.59	0.32	0.24
query33	1.36	1.53	1.39
query34	15.37	2.18	1.76
query35	1.75	1.72	1.71
query36	0.47	0.30	0.27
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.11	0.08	0.08
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.42 s
Total hot run time: 14.59 s

@hello-stephen

hello-stephen commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 80.45% (144/179) 🎉
Increment coverage report
Complete coverage report

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929924060929Awaiting requested review from 924060929924060929 will be requested when the pull request is marked ready for review924060929 is a code owner
@engleflyengleflyAwaiting requested review from engleflyenglefly will be requested when the pull request is marked ready for reviewenglefly is a code owner
@starocean999starocean999Awaiting requested review from starocean999starocean999 will be requested when the pull request is marked ready for reviewstarocean999 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@morrySnow@hello-stephen