[fix](fd)drop Function dependencies from join outer side by englefly · Pull Request #65982 · apache/doris

englefly · 2026-07-24T03:07:05Z

What problem does this PR solve?

Issue Number: N/A (no issue linked)

Related PR: N/A

Problem Summary:

Nereids derives functional dependencies (FDs) from each operator's children via DataTrait, and rewrite rules such as EliminateGroupByKey, EliminateGroupByKeyByUniform, EliminateOrderByKey and ConstantPropagation consume these FDs to drop functionally-determined grouping/ordering keys. If an FD is derived incorrectly, those rules may produce wrong query results.

LogicalJoin.computeFd() and PhysicalHashJoin.computeFd() previously propagated FDs from both join inputs, only excluding the semi/anti-join side:

if (!joinType.isLeftSemiOrAntiJoin()) {
builder.addFuncDepsDG(right().getLogicalProperties().getTrait());
}
if (!joinType.isRightSemiOrAntiJoin()) {
builder.addFuncDepsDG(left().getLogicalProperties().getTrait());
}

For outer joins the nullable side is null-extended: unmatched rows are padded with NULLs, which invalidates FDs from that side. For example, in t1 LEFT OUTER JOIN t2, if the right side has the FD t2.a -> t2.b and a is nullable, a matched row with a = NULL, b = 1 and an unmatched row (a = NULL, b = NULL) together violate a -> b on the join output. The old code still propagated such FDs from the nullable side for LEFT OUTER JOIN (right side), RIGHT OUTER JOIN (left side) and FULL OUTER JOIN (both sides), so a downstream rule could remove a group-by key that is not actually functionally determined and change the query result.

This PR fixes the FD derivation on join outputs:

computeFd() in LogicalJoin and PhysicalHashJoin is rewritten with an explicit switch over join types:
- inner / cross joins: propagate FDs from both sides;
- semi / anti joins: propagate FDs only from the output side;
- outer joins: propagate FDs from the preserved side, and from the nullable side only the FDs whose determinant is NOT NULL in the child — matched rows then always carry a non-null determinant, so they cannot collide with the (NULL, NULL) null-extension of unmatched rows;
- full outer join: keep only the NOT-NULL-determinant FDs from both sides.
A new DataTrait.Builder.addFuncDepsDGForOuterJoinNullableSide() / FuncDepsDG.Builder.addDepsForOuterJoinNullableSide() implements the NOT-NULL-determinant filter.
The nullability check is performed against the current child output rather than the slot stored in the FD graph: slots are keyed by ExprId and may carry a stale nullable flag (e.g. after LogicalSubQueryAliasToLogicalProject inlining), so a determinant that became nullable in the immediate child is dropped.

Tests in FdTest are updated (FOJ/LOJ/ROJ no longer propagate nullable-side FDs, while NOT-NULL-determinant FDs from the nullable side are kept), and a new testNestedOuterJoinNullableDeterminant covers the nested outer-join case where the determinant's stale non-nullable flag must not leak through, verified on both the logical and the physical join paths.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-07-24T03:07:16Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

englefly · 2026-07-24T03:07:19Z

run buildall

englefly · 2026-07-24T03:07:24Z

/review

github-actions

Request changes: one P2 optimizer performance regression remains.

Critical checkpoint conclusions:

Goal and correctness: the patch prevents unsafe direct functional dependencies from surviving a null-extended outer-join side, and no unsafe edge is reconstructed through unique, uniform, equality, transitive, or pruning paths. However, it also drops still-valid dependencies whose determinant contains an originally non-null child slot; see the inline finding.
Scope and parallel paths: all 14 current join types, ASOF directionality, semi/anti/NAAJ and mark shapes were checked. The logical and physical hash-join switches agree; nested-loop recomputation blocks FD propagation. No additional path-specific issue was found.
Tests: the changed FdTest assertions inspect only the logical plan and use NOT NULL determinants, so they codify the safe dependency loss rather than reproduce the nullable-determinant collision. The physical reset/recomputation path is not covered.
Performance: the accepted issue prevents EliminateGroupByKey and EliminateOrderByKey from applying valid reductions. No other CPU, memory, or hot-path concern was found.
Concurrency, lifecycle, configuration, compatibility, persistence/transactions, data writes, FE-BE propagation, and observability: not involved in this change.
User focus: no additional focus was provided; the complete PR was reviewed.

Validation was static only. No build or test was run because the governing review prompt prohibits builds in this environment.

github-actions

Corrected inline anchor for the accepted P2 finding. The complete checkpoint summary is in review 4769895466.

hello-stephen · 2026-07-24T07:43:59Z

TPC-H: Total hot run time: 23165 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22ab5594d21ff2b49579e97bf01db0fb3e4e3b1b, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17740	3517	3514	3514
q2	2028	287	185	185
q3	10458	1183	764	764
q4	4673	312	225	225
q5	7606	732	475	475
q6	181	140	109	109
q7	723	666	491	491
q8	10172	1480	1528	1480
q9	4626	3046	3042	3042
q10	6728	1175	975	975
q11	521	326	306	306
q12	739	519	424	424
q13	17934	2314	1854	1854
q14	191	183	169	169
q15	q16	549	504	457	457
q17	972	1017	1013	1013
q18	4640	3602	3488	3488
q19	2453	1287	1081	1081
q20	790	680	539	539
q21	5867	2436	2287	2287
q22	405	353	287	287
Total cold run time: 99996 ms
Total hot run time: 23165 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3953	3793	3772	3772
q2	246	266	188	188
q3	2702	2728	2518	2518
q4	1338	1342	1049	1049
q5	2650	2515	2500	2500
q6	211	154	105	105
q7	1299	1192	1066	1066
q8	1990	1720	1695	1695
q9	3680	3617	3631	3617
q10	2441	2385	2132	2132
q11	410	339	314	314
q12	583	539	401	401
q13	2251	2586	1987	1987
q14	237	214	206	206
q15	q16	500	517	477	477
q17	1228	1210	1179	1179
q18	5296	4864	4702	4702
q19	1181	1182	1174	1174
q20	1111	1122	954	954
q21	4278	3786	3708	3708
q22	510	447	427	427
Total cold run time: 38095 ms
Total hot run time: 34171 ms

hello-stephen · 2026-07-24T07:52:11Z

TPC-DS: Total hot run time: 117085 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22ab5594d21ff2b49579e97bf01db0fb3e4e3b1b, data reload: false
query5	4343	563	415	415
query6	485	227	205	205
query7	4834	541	315	315
query8	321	172	159	159
query9	8778	3923	3842	3842
query10	476	326	327	326
query11	5570	1353	1191	1191
query12	157	98	96	96
query13	1255	556	448	448
query14	6068	2820	2600	2600
query14_1	2363	2342	2339	2339
query15	200	156	140	140
query16	1056	543	507	507
query17	970	626	487	487
query18	2408	442	325	325
query19	215	189	158	158
query20	103	92	95	92
query21	227	147	130	130
query22	7433	7671	7278	7278
query23	9535	9095	8879	8879
query23_1	8931	8986	8952	8952
query24	7256	1397	1004	1004
query24_1	1007	1046	985	985
query25	560	420	374	374
query26	1289	357	204	204
query27	2659	593	365	365
query28	4385	2155	2135	2135
query29	1068	592	487	487
query30	306	235	199	199
query31	906	654	698	654
query32	109	66	76	66
query33	597	303	260	260
query34	1163	1156	669	669
query35	539	574	452	452
query36	1035	1062	1044	1044
query37	164	104	90	90
query38	1273	1027	985	985
query39	633	610	612	610
query39_1	588	584	591	584
query40	290	162	149	149
query41	67	66	65	65
query42	92	91	93	91
query43	292	300	256	256
query44	1326	793	784	784
query45	130	132	123	123
query46	1048	1135	677	677
query47	1471	1390	1349	1349
query48	431	414	291	291
query49	574	316	244	244
query50	1033	421	323	323
query51	10916	10975	11466	10975
query52	85	86	75	75
query53	234	268	201	201
query54	274	233	205	205
query55	73	70	66	66
query56	328	283	282	282
query57	898	863	764	764
query58	283	254	243	243
query59	1294	1332	1260	1260
query60	296	251	241	241
query61	153	142	147	142
query62	399	290	275	275
query63	235	191	188	188
query64	2756	908	726	726
query65	2124	2058	2041	2041
query66	1835	400	321	321
query67	16205	15964	15959	15959
query68	3141	1453	925	925
query69	408	301	276	276
query70	1132	1019	1021	1019
query71	406	335	327	327
query72	2908	2517	2295	2295
query73	804	786	469	469
query74	1696	1521	1414	1414
query75	1567	1470	1310	1310
query76	2349	978	733	733
query77	320	335	254	254
query78	6294	5742	5465	5465
query79	2688	1054	774	774
query80	2089	515	474	474
query81	497	255	234	234
query82	612	141	109	109
query83	368	321	293	293
query84	322	159	129	129
query85	959	575	528	528
query86	430	302	301	301
query87	1145	1109	1034	1034
query88	3783	2919	2924	2919
query89	404	302	269	269
query90	1925	187	187	187
query91	205	182	168	168
query92	61	64	57	57
query93	1771	1498	963	963
query94	748	357	343	343
query95	742	563	438	438
query96	970	819	367	367
query97	1357	1309	1249	1249
query98	210	176	169	169
query99	602	560	508	508
Total cold run time: 208877 ms
Total hot run time: 117085 ms

hello-stephen · 2026-07-24T07:56:47Z

ClickBench: Total hot run time: 18.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22ab5594d21ff2b49579e97bf01db0fb3e4e3b1b, data reload: false
query1	0.00	0.00	0.00
query2	0.09	0.04	0.04
query3	0.25	0.13	0.12
query4	1.60	0.12	0.12
query5	0.18	0.17	0.17
query6	1.19	0.98	0.99
query7	0.04	0.01	0.00
query8	0.05	0.03	0.04
query9	0.32	0.25	0.26
query10	0.43	0.45	0.43
query11	0.19	0.11	0.13
query12	0.16	0.12	0.12
query13	0.37	0.39	0.39
query14	0.59	0.61	0.61
query15	0.47	0.44	0.44
query16	0.26	0.25	0.25
query17	0.93	0.92	0.93
query18	0.24	0.22	0.23
query19	1.51	1.52	1.47
query20	0.01	0.02	0.01
query21	15.43	0.23	0.16
query22	4.85	0.05	0.05
query23	16.14	0.31	0.13
query24	2.94	0.44	0.30
query25	0.12	0.05	0.04
query26	0.73	0.19	0.15
query27	0.05	0.03	0.03
query28	3.54	0.85	0.46
query29	12.60	4.29	3.42
query30	0.27	0.14	0.16
query31	2.76	0.49	0.21
query32	3.35	0.38	0.29
query33	1.71	1.73	1.69
query34	15.30	2.78	2.18
query35	2.13	2.11	2.09
query36	0.49	0.34	0.33
query37	0.07	0.06	0.05
query38	0.05	0.03	0.04
query39	0.03	0.03	0.03
query40	0.15	0.12	0.10
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 91.75 s
Total hot run time: 18.65 s

github-actions · 2026-07-27T04:14:33Z

PR approved by at least one committer and no changes requested.

englefly · 2026-07-29T03:00:07Z

run buildall

englefly · 2026-07-29T03:00:14Z

/review

github-actions

Request changes: one P1 correctness issue and two distinct P2 FD-preservation gaps remain. The existing thread also remains relevant; no duplicate inline was added for its composite-determinant and test-coverage points.

Critical checkpoint conclusions:

Goal and correctness: the PR aims to retain exactly the functional dependencies that survive outer-join null extension. MAIN-F1 shows the implementation can retain an invalid historical-slot FD and let EliminateGroupByKey merge distinct groups. MAIN-F2 and MAIN-F3 show it also drops safe transitive and null-rejected-key dependencies.
Scope and clarity: the five-file change is focused. Both logical and physical hash-join switches cover all 14 current join types with matching side propagation; no unrelated source change was found.
Concurrency and lifecycle: no runtime concurrency, locks, shared mutable state, or static initialization are involved. The relevant optimizer-property lifecycle was traced through lazy logical traits and bottom-up physical recomputation; historical Slot nullability in that lifecycle is the P1 issue.
Parallel and special paths: ordinary/right/full/ASOF outer joins, semi/anti/NAAJ, mark joins, implementation conversion, output pruning, and the FD-blocking nested-loop path were checked. MAIN-F3 is intentionally limited to a proven null-rejecting key on the non-preserved side of a one-sided join; full outer and null-safe/unproven predicates are excluded.
Tests: the positive logical assertions cover direct schema-non-null determinants. The new nullable-key negative assertions are vacuous because that scan never seeds the claimed FD, and no added test reaches physical recomputation; those points are already covered by the existing thread and were not duplicated inline.
Configuration, compatibility, persistence, transactions, data writes, FE-BE propagation, error boundaries, and observability: not involved.
Performance: graph copying remains linear, but MAIN-F2 and MAIN-F3 prevent valid group/order-key reductions.
User focus: no additional review focus was provided; the whole PR was reviewed.
Completion: three bounded rounds converged, with every final-round full and risk reviewer returning NO_NEW_VALUABLE_FINDINGS against this exact comment set.

Validation was static only. No build or test was run because the governing review prompt prohibits builds in this environment.

hello-stephen · 2026-07-29T04:37:00Z

TPC-H: Total hot run time: 29678 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e9755acbb4e949412286ecaf04b1387d5e69ee5, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17838	4366	4214	4214
q2	2010	340	217	217
q3	10302	1487	840	840
q4	4693	479	341	341
q5	7544	956	559	559
q6	184	171	135	135
q7	772	839	618	618
q8	9345	1538	1571	1538
q9	5578	4346	4303	4303
q10	6737	1751	1463	1463
q11	495	352	338	338
q12	742	574	450	450
q13	18088	3358	2748	2748
q14	266	263	238	238
q15	q16	777	773	707	707
q17	1043	897	1009	897
q18	7047	5757	5659	5659
q19	1303	1283	1052	1052
q20	801	671	621	621
q21	6099	2559	2440	2440
q22	440	357	300	300
Total cold run time: 102104 ms
Total hot run time: 29678 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4417	4364	4351	4351
q2	293	318	216	216
q3	4600	4982	4404	4404
q4	2066	2124	1347	1347
q5	4403	4300	4259	4259
q6	233	177	128	128
q7	1800	2224	1811	1811
q8	2588	2203	2225	2203
q9	8008	8048	7832	7832
q10	4692	4612	4188	4188
q11	564	413	397	397
q12	744	767	548	548
q13	3357	3642	2943	2943
q14	289	294	263	263
q15	q16	722	751	658	658
q17	1376	1348	1321	1321
q18	8046	7360	7427	7360
q19	1151	1156	1123	1123
q20	2204	2199	1948	1948
q21	5254	4662	4489	4489
q22	533	477	404	404
Total cold run time: 57340 ms
Total hot run time: 52193 ms

hello-stephen · 2026-07-29T04:41:59Z

FE UT Coverage Report

Increment line coverage 80.00% (36/45) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-07-29T04:48:11Z

TPC-DS: Total hot run time: 177247 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e9755acbb4e949412286ecaf04b1387d5e69ee5, data reload: false
query5	4355	624	477	477
query6	459	218	203	203
query7	4857	554	333	333
query8	347	181	168	168
query9	8799	4052	4033	4033
query10	479	340	297	297
query11	5840	2345	2129	2129
query12	163	110	98	98
query13	1273	590	467	467
query14	6206	5192	4828	4828
query14_1	4232	4210	4166	4166
query15	215	209	178	178
query16	1007	483	435	435
query17	912	673	549	549
query18	2433	464	330	330
query19	196	184	145	145
query20	108	107	103	103
query21	233	155	134	134
query22	13606	13538	13290	13290
query23	17453	16473	16079	16079
query23_1	16200	16188	16146	16146
query24	7531	1759	1295	1295
query24_1	1262	1299	1317	1299
query25	583	469	381	381
query26	1342	353	212	212
query27	2617	594	388	388
query28	4472	1968	1960	1960
query29	1063	627	502	502
query30	336	270	242	242
query31	1116	1094	981	981
query32	100	66	62	62
query33	534	325	266	266
query34	1157	1131	658	658
query35	767	807	660	660
query36	1048	1039	881	881
query37	157	107	98	98
query38	1899	1693	1664	1664
query39	889	870	855	855
query39_1	841	846	835	835
query40	258	169	145	145
query41	72	68	68	68
query42	97	92	90	90
query43	322	335	281	281
query44	1397	774	757	757
query45	202	191	178	178
query46	1019	1151	700	700
query47	2140	2117	2010	2010
query48	426	417	307	307
query49	601	432	322	322
query50	1041	443	332	332
query51	10748	10662	10458	10458
query52	88	89	79	79
query53	257	279	204	204
query54	302	251	238	238
query55	76	73	68	68
query56	316	300	307	300
query57	1292	1302	1223	1223
query58	320	304	275	275
query59	1576	1621	1425	1425
query60	321	283	263	263
query61	180	181	177	177
query62	584	497	431	431
query63	238	193	202	193
query64	2815	1056	886	886
query65	4682	4572	4614	4572
query66	1816	504	373	373
query67	29248	29201	28985	28985
query68	3348	1448	960	960
query69	416	297	259	259
query70	904	809	826	809
query71	347	319	327	319
query72	2988	2638	2370	2370
query73	853	821	416	416
query74	5082	4904	4725	4725
query75	2537	2485	2157	2157
query76	2356	1186	788	788
query77	344	372	281	281
query78	11815	11897	11291	11291
query79	1409	1127	780	780
query80	1315	531	464	464
query81	572	335	299	299
query82	611	154	120	120
query83	405	318	298	298
query84	323	162	134	134
query85	955	598	529	529
query86	405	257	216	216
query87	1818	1820	1760	1760
query88	3690	2794	2757	2757
query89	431	382	335	335
query90	1918	189	189	189
query91	194	190	161	161
query92	62	59	56	56
query93	1682	1560	1027	1027
query94	727	336	323	323
query95	825	524	481	481
query96	1044	806	344	344
query97	2625	2616	2494	2494
query98	220	211	200	200
query99	1088	1104	960	960
Total cold run time: 263212 ms
Total hot run time: 177247 ms

hello-stephen · 2026-07-29T04:53:11Z

ClickBench: Total hot run time: 24.96 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e9755acbb4e949412286ecaf04b1387d5e69ee5, data reload: false
query1	0.01	0.01	0.00
query2	0.13	0.05	0.05
query3	0.25	0.14	0.14
query4	1.61	0.14	0.13
query5	0.24	0.22	0.22
query6	1.19	1.10	1.07
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.37	0.31	0.30
query10	0.54	0.55	0.53
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.47	0.47	0.46
query14	1.05	0.99	1.00
query15	0.62	0.58	0.59
query16	0.32	0.32	0.32
query17	1.10	1.10	1.10
query18	0.23	0.21	0.21
query19	2.08	1.94	1.91
query20	0.02	0.01	0.01
query21	15.43	0.23	0.13
query22	4.86	0.05	0.06
query23	16.12	0.31	0.12
query24	3.07	0.43	0.31
query25	0.11	0.05	0.04
query26	0.75	0.20	0.16
query27	0.04	0.03	0.03
query28	3.48	0.92	0.53
query29	12.51	4.18	3.29
query30	0.28	0.17	0.15
query31	2.76	0.60	0.31
query32	3.22	0.59	0.49
query33	3.24	3.25	3.26
query34	15.66	4.22	3.49
query35	3.52	3.48	3.52
query36	0.55	0.44	0.42
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.82 s
Total hot run time: 24.96 s

englefly · 2026-08-04T11:15:59Z

run buildall

hello-stephen · 2026-08-04T13:24:17Z

FE UT Coverage Report

Increment line coverage 81.54% (53/65) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-04T13:41:53Z

TPC-H: Total hot run time: 28424 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79d097ac680dbdaa753c656d5f195a65a3bfb703, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17650	3912	3907	3907
q2	2045	314	199	199
q3	10256	1399	799	799
q4	4687	467	349	349
q5	7609	844	542	542
q6	179	165	132	132
q7	726	800	590	590
q8	9746	1453	1577	1453
q9	5980	4045	4001	4001
q10	6829	1615	1338	1338
q11	500	339	343	339
q12	755	561	459	459
q13	18097	3221	2701	2701
q14	258	261	253	253
q15	q16	730	716	661	661
q17	1014	954	981	954
q18	6468	5600	5569	5569
q19	1301	1217	1022	1022
q20	789	693	603	603
q21	5973	2600	2261	2261
q22	428	350	292	292
Total cold run time: 102020 ms
Total hot run time: 28424 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4209	4146	4152	4146
q2	276	313	202	202
q3	4506	4898	4361	4361
q4	2161	2285	1422	1422
q5	4205	4094	4059	4059
q6	226	172	126	126
q7	1680	1582	1419	1419
q8	2708	2081	2032	2032
q9	7241	7160	7208	7160
q10	4240	4272	3884	3884
q11	553	399	378	378
q12	722	712	500	500
q13	3136	3432	3035	3035
q14	297	312	278	278
q15	q16	733	717	637	637
q17	1282	1284	1254	1254
q18	12165	10973	11907	10973
q19	1165	1138	1071	1071
q20	2250	2235	1972	1972
q21	5644	4945	4705	4705
q22	524	466	416	416
Total cold run time: 59923 ms
Total hot run time: 54030 ms

hello-stephen · 2026-08-04T13:52:33Z

TPC-DS: Total hot run time: 166787 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79d097ac680dbdaa753c656d5f195a65a3bfb703, data reload: false
query5	4326	587	479	479
query6	490	214	214	214
query7	4932	588	348	348
query8	321	167	147	147
query9	8782	4067	4062	4062
query10	457	370	309	309
query11	5839	2189	2033	2033
query12	148	124	94	94
query13	1238	633	443	443
query14	6031	4413	4127	4127
query14_1	3970	3904	3933	3904
query15	202	192	173	173
query16	990	512	439	439
query17	906	692	529	529
query18	2455	458	334	334
query19	207	186	149	149
query20	102	100	101	100
query21	229	156	139	139
query22	13128	12994	12759	12759
query23	15853	15183	14591	14591
query23_1	14786	14651	14771	14651
query24	7664	1682	1247	1247
query24_1	1238	1250	1244	1244
query25	553	462	391	391
query26	1336	362	210	210
query27	2583	576	417	417
query28	4580	2039	2053	2039
query29	1106	650	513	513
query30	350	265	224	224
query31	1175	1117	1047	1047
query32	117	64	62	62
query33	532	322	253	253
query34	1222	1149	677	677
query35	749	761	637	637
query36	767	797	691	691
query37	164	107	97	97
query38	1843	1796	1696	1696
query39	826	835	787	787
query39_1	787	786	792	786
query40	269	168	147	147
query41	78	71	71	71
query42	99	99	100	99
query43	325	325	289	289
query44	1493	808	781	781
query45	195	175	179	175
query46	1087	1185	719	719
query47	1647	1679	1489	1489
query48	416	442	321	321
query49	656	417	296	296
query50	1175	451	348	348
query51	10464	10326	10519	10326
query52	88	91	77	77
query53	265	297	217	217
query54	329	251	283	251
query55	75	71	67	67
query56	297	300	285	285
query57	1022	1004	956	956
query58	281	257	259	257
query59	1510	1679	1370	1370
query60	314	277	258	258
query61	158	152	155	152
query62	405	324	271	271
query63	236	195	200	195
query64	2810	1048	847	847
query65	3867	3763	3829	3763
query66	1881	477	362	362
query67	28041	27987	27971	27971
query68	3137	1528	1020	1020
query69	409	305	268	268
query70	880	787	803	787
query71	369	343	322	322
query72	3016	2653	2312	2312
query73	839	835	431	431
query74	4630	4524	4282	4282
query75	2356	2319	1983	1983
query76	2326	1154	765	765
query77	330	371	280	280
query78	11068	10992	10590	10590
query79	1382	1163	716	716
query80	851	545	491	491
query81	491	337	283	283
query82	646	171	131	131
query83	411	339	302	302
query84	336	165	129	129
query85	974	639	540	540
query86	375	226	221	221
query87	1987	1964	1842	1842
query88	3803	2896	2803	2803
query89	397	324	281	281
query90	1822	196	197	196
query91	201	191	191	191
query92	62	59	54	54
query93	1561	1575	954	954
query94	605	367	294	294
query95	787	517	488	488
query96	1081	818	375	375
query97	2468	2480	2335	2335
query98	209	199	197	197
query99	735	728	614	614
Total cold run time: 253127 ms
Total hot run time: 166787 ms

hello-stephen · 2026-08-04T13:57:19Z

ClickBench: Total hot run time: 23.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79d097ac680dbdaa753c656d5f195a65a3bfb703, data reload: false
query1	0.00	0.00	0.00
query2	0.09	0.06	0.05
query3	0.25	0.13	0.14
query4	1.63	0.15	0.14
query5	0.25	0.23	0.22
query6	1.15	0.84	0.83
query7	0.03	0.01	0.01
query8	0.05	0.04	0.03
query9	0.38	0.30	0.32
query10	0.55	0.57	0.58
query11	0.18	0.12	0.13
query12	0.17	0.13	0.13
query13	0.45	0.46	0.46
query14	1.01	0.99	1.00
query15	0.60	0.58	0.58
query16	0.32	0.33	0.31
query17	1.09	1.10	1.06
query18	0.21	0.20	0.20
query19	2.06	1.96	1.98
query20	0.02	0.01	0.02
query21	15.42	0.21	0.13
query22	4.85	0.05	0.06
query23	16.14	0.32	0.12
query24	3.00	0.40	0.36
query25	0.11	0.05	0.05
query26	0.74	0.19	0.15
query27	0.05	0.03	0.03
query28	3.56	0.78	0.35
query29	12.51	3.95	3.16
query30	0.27	0.16	0.15
query31	2.77	0.55	0.31
query32	3.23	0.59	0.50
query33	3.12	3.18	3.13
query34	15.63	4.02	3.29
query35	3.23	3.24	3.19
query36	0.55	0.45	0.40
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.16	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 96.18 s
Total hot run time: 23.81 s

hello-stephen · 2026-08-04T15:10:45Z

FE Regression Coverage Report

Increment line coverage 87.69% (57/65) 🎉
Increment coverage report
Complete coverage report

englefly · 2026-08-06T07:03:50Z

run external

hello-stephen · 2026-08-06T09:16:55Z

FE Regression Coverage Report

Increment line coverage 6.79% (57/840) 🎉
Increment coverage report
Complete coverage report

diss

### What problem does this PR solve? Issue Number: N/A (no issue linked) Related PR: N/A Problem Summary: Nereids derives functional dependencies (FDs) from each operator's children via `DataTrait`, and rewrite rules such as `EliminateGroupByKey`, `EliminateGroupByKeyByUniform`, `EliminateOrderByKey` and `ConstantPropagation` consume these FDs to drop functionally-determined grouping/ordering keys. If an FD is derived incorrectly, those rules may produce wrong query results. `LogicalJoin.computeFd()` and `PhysicalHashJoin.computeFd()` previously propagated FDs from both join inputs, only excluding the semi/anti-join side: ```java if (!joinType.isLeftSemiOrAntiJoin()) { builder.addFuncDepsDG(right().getLogicalProperties().getTrait()); } if (!joinType.isRightSemiOrAntiJoin()) { builder.addFuncDepsDG(left().getLogicalProperties().getTrait()); } ``` For outer joins the nullable side is null-extended: unmatched rows are padded with NULLs, which invalidates FDs from that side. For example, in `t1 LEFT OUTER JOIN t2`, if the right side has the FD `t2.a -> t2.b` and `a` is nullable, a matched row with `a = NULL, b = 1` and an unmatched row `(a = NULL, b = NULL)` together violate `a -> b` on the join output. The old code still propagated such FDs from the nullable side for `LEFT OUTER JOIN` (right side), `RIGHT OUTER JOIN` (left side) and `FULL OUTER JOIN` (both sides), so a downstream rule could remove a group-by key that is not actually functionally determined and change the query result. This PR fixes the FD derivation on join outputs: 1. `computeFd()` in `LogicalJoin` and `PhysicalHashJoin` is rewritten with an explicit switch over join types: - inner / cross joins: propagate FDs from both sides; - semi / anti joins: propagate FDs only from the output side; - outer joins: propagate FDs from the preserved side, and from the nullable side only the FDs whose determinant is NOT NULL in the child — matched rows then always carry a non-null determinant, so they cannot collide with the `(NULL, NULL)` null-extension of unmatched rows; - full outer join: keep only the NOT-NULL-determinant FDs from both sides. 2. A new `DataTrait.Builder.addFuncDepsDGForOuterJoinNullableSide()` / `FuncDepsDG.Builder.addDepsForOuterJoinNullableSide()` implements the NOT-NULL-determinant filter. 3. The nullability check is performed against the *current* child output rather than the slot stored in the FD graph: slots are keyed by ExprId and may carry a stale `nullable` flag (e.g. after `LogicalSubQueryAliasToLogicalProject` inlining), so a determinant that became nullable in the immediate child is dropped. Tests in `FdTest` are updated (FOJ/LOJ/ROJ no longer propagate nullable-side FDs, while NOT-NULL-determinant FDs from the nullable side are kept), and a new `testNestedOuterJoinNullableDeterminant` covers the nested outer-join case where the determinant's stale non-nullable flag must not leak through, verified on both the logical and the physical join paths. ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…65982 (#66775) Cherry-picked from #65982 Co-authored-by: minghong <zhouminghong@selectdb.com>

…rapping (#64849) ### What problem does this PR solve? When a group-by key is functionally dependent on another key (e.g. s_suppkey -> s_name via PK) but required in output, remove it from GROUP BY and wrap with ANY_VALUE(). Previously EliminateGroupByKey kept such keys in GROUP BY to preserve SQL semantics. Now they are replaced with ANY_VALUE wrappers in the output, allowing the group-by set to be minimized while keeping the column in SELECT. Public findCanBeRemovedExpressions() API preserved for backward compatibility. Internal logic split into FindResult with separate removeExpression and wrapWithAnyValue sets. Test: testEliminateByPkWithOutputNeeded verifies ANY_VALUE wrapping when SELECT contains an FD-redundant group-by key. Issue Number: close #xxx Related PR: #65982 #66801 #66803 上面 3 个 pr 是原有 master 的bug fix. pick 这个 pr 前, 确保上面 3 个 pr 已经 pick Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

drop FD from join outer side
22ab559

englefly requested review from 924060929, morrySnow and starocean999 as code owners July 24, 2026 03:07

github-actionsBot requested changes Jul 24, 2026
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java

morrySnow added dev/4.0.x dev/4.1.x labels Jul 27, 2026

morrySnow previously approved these changes Jul 27, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 27, 2026

keep a->b from outer side, if a is not nullable
0e9755a

englefly dismissed morrySnow’s stale review via 0e9755aJuly 29, 2026 02:59

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jul 29, 2026

github-actionsBot previously requested changes Jul 29, 2026
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDepsDG.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDepsDG.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java Outdated

[P1] Check determinant nullability against the current child output
79d097a

feiniaofeiafei approved these changes Aug 7, 2026
View reviewed changes

starocean999 approved these changes Aug 14, 2026
View reviewed changes

englefly merged commit 99246b4 into apache:masterAug 14, 2026
32 of 33 checks passed

englefly deleted the fd-outer-join branch August 14, 2026 08:34

github-actionsBot mentioned this pull request Aug 14, 2026
branch-4.1: [fix](fd)drop Function dependencies from join outer side #65982 #66775
Merged

englefly mentioned this pull request Aug 15, 2026
[opt](aggregate) eliminate FD-redundant group-by keys via ANY_VALUE wrapping #64849
Merged
16 tasks

yiguolei pushed a commit that referenced this pull request Aug 15, 2026
branch-4.1: [fix](fd)drop Function dependencies from join outer side#…
daec798
…65982 (#66775) Cherry-picked from #65982 Co-authored-by: minghong <zhouminghong@selectdb.com>

yiguolei added dev/4.1.4-merged and removed dev/4.1.x labels Aug 15, 2026

englefly added the dev/4.2.x label Aug 16, 2026

morrySnow removed the dev/4.2.x label Aug 31, 2026

yiguolei mentioned this pull request Sep 1, 2026
4.1.4 Release Note #67355
Open

Uh oh!

Conversation

englefly commented Jul 24, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 24, 2026

Uh oh!

englefly commented Jul 24, 2026

Uh oh!

englefly commented Jul 24, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hello-stephen commented Jul 24, 2026

Uh oh!

hello-stephen commented Jul 24, 2026

Uh oh!

hello-stephen commented Jul 24, 2026

Uh oh!

github-actionsBot commented Jul 27, 2026

Uh oh!

englefly commented Jul 29, 2026

Uh oh!

englefly commented Jul 29, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

englefly commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

FE Regression Coverage Report

Uh oh!

englefly commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

englefly commented Jul 24, 2026•
edited
Loading