[improvement](fe) Infer set operator distinct by NDV

[feiniaofeiafei]

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

InferSetOperatorDistinct previously relied on a cost-based rewrite job to decide whether to add local distinct aggregates under set operations. This introduces extra optimizer cost.

This PR changes InferSetOperatorDistinct to run as a normal top-down rewrite rule and makes the rule decide whether to generate local distinct aggregates by child NDV/cardinality statistics. The NDV heuristic follows the existing logic in EagerAggRewriter#checkStats, but is kept local to InferSetOperatorDistinct because set-operation local distinct inference and eager aggregation pushdown have different optimization boundaries.

Each child of a DISTINCT set operation is judged independently, so one child can get a local distinct aggregate while another child can remain unchanged when its NDV indicates poor deduplication benefit.

Release note

None

Check List (For Author)

• Test: Unit Test
  • ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.InferSetOperatorDistinctTest
• Behavior changed: Yes. Optimizer rewrite behavior changes for DISTINCT set operations; SQL semantics are unchanged.
• Does this need documentation: No

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morrySnow]

Overall Assessment

The PR makes a good architectural improvement — removing the cost-based rewrite job in favor of stats-based NDV heuristics reduces optimizer overhead and enables per-child independent decisions. The core logic of using NDV statistics to decide whether to add local distinct aggregates is sound.

However, there are two significant concerns that need attention:

1. The regression test (infer_set_operator_distinct.groovy + .out) was deleted without replacement beyond a single-scenario unit test. The original test covered 20+ plan shapes (UNION/EXCEPT/INTERSECT, nested operations, joins, complex subqueries, hint combinations). The new unit test only validates ONE scenario. This is a major test coverage regression.

2. The use_INFER_SET_OPERATOR_DISTINCT / use_NO_INFER_SET_OPERATOR_DISTINCT hints are now silently ignored. Previously, users could force the rule on/off via these hints (the cost-based job checked them). Now, with the rule as a plain topDown, there is no mechanism to disable it.

See inline comments for detailed suggestions.

[morrySnow]

Test coverage concern: This single test only validates the per-child independent decision for an INTERSECT with one column per child. The deleted regression test (infer_set_operator_distinct.groovy) covered 20+ scenarios including:

• Different set operation types: UNION, EXCEPT, INTERSECT
• Nested set operations (e.g., INTERSECT of EXCEPT of UNION)
• Joins with set operations (INNER, LEFT, RIGHT, FULL OUTER, LEFT SEMI)
• Complex subqueries with filters
• Multi-column scenarios
• Hint-based control (use_INFER_SET_OPERATOR_DISTINCT, use_NO_INFER_SET_OPERATOR_DISTINCT, /*+ SET_VAR(...) */)

Suggestions:

1. Add at least a few more scenarios covering UNION and EXCEPT set operations
2. Add a test for the case where stats derivation falls back (stats is null on the plan)
3. Add a test for a multi-column group key scenario
4. Consider retaining a subset of the regression test for end-to-end validation

[feiniaofeiafei]

run buildall

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 82.09% (55/67) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

TPC-H: Total hot run time: 29319 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c06467b2dc0bd4a914293b3a0dae97c52b8dc4bb, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17726	4116	4166	4116
q2	2052	318	190	190
q3	10323	1494	830	830
q4	4674	473	335	335
q5	7524	877	578	578
q6	181	178	138	138
q7	786	862	633	633
q8	9339	1643	1682	1643
q9	5925	4543	4491	4491
q10	6770	1805	1533	1533
q11	448	279	244	244
q12	636	430	296	296
q13	18072	3458	2801	2801
q14	274	257	241	241
q15	q16	799	777	718	718
q17	1035	981	1005	981
q18	7021	5770	5492	5492
q19	1298	1307	1095	1095
q20	520	424	266	266
q21	5936	2656	2400	2400
q22	440	352	298	298
Total cold run time: 101779 ms
Total hot run time: 29319 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4448	4389	4369	4369
q2	342	371	220	220
q3	4691	4995	4463	4463
q4	2090	2167	1384	1384
q5	4581	4430	4496	4430
q6	249	188	130	130
q7	1791	2109	1747	1747
q8	2749	2288	2428	2288
q9	8289	8265	8147	8147
q10	4826	4779	4304	4304
q11	604	416	377	377
q12	806	756	536	536
q13	3341	3715	2921	2921
q14	297	307	287	287
q15	q16	713	736	633	633
q17	1391	1541	1347	1347
q18	8360	7188	7156	7156
q19	1124	1099	1090	1090
q20	2232	2242	1947	1947
q21	5368	4603	4485	4485
q22	518	469	409	409
Total cold run time: 58810 ms
Total hot run time: 52670 ms

[hello-stephen]

TPC-H: Total hot run time: 29318 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c06467b2dc0bd4a914293b3a0dae97c52b8dc4bb, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17659	4290	4190	4190
q2	2039	314	185	185
q3	10333	1436	834	834
q4	4683	480	346	346
q5	7530	863	585	585
q6	186	174	141	141
q7	781	828	623	623
q8	9327	1482	1593	1482
q9	5852	4580	4534	4534
q10	6733	1790	1524	1524
q11	442	276	240	240
q12	625	433	299	299
q13	18050	3404	2786	2786
q14	266	263	238	238
q15	q16	781	792	731	731
q17	1016	954	994	954
q18	6926	5824	5607	5607
q19	1297	1299	1040	1040
q20	489	402	278	278
q21	5828	2629	2387	2387
q22	440	377	314	314
Total cold run time: 101283 ms
Total hot run time: 29318 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4585	4474	4485	4474
q2	330	377	227	227
q3	4594	4986	4422	4422
q4	2092	2171	1404	1404
q5	4533	4381	4371	4371
q6	240	180	132	132
q7	1699	1933	1889	1889
q8	2671	2276	2209	2209
q9	8333	8270	7998	7998
q10	4858	4783	4332	4332
q11	588	410	397	397
q12	782	779	590	590
q13	3378	3566	2982	2982
q14	309	299	283	283
q15	q16	699	733	647	647
q17	1380	1404	1483	1404
q18	7754	7423	7283	7283
q19	1230	1088	1100	1088
q20	2228	2206	1954	1954
q21	5368	4629	4427	4427
q22	529	450	400	400
Total cold run time: 58180 ms
Total hot run time: 52913 ms

[hello-stephen]

TPC-DS: Total hot run time: 173178 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c06467b2dc0bd4a914293b3a0dae97c52b8dc4bb, data reload: false
query5	4330	637	481	481
query6	451	186	171	171
query7	4848	544	297	297
query8	339	185	170	170
query9	8768	4012	4013	4012
query10	482	314	259	259
query11	5949	2366	2147	2147
query12	149	100	109	100
query13	1248	645	438	438
query14	6217	5333	4997	4997
query14_1	4365	4282	4294	4282
query15	206	199	179	179
query16	983	481	454	454
query17	899	668	566	566
query18	2412	468	338	338
query19	200	181	139	139
query20	104	108	105	105
query21	218	134	112	112
query22	13641	13593	13422	13422
query23	17338	16624	16342	16342
query23_1	16357	16286	16267	16267
query24	7582	1767	1274	1274
query24_1	1346	1333	1311	1311
query25	552	442	378	378
query26	1302	341	174	174
query27	2683	551	335	335
query28	4494	2053	2046	2046
query29	1072	643	510	510
query30	321	240	205	205
query31	1126	1095	979	979
query32	119	63	62	62
query33	525	327	289	289
query34	1167	1130	636	636
query35	782	786	685	685
query36	1415	1406	1228	1228
query37	155	106	92	92
query38	1890	1717	1674	1674
query39	936	921	897	897
query39_1	881	884	874	874
query40	226	128	112	112
query41	80	69	70	69
query42	91	90	87	87
query43	318	326	282	282
query44	1427	783	788	783
query45	206	190	180	180
query46	1142	1232	774	774
query47	2357	2356	2203	2203
query48	420	416	287	287
query49	589	424	320	320
query50	948	356	260	260
query51	4300	4321	4263	4263
query52	85	84	72	72
query53	247	263	190	190
query54	292	236	211	211
query55	76	70	71	70
query56	255	250	228	228
query57	1462	1405	1310	1310
query58	254	221	221	221
query59	1556	1673	1398	1398
query60	303	242	262	242
query61	175	180	180	180
query62	708	659	593	593
query63	230	194	204	194
query64	2593	805	604	604
query65	4885	4809	4782	4782
query66	1791	457	333	333
query67	29793	29779	29520	29520
query68	3060	1537	918	918
query69	407	296	259	259
query70	1067	991	959	959
query71	297	228	200	200
query72	3007	2692	2352	2352
query73	814	774	433	433
query74	5104	4965	4784	4784
query75	2553	2560	2174	2174
query76	2311	1164	809	809
query77	348	369	288	288
query78	12579	12544	11890	11890
query79	1374	1234	757	757
query80	1271	476	379	379
query81	525	276	242	242
query82	629	161	125	125
query83	351	273	258	258
query84	319	148	114	114
query85	895	513	416	416
query86	421	298	287	287
query87	1847	1813	1783	1783
query88	3709	2781	2748	2748
query89	418	384	337	337
query90	1941	183	183	183
query91	170	158	130	130
query92	62	59	56	56
query93	1579	1417	986	986
query94	705	357	317	317
query95	685	456	342	342
query96	1106	778	324	324
query97	2714	2702	2571	2571
query98	216	202	208	202
query99	1182	1170	1049	1049
Total cold run time: 258377 ms
Total hot run time: 173178 ms

[hello-stephen]

ClickBench: Total hot run time: 25.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c06467b2dc0bd4a914293b3a0dae97c52b8dc4bb, data reload: false
query1	0.01	0.00	0.01
query2	0.10	0.04	0.04
query3	0.25	0.14	0.13
query4	1.60	0.14	0.13
query5	0.23	0.26	0.22
query6	1.26	1.06	1.07
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.39	0.31	0.33
query10	0.54	0.57	0.55
query11	0.19	0.14	0.14
query12	0.20	0.14	0.14
query13	0.47	0.47	0.48
query14	1.00	1.02	1.00
query15	0.60	0.59	0.61
query16	0.30	0.32	0.30
query17	1.06	1.05	1.11
query18	0.23	0.22	0.21
query19	2.03	1.94	1.94
query20	0.01	0.01	0.01
query21	15.43	0.21	0.13
query22	4.93	0.06	0.05
query23	16.12	0.30	0.12
query24	2.96	0.41	0.32
query25	0.11	0.04	0.04
query26	0.74	0.21	0.14
query27	0.04	0.03	0.05
query28	3.50	0.97	0.54
query29	12.55	4.33	3.44
query30	0.27	0.15	0.15
query31	2.77	0.60	0.31
query32	3.22	0.60	0.49
query33	3.24	3.21	3.28
query34	15.49	4.24	3.54
query35	3.54	3.52	3.52
query36	0.55	0.44	0.43
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.05	0.04	0.04
Total cold run time: 96.56 s
Total hot run time: 25.15 s

[feiniaofeiafei]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 29338 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f21ccb787b6777f1f548c7c4093bf1dc4dae2a10, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17699	4032	4031	4031
q2	2002	337	196	196
q3	10307	1388	805	805
q4	4693	476	340	340
q5	7559	855	569	569
q6	181	173	137	137
q7	788	900	627	627
q8	9372	1699	1621	1621
q9	5781	4520	4483	4483
q10	6777	1785	1551	1551
q11	446	274	249	249
q12	625	421	293	293
q13	18088	3360	2684	2684
q14	272	268	246	246
q15	q16	776	786	709	709
q17	973	971	1020	971
q18	6857	5832	5587	5587
q19	1173	1256	1105	1105
q20	487	411	268	268
q21	5657	2556	2618	2556
q22	437	362	310	310
Total cold run time: 100950 ms
Total hot run time: 29338 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4461	4398	4320	4320
q2	336	381	236	236
q3	4756	4933	4426	4426
q4	2066	2168	1381	1381
q5	4405	4287	4317	4287
q6	231	176	131	131
q7	1739	1653	2090	1653
q8	2603	2262	2179	2179
q9	8162	8234	7930	7930
q10	4871	4768	4318	4318
q11	583	447	414	414
q12	740	762	548	548
q13	3325	3610	2950	2950
q14	298	312	270	270
q15	q16	706	726	634	634
q17	1352	1317	1320	1317
q18	8047	7272	7528	7272
q19	1173	1175	1132	1132
q20	2204	2210	1952	1952
q21	5276	4580	4525	4525
q22	518	449	405	405
Total cold run time: 57852 ms
Total hot run time: 52280 ms

[hello-stephen]

TPC-DS: Total hot run time: 173154 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f21ccb787b6777f1f548c7c4093bf1dc4dae2a10, data reload: false
query5	4314	630	480	480
query6	440	188	174	174
query7	4859	580	323	323
query8	345	178	172	172
query9	8740	4137	4076	4076
query10	465	312	251	251
query11	5842	2359	2146	2146
query12	161	105	103	103
query13	1257	611	438	438
query14	6293	5303	4965	4965
query14_1	4303	4326	4333	4326
query15	211	207	183	183
query16	1009	465	464	464
query17	968	721	604	604
query18	2437	508	349	349
query19	217	197	150	150
query20	112	108	116	108
query21	218	150	120	120
query22	13612	13727	13407	13407
query23	17496	16536	16167	16167
query23_1	16254	16339	16264	16264
query24	7490	1839	1310	1310
query24_1	1327	1336	1320	1320
query25	584	496	358	358
query26	1329	323	162	162
query27	2714	600	356	356
query28	4484	2093	2069	2069
query29	1057	617	465	465
query30	307	236	205	205
query31	1118	1067	953	953
query32	110	61	58	58
query33	520	309	244	244
query34	1177	1138	653	653
query35	773	771	654	654
query36	1357	1395	1190	1190
query37	152	103	89	89
query38	1890	1715	1634	1634
query39	917	907	892	892
query39_1	868	901	862	862
query40	215	124	102	102
query41	64	65	64	64
query42	90	83	89	83
query43	318	326	273	273
query44	1511	794	782	782
query45	213	187	183	183
query46	1111	1206	722	722
query47	2398	2334	2216	2216
query48	378	398	278	278
query49	587	426	311	311
query50	998	355	257	257
query51	4306	4274	4246	4246
query52	80	80	71	71
query53	250	268	193	193
query54	284	224	191	191
query55	71	69	64	64
query56	228	234	209	209
query57	1451	1421	1307	1307
query58	241	212	199	199
query59	1551	1623	1418	1418
query60	274	231	225	225
query61	153	147	149	147
query62	697	658	583	583
query63	225	187	187	187
query64	2542	769	589	589
query65	4874	4807	4787	4787
query66	1795	453	328	328
query67	29797	29658	29536	29536
query68	3185	1595	961	961
query69	403	302	254	254
query70	1055	966	976	966
query71	298	241	222	222
query72	2960	2677	2474	2474
query73	882	874	440	440
query74	5097	4989	4787	4787
query75	2615	2579	2179	2179
query76	2321	1258	786	786
query77	397	407	289	289
query78	12425	12419	11706	11706
query79	1421	1149	775	775
query80	611	506	420	420
query81	456	280	242	242
query82	592	167	124	124
query83	361	280	256	256
query84	313	148	113	113
query85	858	523	420	420
query86	370	290	293	290
query87	1837	1822	1739	1739
query88	3762	2807	2791	2791
query89	436	386	338	338
query90	1971	179	183	179
query91	174	160	132	132
query92	64	63	57	57
query93	1581	1477	913	913
query94	527	351	295	295
query95	675	380	354	354
query96	1030	762	370	370
query97	2707	2672	2566	2566
query98	215	204	202	202
query99	1174	1146	1032	1032
Total cold run time: 257670 ms
Total hot run time: 173154 ms

[hello-stephen]

ClickBench: Total hot run time: 25.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f21ccb787b6777f1f548c7c4093bf1dc4dae2a10, data reload: false
query1	0.00	0.01	0.00
query2	0.10	0.05	0.05
query3	0.25	0.13	0.13
query4	1.60	0.13	0.13
query5	0.25	0.22	0.23
query6	1.26	1.07	1.04
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.43	0.32	0.30
query10	0.54	0.54	0.56
query11	0.19	0.14	0.14
query12	0.18	0.14	0.15
query13	0.48	0.49	0.47
query14	1.00	1.02	1.00
query15	0.60	0.58	0.59
query16	0.30	0.33	0.33
query17	1.11	1.08	1.11
query18	0.22	0.21	0.21
query19	2.02	1.88	1.96
query20	0.02	0.01	0.01
query21	15.43	0.23	0.14
query22	4.85	0.05	0.06
query23	16.16	0.29	0.12
query24	2.94	0.43	0.34
query25	0.12	0.06	0.04
query26	0.72	0.21	0.16
query27	0.03	0.04	0.03
query28	3.57	0.88	0.52
query29	12.47	4.34	3.43
query30	0.28	0.17	0.15
query31	2.76	0.59	0.31
query32	3.22	0.60	0.49
query33	3.18	3.29	3.22
query34	15.54	4.19	3.52
query35	3.56	3.53	3.52
query36	0.56	0.42	0.43
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.16	0.16
query41	0.08	0.04	0.04
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.55 s
Total hot run time: 25.15 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 79.71% (55/69) 🎉
Increment coverage report
Complete coverage report

[feiniaofeiafei]

run nonConcurrent

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 35.48% (55/155) 🎉
Increment coverage report
Complete coverage report

[morrySnow]

/review

[morrySnow]

Review Summary

The core direction of this PR — switching from cost-based to stats-based NDV heuristics — is sound and removes unnecessary optimizer overhead. The per-child independent decision logic is correctly implemented. However, I found several issues that should be addressed:

Key findings:

1. rejectNLJ method has multiple issues: misleading name, only peels one level of project, and dead code in groupByCardinality
2. Missing documentation for NDV heuristic thresholds: The magic constants lack rationale comments
3. Dead code: The fallback return 2; in groupByCardinality and rowCount == 0 check are unreachable
4. Hint support silently dropped: use_INFER_SET_OPERATOR_DISTINCT / use_NO_INFER_SET_OPERATOR_DISTINCT hints are now ignored — needs a TODO or deprecation plan
5. Test coverage gap: The regression test covering 20+ scenarios was deleted; only 1 scenario remains

See inline comments for detailed suggestions.

[feiniaofeiafei]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 28918 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5aab41304991ba454c25867e7a4fcac69863616c, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17641	4132	4100	4100
q2	2090	317	189	189
q3	10233	1439	818	818
q4	4678	483	334	334
q5	7495	868	602	602
q6	186	182	143	143
q7	799	855	627	627
q8	9855	1647	1646	1646
q9	6207	4583	4515	4515
q10	6796	1823	1511	1511
q11	442	271	238	238
q12	635	457	285	285
q13	18134	3408	2751	2751
q14	286	256	234	234
q15	q16	783	773	701	701
q17	3560	1053	697	697
q18	6879	5763	5509	5509
q19	2029	1287	1143	1143
q20	497	407	259	259
q21	5857	2597	2321	2321
q22	440	360	295	295
Total cold run time: 105522 ms
Total hot run time: 28918 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4588	4471	4478	4471
q2	327	372	222	222
q3	4620	4951	4398	4398
q4	2141	2211	1356	1356
q5	4595	4445	4788	4445
q6	288	214	145	145
q7	2097	1853	1593	1593
q8	2753	2251	2320	2251
q9	8267	8318	8235	8235
q10	4919	4809	4321	4321
q11	615	416	397	397
q12	815	769	528	528
q13	3251	3670	3023	3023
q14	312	307	278	278
q15	q16	738	742	641	641
q17	1391	1429	1380	1380
q18	8056	7400	6880	6880
q19	1094	1065	1104	1065
q20	2242	2212	2020	2020
q21	5372	4701	4476	4476
q22	518	442	418	418
Total cold run time: 58999 ms
Total hot run time: 52543 ms

[hello-stephen]

TPC-DS: Total hot run time: 171068 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5aab41304991ba454c25867e7a4fcac69863616c, data reload: false
query5	4315	628	481	481
query6	438	189	167	167
query7	4816	564	308	308
query8	336	184	167	167
query9	8794	4056	4056	4056
query10	434	334	255	255
query11	5891	2349	2126	2126
query12	159	108	114	108
query13	1266	596	428	428
query14	6400	5265	4924	4924
query14_1	4271	4284	4260	4260
query15	207	197	182	182
query16	1031	469	400	400
query17	1123	701	571	571
query18	2693	471	336	336
query19	197	177	140	140
query20	116	107	107	107
query21	215	137	115	115
query22	13659	13565	13390	13390
query23	17208	16401	16050	16050
query23_1	16245	16179	16247	16179
query24	7570	1784	1313	1313
query24_1	1349	1321	1290	1290
query25	556	471	410	410
query26	1326	338	173	173
query27	2638	570	339	339
query28	4446	2015	1984	1984
query29	1089	628	507	507
query30	318	233	205	205
query31	1113	1087	972	972
query32	121	64	60	60
query33	540	333	257	257
query34	1184	1142	649	649
query35	769	786	661	661
query36	1390	1384	1193	1193
query37	155	109	96	96
query38	1888	1744	1654	1654
query39	951	944	894	894
query39_1	897	896	861	861
query40	225	129	107	107
query41	74	69	69	69
query42	91	87	88	87
query43	323	331	276	276
query44	1438	802	781	781
query45	213	197	183	183
query46	1074	1229	763	763
query47	2380	2386	2232	2232
query48	419	431	316	316
query49	596	434	321	321
query50	1113	359	269	269
query51	4440	4341	4305	4305
query52	81	82	71	71
query53	257	260	196	196
query54	303	226	210	210
query55	74	71	67	67
query56	250	234	235	234
query57	1453	1436	1309	1309
query58	249	223	219	219
query59	1577	1659	1402	1402
query60	292	257	232	232
query61	180	174	174	174
query62	698	652	587	587
query63	245	198	193	193
query64	2554	754	617	617
query65	4867	4740	4766	4740
query66	1749	469	344	344
query67	29706	29644	29498	29498
query68	3253	1614	877	877
query69	412	288	273	273
query70	1049	995	977	977
query71	296	231	212	212
query72	2956	2589	1891	1891
query73	850	796	408	408
query74	5092	4962	4745	4745
query75	2574	2519	2168	2168
query76	2349	1188	789	789
query77	375	377	275	275
query78	12392	12484	11862	11862
query79	1404	1207	769	769
query80	1279	476	387	387
query81	523	283	241	241
query82	669	157	126	126
query83	362	276	256	256
query84	268	146	109	109
query85	923	519	442	442
query86	433	316	293	293
query87	1843	1855	1766	1766
query88	3709	2789	2757	2757
query89	428	378	336	336
query90	2008	177	185	177
query91	174	158	132	132
query92	63	60	55	55
query93	1572	1448	848	848
query94	786	357	300	300
query95	662	367	346	346
query96	1021	759	359	359
query97	2688	2686	2551	2551
query98	216	204	198	198
query99	1196	1144	1016	1016
Total cold run time: 259288 ms
Total hot run time: 171068 ms

[hello-stephen]

ClickBench: Total hot run time: 25.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5aab41304991ba454c25867e7a4fcac69863616c, data reload: false
query1	0.01	0.00	0.01
query2	0.11	0.05	0.05
query3	0.25	0.13	0.14
query4	1.61	0.16	0.15
query5	0.24	0.22	0.22
query6	1.26	1.09	1.06
query7	0.04	0.01	0.01
query8	0.05	0.04	0.03
query9	0.37	0.31	0.31
query10	0.57	0.54	0.60
query11	0.19	0.14	0.14
query12	0.18	0.15	0.14
query13	0.47	0.47	0.49
query14	1.01	1.01	1.00
query15	0.62	0.59	0.58
query16	0.33	0.31	0.33
query17	1.09	1.12	1.09
query18	0.22	0.22	0.21
query19	1.99	2.00	1.96
query20	0.01	0.01	0.01
query21	15.49	0.21	0.13
query22	4.80	0.05	0.06
query23	16.12	0.31	0.12
query24	3.29	0.39	0.32
query25	0.13	0.05	0.04
query26	0.72	0.21	0.15
query27	0.05	0.05	0.04
query28	3.58	0.96	0.55
query29	12.51	4.30	3.49
query30	0.27	0.14	0.17
query31	2.76	0.56	0.32
query32	3.22	0.59	0.48
query33	3.22	3.22	3.23
query34	15.81	4.18	3.49
query35	3.50	3.51	3.50
query36	0.57	0.42	0.42
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 97.18 s
Total hot run time: 25.23 s

[hello-stephen]

FE UT Coverage Report

Increment line coverage 54.84% (34/62) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 77.42% (48/62) 🎉
Increment coverage report
Complete coverage report

[feiniaofeiafei]

run nonConcurrent

[feiniaofeiafei]

run nonConcurrent

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 77.42% (48/62) 🎉
Increment coverage report
Complete coverage report

[morrySnow]

/review

[github-actions]

Automated review summary for PR #64618 (5aab41304991ba454c25867e7a4fcac69863616c).

I reviewed all changed files and the relevant surrounding optimizer, stats, set-operation, and test paths. I did not find a new substantiated issue that is not already covered by existing inline review threads, so I am submitting a summary-only comment with no inline comments.

Critical checkpoint conclusions:

• Goal and proof: The PR moves InferSetOperatorDistinct from a cost-based rewrite wrapper to a normal top-down rewrite and gates local distinct insertion by NDV/cardinality stats. The new JUnit proves one per-child NDV decision case, while broader deleted regression coverage has already been raised in existing review threads.
• Scope and clarity: Code changes are focused on rule placement, the NDV heuristic, and expected plan-shape churn.
• Concurrency/lifecycle/config/persistence: No new concurrency, lifecycle, config, persistence, FE-BE protocol, or storage-format path is introduced.
• Parallel paths and compatibility: I checked UNION, INTERSECT, EXCEPT, aggregate children, set-op child-output mapping, stats derivation, and no-cost/hint paths. The no-cost rewriter and hint behavior concern is already covered by existing threads on Rewriter.java:607.
• Test coverage/results: git diff --check is clean. I did not run FE tests because thirdparty/installed/bin/protoc is missing, which fe/AGENTS.md treats as a stop condition. The regression-suite deletion versus replacement JUnit coverage concern is already covered by existing threads on InferSetOperatorDistinctTest.java:44.
• User focus: No additional user-provided focus points were listed.

Subagent conclusions:

• optimizer-rewrite: no new candidate; confirmed the generated aggregate preserves set-op child output slots and found no separate join/aggregate semantic regression beyond existing threads.
• tests-session-config: no new candidate; confirmed coverage and hint/no-cost concerns are existing-thread duplicates, and noted the local FE test stop condition.
• Final convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same zero-inline-comment final set.

[github-actions]

PR approved by at least one committer and no changes requested.

LGTM