[feature](nereids) support multi_distinct_collect_list and multi_distinct_array_agg by Baymine · Pull Request #65245 · apache/doris

Baymine · 2026-07-06T07:01:42Z

What problem does this PR solve?

Issue Number: close#65244

Problem Summary:
When a single GROUP BY contained more than one DISTINCT aggregate and any of
them was collect_list(distinct ...) or array_agg(distinct ...), planning failed
with "... can't support multi distinct". CheckMultiDistinct rejects any distinct
aggregate that is not a SupportMultiDistinct, and collect_list / array_agg did
not implement that interface, so queries such as

select g, collect_list(distinct a), collect_list(distinct b) from t group by g
select g, array_agg(distinct a), array_agg(distinct b) from t group by g

could not run at all.

This adds the missing multi-distinct variants and wires them into the existing
framework:

FE: CollectList and ArrayAgg now implement SupportMultiDistinct and convert to
the new MultiDistinctCollectList / MultiDistinctArrayAgg functions. The new
functions implement MultiDistinction, are registered in
BuiltinAggregateFunctions, and get visitor hooks in AggregateFunctionVisitor.
BE: registers multi_distinct_collect_list and multi_distinct_array_agg. Both
dedup in the aggregate state via the existing Set-backed collect data, because
the Nereids multi-distinct plan does not push the distinct argument into the
group-by key.

Release note

Support multiple DISTINCT aggregates in one GROUP BY when they include
collect_list(distinct ...) or array_agg(distinct ...).

Check List (For Author)

Test:
- Unit Test: MultiDistinctArrayAggTest, MultiDistinctCollectListTest, CheckMultiDistinctTest (28 cases, all pass).
- Regression test: added regression-test/suites/nereids_p0/multi_distinct/multi_distinct_collect_list.groovy (runs in CI).
Behavior changed:
- Yes. Queries that previously failed with "can't support multi distinct" for collect_list/array_agg now plan and run.
Does this need documentation?
- No.

…inct_array_agg ### What problem does this PR solve? Issue Number: closeapache#65244 Problem Summary: When a single GROUP BY contained more than one DISTINCT aggregate and any of them was collect_list(distinct ...) or array_agg(distinct ...), planning failed with "... can't support multi distinct". CheckMultiDistinct rejects any distinct aggregate that is not a SupportMultiDistinct, and collect_list / array_agg did not implement that interface, so queries such as select g, collect_list(distinct a), collect_list(distinct b) from t group by g select g, array_agg(distinct a), array_agg(distinct b) from t group by g could not run at all. This adds the missing multi-distinct variants and wires them into the existing framework: - FE: CollectList and ArrayAgg now implement SupportMultiDistinct and convert to the new MultiDistinctCollectList / MultiDistinctArrayAgg functions. The new functions implement MultiDistinction, are registered in BuiltinAggregateFunctions, and get visitor hooks in AggregateFunctionVisitor. - BE: registers multi_distinct_collect_list and multi_distinct_array_agg. Both dedup in the aggregate state via the existing Set-backed collect data, because the Nereids multi-distinct plan does not push the distinct argument into the group-by key. ### Release note Support multiple DISTINCT aggregates in one GROUP BY when they include collect_list(distinct ...) or array_agg(distinct ...). ### Check List (For Author) - Test: - [x] Unit Test: MultiDistinctArrayAggTest, MultiDistinctCollectListTest, CheckMultiDistinctTest (28 cases, all pass). - [x] Regression test: added regression-test/suites/nereids_p0/multi_distinct/multi_distinct_collect_list.groovy (verified via CI; local cluster restart was skipped in this environment). - Behavior changed: Yes. Queries that previously failed with "can't support multi distinct" for collect_list/array_agg now plan and run. - Does this need documentation: No.

hello-stephen · 2026-07-06T07:01:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Baymine · 2026-07-06T07:06:05Z

run buildall

hello-stephen · 2026-07-06T08:40:39Z

FE UT Coverage Report

Increment line coverage 94.44% (34/36) 🎉
Increment coverage report
Complete coverage report

morrySnow

Review Summary

This PR adds multi_distinct_collect_list and multi_distinct_array_agg support — a solid feature enabling queries with multiple DISTINCT aggregates in one GROUP BY that include collect_list/array_agg. The overall approach (FE: SupportMultiDistinct + MultiDistinction classes; BE: registering new function names) follows existing patterns. Good test coverage.

Below are the issues found, ranked by severity:

1. [HIGH] array_agg distinct path silently drops nulls

See inline comment in aggregate_function_array_agg.cpp. The Set-based dedup data structure cannot represent null elements, and the null-skipping adapter drops them before the inner function sees them. This means array_agg(distinct nullable_col) returns different results when it takes the multi-distinct path vs. the single-distinct path.

2. [MEDIUM] canSkewRewrite can produce nested ARRAY<ARRAY>

Adding SupportMultiDistinct to ArrayAgg/CollectList makes canSkewRewrite() (Aggregate.java:~206) return true. In SplitAggMultiPhase.java:~334, phase 3 calls getAggregateFunction() which falls to the else branch: aggFunc.withDistinctAndChildren(false, children). This wraps the already-array intermediate slot in another array_agg(), producing ARRAY<ARRAY<T>>. A query with a skew hint on these functions would return deeply nested arrays. Consider adding a guard in canSkewRewrite() or handling array concatenation in the skew phases.

3. [MEDIUM] DistinctWindowExpression doesn't handle ArrayAgg/CollectList

DistinctWindowExpression.java:87-101 has hardcoded instanceof checks for Count, Sum, GroupConcat — it never queries SupportMultiDistinct. ARRAY_AGG(DISTINCT col) OVER(...) or COLLECT_LIST(DISTINCT col) OVER(...) would not be converted. Consider refactoring to use the SupportMultiDistinct interface like other rules do.

4. [LOW] get_name() returns "collect_set" for multi_distinct_* functions

aggregate_function_collect.h:410-416: get_name() returns "collect_set" for all Set-based data. Three different functions (collect_set, multi_distinct_collect_list, multi_distinct_array_agg) all report as "collect_set" in EXPLAIN ANALYZE profiles and error messages, making debugging confusing.

5. [LOW] Unused #include

See inline comment in aggregate_function_array_agg.cpp.

6. [LOW] convertToMultiDistinct silently drops limit argument

See inline comment in CollectList.java.

hello-stephen · 2026-07-06T12:18:07Z

BE Regression && UT Coverage Report

Increment line coverage 88.89% (16/18) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.02% (29143/39913)
Line Coverage	56.63% (315018/556315)
Region Coverage	53.05% (261883/493638)
Branch Coverage	54.16% (115158/212637)

hello-stephen · 2026-07-06T12:32:38Z

FE Regression Coverage Report

Increment line coverage 35.48% (22/62) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Issue Number: closeapache#65244 Problem Summary: Follow-up fixes for the multi_distinct_collect_list / multi_distinct_array_agg review: 1. array_agg(distinct) dropped NULLs on the multi-distinct path. The Set-backed collect data cannot store a null and the not-nullable wrapper skipped null rows, so array_agg(distinct nullable_col) returned a different result than the single-distinct path (which keeps one NULL). The multi_distinct_array_agg path now feeds the nullable column in directly and records whether a NULL was seen (has_null), emitting a single NULL element on output, matching array_agg semantics. collect_set / multi_distinct_collect_list keep ignoring NULLs. 2. Skew rewrite could produce ARRAY<ARRAY<T>>. Aggregate.canSkewRewrite() now excludes array-returning aggregates, so array_agg/collect_list with a skew hint fall back to the normal multi-distinct plan instead of re-wrapping the array intermediate in another array_agg. 3. DISTINCT array_agg/collect_list as window functions were not converted. DistinctWindowExpression now uses the SupportMultiDistinct interface instead of hardcoded instanceof checks, so ARRAY_AGG(DISTINCT ..) OVER(..) and COLLECT_LIST(DISTINCT ..) OVER(..) are rewritten like count/sum/group_concat. 4. get_name() reported "collect_set" for every Set-backed collect function. The real function name is now threaded into AggregateFunctionCollect so collect_set / multi_distinct_collect_list / multi_distinct_array_agg report distinct names in EXPLAIN / profiles. 5. Removed the unused #include "common/status.h" from aggregate_function_array_agg.cpp. 6. FE and BE advertised different type support: MultiDistinctArrayAgg / MultiDistinctCollectList accept AnyDataType, but the Set-backed BE path only handles scalar/string element types and would raise an internal error for complex/object types. SupportMultiDistinct.checkSupportMultiDistinct() (called from CheckMultiDistinct when the multi-distinct rewrite is required) now rejects those types during FE analysis with a user-facing error. Single distinct (no rewrite) still supports complex types as before. ### Release note None. ### Check List (For Author) - Test: - [x] Unit Test: FE MultiDistinctArrayAggTest (9), MultiDistinctCollectListTest (12), CheckMultiDistinctTest (10, incl. complex-type rejection) all pass. BE VAggCollectTest adds test_multi_distinct_array_agg_preserves_null (NULL preservation add/serialize/deserialize/merge round trip) and test_multi_distinct_functions_are_registered; all 6 pass. - [x] Regression test: extended regression-test/suites/nereids_p0/multi_distinct/multi_distinct_collect_list.groovy with NULL-handling, DISTINCT window-function, and complex-type rejection cases. - Behavior changed: Yes. array_agg(distinct nullable) on the multi-distinct path preserves a NULL element; DISTINCT array_agg/collect_list are usable as window functions; complex element types are rejected in FE for the multi-distinct rewrite instead of failing at BE runtime. - Does this need documentation: No.

Baymine · 2026-07-07T13:33:24Z

run buildall

hello-stephen · 2026-07-07T15:29:53Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.53% (23196/41036)
Line Coverage	40.08% (225118/561713)
Region Coverage	36.03% (177743/493298)
Branch Coverage	37.08% (79000/213035)

Baymine · 2026-07-13T09:57:57Z

run buildall

hello-stephen · 2026-07-13T10:29:27Z

TPC-H: Total hot run time: 29647 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39c337d0e3887ebe78faa4e308c0f018dfe57d38, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17873	4082	4033	4033
q2	2027	330	199	199
q3	10262	1440	816	816
q4	4691	481	347	347
q5	7522	864	577	577
q6	204	182	139	139
q7	784	842	612	612
q8	10126	1573	1626	1573
q9	5925	4406	4391	4391
q10	6823	1798	1524	1524
q11	507	354	315	315
q12	755	576	441	441
q13	18133	3430	2723	2723
q14	277	271	257	257
q15	q16	785	785	717	717
q17	1052	1033	889	889
q18	7040	5792	5688	5688
q19	1454	1383	1099	1099
q20	739	644	560	560
q21	5699	2577	2453	2453
q22	431	351	294	294
Total cold run time: 103109 ms
Total hot run time: 29647 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4383	4282	4283	4282
q2	286	340	214	214
q3	4663	5051	4394	4394
q4	2059	2166	1363	1363
q5	4456	4303	4360	4303
q6	238	182	128	128
q7	2144	1912	1632	1632
q8	2463	2176	2105	2105
q9	8007	7864	7834	7834
q10	4793	4773	4335	4335
q11	572	561	404	404
q12	751	798	541	541
q13	3290	3614	3017	3017
q14	309	311	270	270
q15	q16	717	741	668	668
q17	1356	1327	1310	1310
q18	8140	7624	7064	7064
q19	1091	1098	1055	1055
q20	2260	2230	1960	1960
q21	5215	4583	4464	4464
q22	501	458	420	420
Total cold run time: 57694 ms
Total hot run time: 51763 ms

hello-stephen · 2026-07-13T10:40:56Z

TPC-DS: Total hot run time: 180004 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39c337d0e3887ebe78faa4e308c0f018dfe57d38, data reload: false
query5	4372	623	488	488
query6	474	212	192	192
query7	4966	591	342	342
query8	352	183	172	172
query9	8808	4035	4012	4012
query10	472	346	286	286
query11	5848	2364	2177	2177
query12	163	106	100	100
query13	1278	582	450	450
query14	6312	5280	4899	4899
query14_1	4333	4269	4249	4249
query15	212	202	179	179
query16	1005	463	432	432
query17	1124	709	572	572
query18	2434	470	342	342
query19	202	191	144	144
query20	108	104	103	103
query21	229	148	130	130
query22	13629	13675	13472	13472
query23	17420	16507	16249	16249
query23_1	16229	16231	16184	16184
query24	7517	1755	1287	1287
query24_1	1298	1305	1294	1294
query25	539	448	371	371
query26	1335	353	207	207
query27	2604	615	387	387
query28	4428	2040	2021	2021
query29	1108	626	495	495
query30	340	272	230	230
query31	1121	1084	972	972
query32	103	64	60	60
query33	540	323	271	271
query34	1162	1150	656	656
query35	781	799	686	686
query36	1438	1417	1253	1253
query37	154	107	95	95
query38	1877	1718	1653	1653
query39	926	931	877	877
query39_1	867	902	877	877
query40	255	167	142	142
query41	74	69	68	68
query42	100	95	97	95
query43	320	328	287	287
query44	1438	793	779	779
query45	201	198	183	183
query46	1048	1194	765	765
query47	2388	2330	2241	2241
query48	427	422	296	296
query49	584	433	318	318
query50	1110	425	332	332
query51	10635	10548	10868	10548
query52	88	87	78	78
query53	250	283	210	210
query54	314	246	237	237
query55	77	74	70	70
query56	320	315	290	290
query57	1452	1422	1364	1364
query58	314	272	264	264
query59	1589	1655	1418	1418
query60	340	275	262	262
query61	171	171	171	171
query62	702	651	576	576
query63	235	203	208	203
query64	2931	1032	866	866
query65	4826	4766	4757	4757
query66	1827	511	388	388
query67	28890	29494	29326	29326
query68	3265	1607	964	964
query69	411	303	271	271
query70	1036	966	980	966
query71	353	303	309	303
query72	3076	2692	2405	2405
query73	829	807	453	453
query74	5096	4950	4732	4732
query75	2617	2572	2205	2205
query76	2328	1179	809	809
query77	352	375	286	286
query78	12407	12414	11641	11641
query79	1467	1185	793	793
query80	1275	526	456	456
query81	524	326	285	285
query82	612	158	122	122
query83	359	312	287	287
query84	270	155	130	130
query85	963	634	495	495
query86	418	305	289	289
query87	1844	1829	1752	1752
query88	3748	2827	2786	2786
query89	458	409	353	353
query90	1903	201	193	193
query91	201	185	166	166
query92	61	59	56	56
query93	1701	1562	996	996
query94	743	347	299	299
query95	765	559	475	475
query96	1099	814	368	368
query97	2707	2668	2552	2552
query98	211	204	200	200
query99	1197	1173	1036	1036
Total cold run time: 265746 ms
Total hot run time: 180004 ms

hello-stephen · 2026-07-13T10:45:56Z

ClickBench: Total hot run time: 25.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39c337d0e3887ebe78faa4e308c0f018dfe57d38, data reload: false
query1	0.00	0.00	0.00
query2	0.09	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.14	0.14
query5	0.28	0.22	0.23
query6	1.23	1.07	1.04
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.39	0.31	0.32
query10	0.56	0.59	0.60
query11	0.20	0.14	0.14
query12	0.18	0.14	0.15
query13	0.48	0.48	0.48
query14	1.03	1.01	0.98
query15	0.61	0.59	0.60
query16	0.35	0.34	0.32
query17	1.08	1.11	1.12
query18	0.23	0.22	0.21
query19	2.04	1.95	1.96
query20	0.02	0.01	0.01
query21	15.45	0.21	0.13
query22	4.96	0.06	0.05
query23	16.10	0.32	0.12
query24	2.89	0.39	0.33
query25	0.12	0.04	0.03
query26	0.73	0.20	0.16
query27	0.04	0.04	0.05
query28	3.45	0.89	0.54
query29	12.47	4.10	3.28
query30	0.28	0.15	0.16
query31	2.79	0.60	0.31
query32	3.22	0.60	0.50
query33	3.20	3.27	3.20
query34	15.71	4.26	3.52
query35	3.54	3.49	3.52
query36	0.55	0.43	0.45
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.17	0.15
query41	0.09	0.03	0.02
query42	0.05	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.77 s
Total hot run time: 25.09 s

hello-stephen · 2026-07-13T13:05:14Z

FE Regression Coverage Report

Increment line coverage 51.92% (27/52) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-07-13T13:20:42Z

BE Regression && UT Coverage Report

Increment line coverage 96.92% (63/65) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.78% (30241/40440)
Line Coverage	58.83% (332336/564880)
Region Coverage	55.55% (278599/501500)
Branch Coverage	56.83% (123216/216824)

feiniaofeiafei · 2026-07-24T03:48:45Z

The issue mentioned in this PR—where the two SQL statements select g, collect_list(distinct a), collect_list(distinct b) from t group by g and select g, array_agg(distinct a), array_agg(distinct b) from t group by g could not be executed—has been resolved after the merge of #65664. This PR implements a different approach to enable execution, using CTE (Common Table Expression) splitting to achieve the desired functionality.

Baymine requested review from 924060929, englefly, morrySnow and starocean999 as code owners July 6, 2026 07:01

morrySnow requested changes Jul 6, 2026
View reviewed changes

Comment threadbe/src/exprs/aggregate/aggregate_function_array_agg.cpp
Comment threadbe/src/exprs/aggregate/aggregate_function_array_agg.cpp
Comment thread...core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/CollectList.java

feiniaofeiafei reviewed Jul 7, 2026
View reviewed changes

Comment threadregression-test/suites/nereids_p0/multi_distinct/multi_distinct_collect_list.groovy

feiniaofeiafei reviewed Jul 7, 2026
View reviewed changes

Comment threadbe/src/exprs/aggregate/aggregate_function_array_agg.cpp

feiniaofeiafei reviewed Jul 7, 2026
View reviewed changes

Comment threadbe/src/exprs/aggregate/aggregate_function_array_agg.cpp

feiniaofeiafei reviewed Jul 7, 2026
View reviewed changes

Comment threadbe/src/exprs/aggregate/aggregate_function_array_agg.cpp

fix the style
39c337d

Baymine requested review from feiniaofeiafei and morrySnow July 13, 2026 10:08

Uh oh!

Conversation

Baymine commented Jul 6, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Jul 6, 2026

Uh oh!

Baymine commented Jul 6, 2026

Uh oh!

hello-stephen commented Jul 6, 2026

FE UT Coverage Report

Uh oh!

morrySnow left a comment

Choose a reason for hiding this comment

Review Summary

1. [HIGH] array_agg distinct path silently drops nulls

2. [MEDIUM] canSkewRewrite can produce nested ARRAY<ARRAY>

3. [MEDIUM] DistinctWindowExpression doesn't handle ArrayAgg/CollectList

4. [LOW] get_name() returns "collect_set" for multi_distinct_* functions

5. [LOW] Unused #include

6. [LOW] convertToMultiDistinct silently drops limit argument

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented Jul 6, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 6, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Baymine commented Jul 7, 2026

Uh oh!

hello-stephen commented Jul 7, 2026

BE UT Coverage Report

Uh oh!

Baymine commented Jul 13, 2026

Uh oh!

hello-stephen commented Jul 13, 2026

Uh oh!

hello-stephen commented Jul 13, 2026

Uh oh!

hello-stephen commented Jul 13, 2026

Uh oh!

hello-stephen commented Jul 13, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Jul 13, 2026

BE Regression && UT Coverage Report

Uh oh!

feiniaofeiafei commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants