[feature](function) Add array_agg_if aggregate function by yujun777 · Pull Request #67534 · apache/doris

yujun777 · 2026-09-04T11:18:28Z

Support conditional array aggregation: rows whose condition is false or NULL are skipped entirely; remaining rows are collected with array_agg semantics, so NULL elements are kept. Returns an empty array when no row matches.

Docs: apache/doris-website#4117

Key changes:

BE: implement array_agg_if(cond, elem) reusing the array_agg state template, parameterized by element argument index and input nullability, so the per-row add() path keeps zero-cost asserts
BE: handle raw (non-nullable) complex element types in the native serde state output/deserialize paths
FE: add ArrayAggIf aggregate expression class with cond-first signature and register it in the builtin aggregate catalog
regression: add three array_agg_if queries to the agg suite with generated expected output

Unit Test:

agg_array_agg_if_test: 9 cases covering cond filtering, NULL element retention, NULL cond, and string/complex (nullable and NOT NULL) element types, each exercised through single-add, merge and serialize/deserialize paths
agg_array_agg_test regression: 9 existing array_agg cases
end-to-end: array_agg_if queries verified against the generated agg.out on a local cluster

Support conditional array aggregation: rows whose condition is false or NULL are skipped entirely; remaining rows are collected with array_agg semantics, so NULL elements are kept. Key changes: - BE: implement array_agg_if(cond, elem) reusing the array_agg state template, parameterized by element argument index and input nullability, so the per-row add() path keeps zero-cost asserts - BE: handle raw (non-nullable) complex element types in the native serde state output/deserialize paths - FE: add ArrayAggIf aggregate expression class with cond-first signature and register it in the builtin aggregate catalog - regression: add three array_agg_if queries to the agg suite with generated expected output Unit Test: - agg_array_agg_if_test: 9 cases covering cond filtering, NULL element retention, NULL cond, and string/complex (nullable and NOT NULL) element types, each exercised through single-add, merge and serialize/deserialize paths - agg_array_agg_test regression: 9 existing array_agg cases - end-to-end: array_agg_if queries verified against the generated agg.out on a local cluster

hello-stephen · 2026-09-04T11:18:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

yujun777 · 2026-09-04T11:18:59Z

run buildall

hello-stephen · 2026-09-04T12:15:31Z

TPC-H: Total hot run time: 17129 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ee504d209061ef0d63ee90188cf89a2c5b1c28b, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17615	3103	3109	3103
q2	2099	266	231	231
q3	10237	932	537	537
q4	4668	272	210	210
q5	7655	622	382	382
q6	136	121	93	93
q7	533	521	379	379
q8	9238	910	936	910
q9	3462	2422	2413	2413
q10	6513	916	747	747
q11	392	205	185	185
q12	608	269	203	203
q13	18121	1545	1202	1202
q14	157	156	142	142
q15	q16	442	402	371	371
q17	1344	899	844	844
q18	3172	2341	2304	2304
q19	1253	943	766	766
q20	388	292	207	207
q21	5662	1669	1902	1669
q22	344	280	231	231
Total cold run time: 94039 ms
Total hot run time: 17129 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3443	3398	3390	3390
q2	528	392	378	378
q3	2298	2330	2197	2197
q4	1227	1226	942	942
q5	2216	2159	2139	2139
q6	173	122	89	89
q7	1042	925	878	878
q8	1637	1431	1425	1425
q9	3198	3153	3147	3147
q10	1950	1880	1702	1702
q11	363	280	255	255
q12	482	450	376	376
q13	1529	1586	1180	1180
q14	178	171	170	170
q15	q16	402	404	362	362
q17	3581	3301	3210	3210
q18	5006	4502	5130	4502
q19	969	902	877	877
q20	1057	980	845	845
q21	3922	3188	3236	3188
q22	407	365	330	330
Total cold run time: 35608 ms
Total hot run time: 31582 ms

hello-stephen · 2026-09-04T12:22:04Z

TPC-DS: Total hot run time: 82762 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ee504d209061ef0d63ee90188cf89a2c5b1c28b, data reload: false
query5	4246	409	346	346
query6	385	133	135	133
query7	4938	433	244	244
query8	291	127	124	124
query9	8690	2915	2935	2915
query10	386	230	188	188
query11	5380	1052	915	915
query12	120	78	78	78
query13	1203	459	336	336
query14	6151	2225	2120	2120
query14_1	1988	1990	1979	1979
query15	178	118	121	118
query16	920	399	368	368
query17	815	464	423	423
query18	2351	346	249	249
query19	172	140	113	113
query20	72	71	71	71
query21	203	102	88	88
query22	5342	5402	5411	5402
query23	6876	6369	6007	6007
query23_1	6195	6069	6003	6003
query24	7267	1103	798	798
query24_1	811	792	787	787
query25	475	315	271	271
query26	1244	237	131	131
query27	2771	408	263	263
query28	4678	1535	1511	1511
query29	918	460	359	359
query30	246	158	135	135
query31	818	405	341	341
query32	142	77	77	77
query33	465	230	183	183
query34	1001	828	511	511
query35	396	420	345	345
query36	573	558	532	532
query37	125	83	73	73
query38	1019	866	819	819
query39	510	484	514	484
query39_1	467	443	463	443
query40	194	90	74	74
query41	53	51	53	51
query42	73	71	72	71
query43	245	240	210	210
query44	1003	544	553	544
query45	107	109	101	101
query46	782	853	534	534
query47	755	770	711	711
query48	326	324	227	227
query49	534	233	187	187
query50	780	256	194	194
query51	8244	8245	8231	8231
query52	69	67	62	62
query53	192	205	150	150
query54	318	179	183	179
query55	73	57	53	53
query56	204	158	149	149
query57	693	655	660	655
query58	235	156	164	156
query59	1237	1233	1117	1117
query60	234	203	188	188
query61	129	135	130	130
query62	346	205	185	185
query63	176	140	139	139
query64	2778	704	561	561
query65	1604	1613	1665	1613
query66	1900	250	201	201
query67	9834	9880	9774	9774
query68	2992	1303	743	743
query69	354	227	198	198
query70	667	627	599	599
query71	254	185	173	173
query72	2393	1742	1536	1536
query73	655	621	329	329
query74	2004	1230	1151	1151
query75	1190	1129	971	971
query76	2382	732	548	548
query77	266	257	207	207
query78	3981	3710	3189	3189
query79	2674	808	590	590
query80	1611	331	278	278
query81	519	158	133	133
query82	990	118	101	101
query83	272	213	191	191
query84	301	108	88	88
query85	827	359	314	314
query86	475	175	173	173
query87	1026	978	913	913
query88	2870	2113	2067	2067
query89	289	197	170	170
query90	2045	128	130	128
query91	130	116	99	99
query92	90	64	68	64
query93	1833	1132	721	721
query94	646	272	235	235
query95	515	337	228	228
query96	806	594	293	293
query97	1049	1048	1002	1002
query98	181	137	133	133
query99	438	353	315	315
Total cold run time: 180100 ms
Total hot run time: 82762 ms

hello-stephen · 2026-09-04T12:26:27Z

ClickBench: Total hot run time: 14.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ee504d209061ef0d63ee90188cf89a2c5b1c28b, data reload: false
query1	0.01	0.00	0.01
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.60	0.09	0.10
query5	0.18	0.17	0.16
query6	1.25	0.67	0.68
query7	0.04	0.01	0.00
query8	0.04	0.03	0.03
query9	0.28	0.22	0.21
query10	0.34	0.36	0.36
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.31	0.31	0.30
query14	0.47	0.47	0.46
query15	0.38	0.34	0.34
query16	0.21	0.21	0.23
query17	0.74	0.68	0.70
query18	0.18	0.17	0.16
query19	1.21	1.11	1.14
query20	0.02	0.01	0.01
query21	15.45	0.16	0.13
query22	5.08	0.05	0.05
query23	16.21	0.25	0.10
query24	3.02	0.30	0.28
query25	0.12	0.04	0.02
query26	0.85	0.15	0.12
query27	0.03	0.03	0.03
query28	3.70	0.57	0.29
query29	12.42	3.17	2.56
query30	0.25	0.13	0.13
query31	2.76	0.38	0.17
query32	3.53	0.33	0.22
query33	1.45	1.50	1.42
query34	15.37	2.22	1.79
query35	1.75	1.74	1.74
query36	0.46	0.29	0.28
query37	0.07	0.03	0.03
query38	0.04	0.03	0.02
query39	0.03	0.02	0.03
query40	0.11	0.09	0.08
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.75 s
Total hot run time: 14.72 s

…al append The three array_agg_if blocks were manually appended to agg.out with an extra blank line, which made the block parser read one extra empty row for the preceding sql_array_agg_not_nullable block and fail the suite. Restore the single-blank-line block separator; the appended content itself matches the framework-generated output. Unit Test: - nereids_function_p0 agg_function suite passes in comparison mode

yujun777 · 2026-09-04T13:45:30Z

run buildall

hello-stephen · 2026-09-04T14:32:08Z

TPC-H: Total hot run time: 16833 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a91cff16057afbcc410f3de71f14445c3837017c, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17566	2978	3001	2978
q2	2108	258	230	230
q3	10212	858	527	527
q4	4673	250	199	199
q5	7684	592	396	396
q6	139	117	96	96
q7	549	504	394	394
q8	9318	929	885	885
q9	3477	2379	2382	2379
q10	6517	876	721	721
q11	398	205	187	187
q12	625	271	200	200
q13	18097	1521	1150	1150
q14	160	152	146	146
q15	q16	429	400	371	371
q17	1371	912	863	863
q18	3097	2265	2205	2205
q19	1263	923	722	722
q20	397	287	206	206
q21	5667	1753	1843	1753
q22	330	269	225	225
Total cold run time: 94077 ms
Total hot run time: 16833 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3363	3285	3301	3285
q2	499	393	392	392
q3	2321	2313	2114	2114
q4	1187	1147	882	882
q5	2218	2119	2115	2115
q6	167	123	88	88
q7	1001	921	866	866
q8	1604	1399	1403	1399
q9	3112	3075	3067	3067
q10	1849	1845	1650	1650
q11	350	266	250	250
q12	449	426	346	346
q13	1478	1524	1156	1156
q14	177	178	151	151
q15	q16	396	393	364	364
q17	3529	3228	3173	3173
q18	4758	4382	4682	4382
q19	885	806	942	806
q20	1014	951	826	826
q21	3795	3155	3239	3155
q22	396	336	325	325
Total cold run time: 34548 ms
Total hot run time: 30792 ms

hello-stephen · 2026-09-04T14:38:39Z

TPC-DS: Total hot run time: 83042 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a91cff16057afbcc410f3de71f14445c3837017c, data reload: false
query5	4230	410	331	331
query6	392	139	132	132
query7	4933	422	239	239
query8	289	133	121	121
query9	8683	2898	2876	2876
query10	386	220	184	184
query11	5359	1054	928	928
query12	115	70	72	70
query13	1183	458	340	340
query14	6117	2242	2121	2121
query14_1	2011	1998	2001	1998
query15	173	128	115	115
query16	909	327	350	327
query17	815	479	373	373
query18	2348	343	248	248
query19	173	145	117	117
query20	78	73	73	73
query21	205	105	87	87
query22	5500	5417	5459	5417
query23	7004	6345	6223	6223
query23_1	6277	6112	6067	6067
query24	7324	1099	796	796
query24_1	794	772	780	772
query25	440	312	267	267
query26	1227	237	133	133
query27	2777	419	265	265
query28	4659	1507	1509	1507
query29	949	456	361	361
query30	247	158	135	135
query31	828	410	342	342
query32	138	78	78	78
query33	474	232	188	188
query34	1021	862	507	507
query35	418	407	351	351
query36	595	575	495	495
query37	124	81	76	76
query38	1050	893	849	849
query39	482	485	480	480
query39_1	448	471	459	459
query40	212	103	82	82
query41	59	56	55	55
query42	76	74	74	74
query43	242	246	214	214
query44	1023	562	560	560
query45	111	115	106	106
query46	788	850	539	539
query47	775	769	724	724
query48	306	311	240	240
query49	538	246	193	193
query50	779	256	194	194
query51	8155	8276	7969	7969
query52	73	82	72	72
query53	199	198	154	154
query54	228	170	192	170
query55	78	64	59	59
query56	205	163	191	163
query57	702	645	640	640
query58	202	163	166	163
query59	1225	1229	1128	1128
query60	253	195	176	176
query61	136	149	118	118
query62	349	211	185	185
query63	190	154	142	142
query64	2832	696	590	590
query65	1700	1666	1650	1650
query66	1830	267	209	209
query67	9767	9825	9718	9718
query68	2794	1161	772	772
query69	361	247	205	205
query70	661	631	626	626
query71	249	182	174	174
query72	2306	1772	1542	1542
query73	649	590	324	324
query74	1572	1251	1157	1157
query75	1187	1110	978	978
query76	2297	743	531	531
query77	259	268	208	208
query78	4091	3782	3296	3296
query79	2240	818	599	599
query80	1567	343	282	282
query81	507	161	136	136
query82	633	136	96	96
query83	284	211	201	201
query84	294	112	93	93
query85	808	361	296	296
query86	402	175	160	160
query87	1069	987	925	925
query88	2783	2114	2107	2107
query89	292	200	180	180
query90	1987	138	130	130
query91	130	122	104	104
query92	83	73	72	72
query93	1399	1130	703	703
query94	657	246	213	213
query95	525	265	247	247
query96	807	575	279	279
query97	1098	1059	1003	1003
query98	146	141	135	135
query99	436	351	310	310
Total cold run time: 178229 ms
Total hot run time: 83042 ms

hello-stephen · 2026-09-04T14:43:01Z

ClickBench: Total hot run time: 14.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a91cff16057afbcc410f3de71f14445c3837017c, data reload: false
query1	0.01	0.00	0.01
query2	0.08	0.04	0.04
query3	0.25	0.11	0.10
query4	1.61	0.09	0.09
query5	0.17	0.15	0.17
query6	1.26	0.72	0.71
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.28	0.20	0.21
query10	0.34	0.33	0.36
query11	0.17	0.12	0.12
query12	0.15	0.12	0.12
query13	0.30	0.32	0.30
query14	0.46	0.44	0.45
query15	0.37	0.35	0.34
query16	0.20	0.22	0.22
query17	0.70	0.67	0.68
query18	0.18	0.17	0.15
query19	1.16	1.15	1.16
query20	0.01	0.02	0.02
query21	15.42	0.16	0.11
query22	5.08	0.04	0.05
query23	16.19	0.25	0.10
query24	3.07	0.31	0.26
query25	0.12	0.04	0.03
query26	0.73	0.17	0.12
query27	0.04	0.02	0.03
query28	3.67	0.55	0.28
query29	12.45	3.22	2.58
query30	0.26	0.12	0.12
query31	2.75	0.35	0.16
query32	3.55	0.32	0.23
query33	1.42	1.48	1.52
query34	15.36	2.18	1.80
query35	1.76	1.71	1.74
query36	0.46	0.28	0.30
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.08	0.07
query41	0.07	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.5 s
Total hot run time: 14.75 s

hello-stephen · 2026-09-04T15:29:56Z

FE UT Coverage Report

Increment line coverage 7.14% (1/14) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-04T19:06:17Z

BE Regression && UT Coverage Report

Increment line coverage 82.11% (202/246) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	65.69% (29856/45450)
Line Coverage	50.55% (322353/637639)
Region Coverage	46.34% (264908/571634)
Branch Coverage	47.53% (122109/256890)

hello-stephen · 2026-09-04T19:57:30Z

FE Regression Coverage Report

Increment line coverage 37.50% (12/32) 🎉
Increment coverage report
Complete coverage report

yujun777 · 2026-09-05T01:45:25Z

run cloud_p0

hello-stephen · 2026-09-05T04:11:57Z

BE Regression && UT Coverage Report

Increment line coverage 82.11% (202/246) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	65.69% (29856/45450)
Line Coverage	50.55% (322353/637639)
Region Coverage	46.34% (264908/571634)
Branch Coverage	47.53% (122109/256890)

yujun777 requested review from 924060929, englefly, morrySnow and starocean999 as code owners September 4, 2026 11:18

Uh oh!

Conversation

yujun777 commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

Uh oh!

yujun777 commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

Uh oh!

yujun777 commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

Uh oh!

hello-stephen commented Sep 4, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 4, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 4, 2026

FE Regression Coverage Report

Uh oh!

yujun777 commented Sep 5, 2026

Uh oh!

hello-stephen commented Sep 5, 2026

BE Regression && UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants