[refactor](nereids) agg strategy refactor by feiniaofeiafei · Pull Request #54079 · apache/doris

feiniaofeiafei · 2025-07-30T03:58:44Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Current Issues:

When the children of an agg satisfy a distribution, redundant PhysicalAggregates are generated.
count(distinct a, b) generates a three-stage gather: the first two stages deduplicate a and b, the third stage gathers the data, and then counts. (This modification avoids a direct gather by adding a local agg before the gather.)
The original code did not use statistical information to determine the different Aggregate strategies. This refactor incorporates statistical information as a basis for selection:
3.1 Utilizes statistical information in the logic for determining whether to use multi_distinct or CTE splitting: When the ndv of the group by key is low and the ndv of the distinct key is high, multi_distinct is advantageous.
3.2 In the single distinct scenario, statistical information is also used to determine whether to use a three-stage or four-stage Aggregate. When the ndv of the group by key is low, a four-stage Aggregate is used.

REWRITE PHASE

DistinctAggStrategySelector

Add the DistinctAggStrategySelector rule to handle scenarios with multiple distincts. Use statistical information to determine when to use CTE splitting or multidistinct.
After this rule is implemented, only single distinct scenarios need to be considered.

DistinctAggregateRewriter

Add the DistinctAggregateSplitter rule to handle scenarios with a single distinct. Determine whether to split or use multi-distinct. Distinguish between scenarios with and without group by.
No group by: Not processed here. With group by: If splitting is possible and statistical information indicates that multi-stage splitting is more optimal, split.

So, after this is complete, cases with distinct without group by:
No processing, move to the cascades phase.
Cases with distinct with group by:

distinct is rewritten to multi-distinct,
it is split, leaving distinct.
if it is neither rewritten to multi-distinct nor split, then splitting is performed directly into two, three, or four stages in the cascades phase.

CASCADES STAGE

SplitAgg

Add a SplitAgg rule to handle scenarios without distinct.

Split the Agg into two layers: local and global.
This allows you to avoid splitting the Agg into two stages, for example, in group_concat scenarios. (If not, shuffle first, then agg.)
This also implements a single-stage Agg.

SplitAggMultiPhaseWithoutGbyKey

ADD implementation rule: SplitAggMultiPhaseWithoutGbyKey
only process agg without group by key, and with one distinct function
provide there three rewrite:

selectcount(distinct a) from t
splitToThreePhase:
agg(count(a); distinct global)
+--gather+--agg(count(a); distinct local)+--agg(group by a; global)+--hashShuffle(a)
splitToFourPhase:
agg(count(a); distinct global)
+--gather+--agg(count(a); distinct local)+--agg(group by a; global)+--hashShuffle(a)+--agg(group by a; local)
twoPhaseAggregateWithFinalMultiDistinct:
agg(sum0(c1))
+--gather+--agg(multi_distinct(a) as c1)+--hashShuffle(a)

splitToThreePhase and twoPhaseAggregateWithFinalMultiDistinct provide plan when scan(or other plan) satisfy distributed expr

SplitAggMultiPhase

ADD implementation rule: SplitAggMultiPhase
only process agg with group by key, and with one distinct function
select count(distinct a) group by b (deduplicated agg hashShuffle by group by key b)

splitToTwoPlusOnePhase:
agg(group by b, count(a); distinct global)
+--agg(group by a,b; global)+--hashShuffle(b)+--agg(group by a,b; local)
agg(group by b, count(a); distinct global)
+--agg(group by a,b; global)+--hashShuffle(b)
splitToTwoPlusTwoPhase:
agg(group by b, count(a); distinct global)
+--hashShuffle(b)+--agg(group by b, count(a); distinct local)+--agg(group by a,b; global)+--hashShuffle(a,b)+--agg(group by a,b; local)
agg(group by b, count(a); distinct global)
+--hashShuffle(b)+--agg(group by b, count(a); distinct local)+--agg(group by a,b; global)+--hashShuffle(a,b)
splitToOnePlusTwoPhase: (deduplicated agg hashShuffle by distinct key a)
agg(group by b, count(a); distinct global)
+--hashShuffle(b)+--agg(group by b, count(a); distinct local)+--agg(group by a,b; global)+--hashShuffle(a)

The second plan of splitToTwoPlusOnePhase and splitToTwoPlusTwoPhase provide plan when scan(or other plan) satisfy group by distributed expr;
splitToOnePlusTwoPhase provide plan when scan(or other plan) satisfy distinct key distribution.

RequestPropertyDeriver

Implement visitPhysicalHashAggregate. The main logic is:
If the request received by the AGG is hash-distributed with the distribution column set S1, and the request sent by the AGG is also hash-distributed with the distribution column set S2,
and S = S2.intersect(S1), and the NDV of S1 is calculated, and if the NDV of S1 is found to be no less than a threshold, then the AGG sends the child a hash distribution based on S1.
Otherwise, the AGG sends the child a hash distribution based on S.
When directly generating a three-stage/four-stage AGG, the partitionby columns are specified. If the partitionby columns are not empty, the child is hash-distributed based on the partitionby columns.

ChildrenPropertiesRegulator

Disabled one-stage AGG: AGG-Distribute
Special handling for one-stage AGG with CTE:
AGG-Distribute-CTE.
If the Distribute option is Gather, the option is disabled.
If the Distribute option is HashShuffle, the option is determined based on statistics.
If no statistics are available, the option is enabled.
If statistics are available but the shuffle column is skewed, the option is disabled.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-07-30T03:58:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

feiniaofeiafei · 2025-07-30T08:29:46Z

run buildall

morrySnow · 2025-07-30T08:04:04Z

 }
 this.hboPlanStatisticsProvider = Objects.requireNonNull(Env.getCurrentEnv().getHboPlanStatisticsManager()
 .getHboPlanStatisticsProvider(), "HboPlanStatisticsProvider is null");
+ this.requestChildrenProperties = childrenProperties;


why need this?

morrySnow · 2025-07-30T08:10:57Z

 // 2. collect agg expressions and generate agg function to slot reference map
 List<Slot> aggFunctionOutput = Lists.newArrayList();
 ArrayList<FunctionCallExpr> execAggregateFunctions = Lists.newArrayListWithCapacity(outputExpressions.size());
+ boolean isPartial = aggregate.getAggregateParam().aggMode.productAggregateBuffer;


is this a bug in base code? maybe we should change the logic that generate aggMode

@feiniaofeiafei why aggMode isPartial but contains partial aggregation function

morrySnow · 2025-07-30T08:14:36Z

 if (curChildIndex == groupExpression.arity()) {
 if (!calculateEnforce(requestChildrenProperties, outputChildrenProperties)) {
- return; // if error exists, return
+ clear();


is a bug in base code too? we should fix it in a seperate PR and add some UT and regression case for it

morrySnow · 2025-07-30T08:31:04Z

+ // rewriteJobs.addAll(jobs(topic("split multi distinct",
+ // custom(RuleType.SPLIT_MULTI_DISTINCT, () -> SplitMultiDistinct.INSTANCE))));
+


redundant code?

morrySnow · 2025-07-30T10:57:44Z

- && children.get(0).getPlan() instanceof PhysicalDistribute) {
- return ImmutableList.of();
+ // 如果origin property 满足group by key, 但是不满足required, 那么禁用这个计划
+ PhysicalProperties originChildProperty = originChildrenProperties.get(0);


morrySnow · 2025-07-30T11:26:35Z

+ public static final List<Class<? extends AggregateFunction>> finalMultiDistinctSupportOtherFunc =
+ ImmutableList.of(Count.class, Sum.class, Min.class, Max.class);


morrySnow · 2025-07-30T11:32:13Z

+ * - Available memory resources
+ * - Query complexity
+ */
+public class DistinctAggStrategySelector extends DefaultPlanRewriter<DistinctSelectorContext>


morrySnow · 2025-07-30T11:32:24Z

+/**
+ * Split multi distinct strategy
+ * */
+public class SplitMultiDistinctStrategy {


feiniaofeiafei · 2025-07-31T06:43:30Z

run buildall

feiniaofeiafei · 2025-08-01T11:03:50Z

run buildall

doris-robot · 2025-08-01T11:25:29Z

TPC-H: Total hot run time: 34723 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9fee5b5f08a575771a8541c3cac9908580b6051, data reload: false
------ Round 1 ----------------------------------
q1	18121	5837	5502	5502
q2	1372	296	223	223
q3	8249	1307	713	713
q4	10183	1115	518	518
q5	7549	2314	2285	2285
q6	176	167	132	132
q7	913	751	610	610
q8	9286	1254	1036	1036
q9	6744	5044	5156	5044
q10	6907	2382	1977	1977
q11	472	273	274	273
q12	361	363	219	219
q13	17776	3524	3016	3016
q14	233	251	229	229
q15	550	469	460	460
q16	915	880	845	845
q17	565	803	363	363
q18	7549	7259	7183	7183
q19	3717	992	554	554
q20	321	310	222	222
q21	3360	3139	2296	2296
q22	1037	1093	1023	1023
Total cold run time: 106356 ms
Total hot run time: 34723 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6102	5844	5798	5798
q2	231	320	218	218
q3	2084	2551	2170	2170
q4	1299	1733	1382	1382
q5	4519	4485	4368	4368
q6	232	184	147	147
q7	2054	1991	1793	1793
q8	2602	2523	2599	2523
q9	7364	7421	7496	7421
q10	3110	3417	3272	3272
q11	561	526	506	506
q12	1169	1003	620	620
q13	3378	3736	3276	3276
q14	291	305	285	285
q15	497	475	462	462
q16	894	918	895	895
q17	1201	1507	1486	1486
q18	8220	7683	7807	7683
q19	5353	905	948	905
q20	3439	1873	1760	1760
q21	12093	4644	4245	4245
q22	1124	1059	955	955
Total cold run time: 67817 ms
Total hot run time: 52170 ms

doris-robot · 2025-08-01T11:37:06Z

TPC-DS: Total hot run time: 175102 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9fee5b5f08a575771a8541c3cac9908580b6051, data reload: false
reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL ============================================
query1	1507	365	364	364
query2	6810	1842	1700	1700
query3	3037	322	220	220
query4	23887	21524	21077	21077
query5	3121	649	519	519
query6	325	258	228	228
query7	4623	520	300	300
query8	294	244	229	229
query9	8574	3192	3205	3192
query10	448	340	303	303
query11	13545	13042	12717	12717
query12	173	131	119	119
query13	1657	540	402	402
query14	8237	7783	7727	7727
query15	206	186	171	171
query16	6700	665	482	482
query17	842	739	624	624
query18	1884	456	321	321
query19	219	203	201	201
query20	135	131	116	116
query21	216	123	109	109
query22	4005	4090	3879	3879
query23	32829	32435	32712	32435
query24	8157	2415	2416	2415
query25	488	500	448	448
query26	1233	292	167	167
query27	2587	495	346	346
query28	4427	2386	2378	2378
query29	707	593	511	511
query30	290	204	202	202
query31	877	804	726	726
query32	88	82	81	81
query33	query34	931	800	475	475
query35	804	863	736	736
query36	1025	1057	922	922
query37	142	117	98	98
query38	2872	2896	2864	2864
query39	895	868	829	829
query40	242	146	133	133
query41	64	60	58	58
query42	144	128	126	126
query43	515	500	472	472
query44	1333	833	833	833
query45	195	195	179	179
query46	940	1038	661	661
query47	1796	1858	1756	1756
query48	395	423	314	314
query49	744	503	423	423
query50	646	692	442	442
query51	3714	3722	3688	3688
query52	127	137	115	115
query53	259	285	206	206
query54	689	652	566	566
query55	92	91	83	83
query56	385	388	365	365
query57	1212	1207	1116	1116
query58	358	337	350	337
query59	2723	2690	2771	2690
query60	436	419	411	411
query61	130	132	116	116
query62	814	742	647	647
query63	257	225	212	212
query64	3931	1076	801	801
query65	3569	3516	3483	3483
query66	1095	459	377	377
query67	18184	18648	17706	17706
query68	14031	599	634	599
query69	1023	317	268	268
query70	1290	1124	1106	1106
query71	682	329	320	320
query72	9156	2287	2336	2287
query73	2971	627	347	347
query74	7197	6820	6634	6634
query75	6775	2600	2174	2174
query76	8497	1192	784	784
query77	1166	403	325	325
query78	8262	8523	8073	8073
query79	13207	581	541	541
query80	932	545	492	492
query81	552	242	210	210
query82	370	151	119	119
query83	355	280	262	262
query84	298	94	79	79
query85	816	373	331	331
query86	367	327	296	296
query87	3140	3101	3045	3045
query88	3687	2230	2218	2218
query89	454	355	315	315
query90	2603	235	229	229
query91	148	140	111	111
query92	92	71	67	67
query93	3587	963	661	661
query94	1002	388	277	277
query95	query96	504	589	283	283
query97	2261	2322	2204	2204
query98	234	217	206	206
query99	1399	1421	1290	1290
Total cold run time: 290844 ms
Total hot run time: 175102 ms

feiniaofeiafei · 2025-08-11T12:18:10Z

run buildall

doris-robot · 2025-08-11T12:42:33Z

TPC-H: Total hot run time: 34390 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68d260d06dbefeaa2f828f9cf998927d9f910934, data reload: false
------ Round 1 ----------------------------------
q1	17972	5145	5057	5057
q2	1311	290	185	185
q3	8349	1233	715	715
q4	10191	1018	524	524
q5	7545	2381	2271	2271
q6	181	160	130	130
q7	903	752	593	593
q8	9287	1295	1113	1113
q9	6750	5042	5083	5042
q10	6858	2383	1942	1942
q11	473	282	277	277
q12	346	348	229	229
q13	17766	3676	3059	3059
q14	235	236	226	226
q15	557	491	482	482
q16	833	832	786	786
q17	601	849	361	361
q18	7374	7306	7251	7251
q19	1227	941	570	570
q20	352	324	213	213
q21	4153	3227	2364	2364
q22	1126	1042	1000	1000
Total cold run time: 104390 ms
Total hot run time: 34390 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5125	5190	5140	5140
q2	230	322	218	218
q3	2199	2654	2274	2274
q4	1383	1781	1367	1367
q5	4233	4546	4577	4546
q6	214	168	127	127
q7	1985	2039	1846	1846
q8	2681	2557	2618	2557
q9	7374	7382	7267	7267
q10	3074	3264	2920	2920
q11	607	540	496	496
q12	682	796	606	606
q13	3484	3954	3343	3343
q14	316	319	277	277
q15	529	489	480	480
q16	879	934	894	894
q17	1195	1529	1426	1426
q18	7869	7865	7814	7814
q19	835	912	1073	912
q20	1869	1869	1749	1749
q21	4974	4812	4349	4349
q22	1091	1038	1013	1013
Total cold run time: 52828 ms
Total hot run time: 51621 ms

doris-robot · 2025-08-11T12:53:29Z

TPC-DS: Total hot run time: 173148 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68d260d06dbefeaa2f828f9cf998927d9f910934, data reload: false
query1	1482	353	363	353
query2	6816	1782	1747	1747
query3	3031	322	218	218
query4	23873	21472	21167	21167
query5	3128	620	485	485
query6	307	218	202	202
query7	4615	500	296	296
query8	268	237	219	219
query9	8596	2909	2912	2909
query10	487	336	268	268
query11	13775	13092	12828	12828
query12	166	106	100	100
query13	1657	533	432	432
query14	7427	5860	5797	5797
query15	214	194	174	174
query16	7727	647	498	498
query17	1200	718	595	595
query18	2008	420	328	328
query19	194	193	165	165
query20	115	106	105	105
query21	214	124	111	111
query22	4126	4191	3988	3988
query23	32572	31473	31430	31430
query24	8055	2334	2374	2334
query25	462	451	411	411
query26	1213	272	161	161
query27	2630	500	345	345
query28	4393	2292	2260	2260
query29	608	548	456	456
query30	265	212	171	171
query31	859	789	720	720
query32	90	78	73	73
query33	546	379	337	337
query34	724	782	461	461
query35	794	830	735	735
query36	961	1020	910	910
query37	131	112	87	87
query38	2938	2968	2973	2968
query39	917	868	856	856
query40	215	121	114	114
query41	60	56	57	56
query42	120	133	121	121
query43	491	483	499	483
query44	1275	813	817	813
query45	172	169	167	167
query46	843	1016	653	653
query47	1750	1784	1739	1739
query48	371	417	313	313
query49	714	486	392	392
query50	646	692	422	422
query51	3573	3651	3586	3586
query52	112	112	110	110
query53	236	253	190	190
query54	584	577	552	552
query55	89	86	87	86
query56	307	311	295	295
query57	1180	1182	1132	1132
query58	298	265	267	265
query59	2690	2757	2629	2629
query60	339	338	322	322
query61	123	120	121	120
query62	802	739	671	671
query63	224	192	188	188
query64	3295	1033	697	697
query65	3607	3549	3540	3540
query66	1121	459	318	318
query67	17281	16806	16900	16806
query68	9616	894	573	573
query69	517	316	294	294
query70	1137	1121	1143	1121
query71	392	316	313	313
query72	5513	4663	4824	4663
query73	635	620	344	344
query74	7239	6951	6826	6826
query75	2826	2539	2128	2128
query76	3401	1142	735	735
query77	616	386	319	319
query78	8315	8428	7740	7740
query79	2600	831	579	579
query80	668	536	462	462
query81	482	250	205	205
query82	228	142	114	114
query83	300	257	225	225
query84	287	99	87	87
query85	793	368	332	332
query86	358	304	292	292
query87	3144	3213	3068	3068
query88	3892	2242	2296	2242
query89	397	324	299	299
query90	1948	227	225	225
query91	141	141	114	114
query92	84	72	63	63
query93	2659	957	643	643
query94	681	411	300	300
query95	379	314	310	310
query96	488	574	280	280
query97	2224	2298	2266	2266
query98	237	208	204	204
query99	1319	1408	1302	1302
Total cold run time: 255618 ms
Total hot run time: 173148 ms

feiniaofeiafei · 2025-08-12T14:05:33Z

run buildall

doris-robot · 2025-08-12T14:23:29Z

TPC-H: Total hot run time: 34130 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6255feba8951592ec0463fa9707ab9475625696, data reload: false
------ Round 1 ----------------------------------
q1	18047	5042	5016	5016
q2	1328	270	191	191
q3	8333	1278	719	719
q4	10180	994	531	531
q5	7498	2393	2292	2292
q6	174	163	130	130
q7	908	759	611	611
q8	9316	1316	1099	1099
q9	6789	5023	5093	5023
q10	6868	2359	1941	1941
q11	464	291	261	261
q12	352	354	221	221
q13	17746	3667	3069	3069
q14	225	244	213	213
q15	545	478	488	478
q16	832	840	764	764
q17	587	853	360	360
q18	7569	7125	7213	7125
q19	1219	936	541	541
q20	335	319	212	212
q21	4009	3103	2340	2340
q22	1092	1011	993	993
Total cold run time: 104416 ms
Total hot run time: 34130 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5113	5096	5243	5096
q2	238	311	215	215
q3	2170	2681	2290	2290
q4	1345	1762	1298	1298
q5	4208	4515	4510	4510
q6	218	166	128	128
q7	1993	1954	1782	1782
q8	2608	2681	2589	2589
q9	7365	7320	7303	7303
q10	3144	3274	2955	2955
q11	596	557	511	511
q12	710	792	668	668
q13	3464	3865	3315	3315
q14	299	309	303	303
q15	549	522	490	490
q16	869	919	989	919
q17	1197	1560	1321	1321
q18	8152	7739	7527	7527
q19	813	816	896	816
q20	1817	1900	1787	1787
q21	5053	4609	4503	4503
q22	1098	1036	999	999
Total cold run time: 53019 ms
Total hot run time: 51325 ms

doris-robot · 2025-08-12T14:34:23Z

TPC-DS: Total hot run time: 172100 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6255feba8951592ec0463fa9707ab9475625696, data reload: false
query1	1479	354	368	354
query2	6816	1808	1735	1735
query3	3033	321	218	218
query4	24834	21435	20869	20869
query5	3210	651	483	483
query6	298	210	207	207
query7	4615	493	285	285
query8	278	237	226	226
query9	8554	2884	2891	2884
query10	472	368	285	285
query11	13598	12953	12745	12745
query12	149	110	99	99
query13	1673	538	425	425
query14	8005	5713	5687	5687
query15	204	179	174	174
query16	7889	646	474	474
query17	1511	710	569	569
query18	1982	404	311	311
query19	200	188	161	161
query20	108	139	101	101
query21	207	124	108	108
query22	4236	4459	4092	4092
query23	32312	31568	31485	31485
query24	8116	2368	2341	2341
query25	450	464	399	399
query26	1212	265	163	163
query27	2619	501	339	339
query28	4334	2294	2247	2247
query29	608	550	446	446
query30	269	202	168	168
query31	889	809	720	720
query32	86	72	76	72
query33	545	385	344	344
query34	736	785	466	466
query35	809	825	751	751
query36	958	1004	937	937
query37	129	118	87	87
query38	2945	2966	2905	2905
query39	916	844	872	844
query40	218	134	124	124
query41	65	63	62	62
query42	137	116	114	114
query43	505	509	483	483
query44	1289	807	802	802
query45	180	173	167	167
query46	840	1009	644	644
query47	1758	1801	1737	1737
query48	382	416	306	306
query49	696	469	386	386
query50	656	667	426	426
query51	3513	3525	3542	3525
query52	108	109	96	96
query53	238	258	191	191
query54	586	583	521	521
query55	91	80	88	80
query56	316	306	299	299
query57	1161	1197	1111	1111
query58	273	284	263	263
query59	2581	2825	2635	2635
query60	345	384	318	318
query61	124	121	122	121
query62	771	748	681	681
query63	218	184	192	184
query64	3250	1016	694	694
query65	3608	3535	3496	3496
query66	1044	405	311	311
query67	17258	16956	16661	16661
query68	8189	911	568	568
query69	538	319	279	279
query70	1205	1137	1110	1110
query71	391	362	297	297
query72	5429	4706	4966	4706
query73	663	667	346	346
query74	7181	6908	6794	6794
query75	2567	2556	2064	2064
query76	3097	1145	735	735
query77	508	395	314	314
query78	8311	8503	7880	7880
query79	2678	757	547	547
query80	648	544	464	464
query81	473	236	194	194
query82	191	137	114	114
query83	273	260	242	242
query84	257	102	82	82
query85	772	381	333	333
query86	343	303	287	287
query87	3146	3211	3096	3096
query88	3769	2148	2224	2148
query89	382	324	280	280
query90	2004	216	217	216
query91	136	141	109	109
query92	84	67	71	67
query93	2237	978	633	633
query94	688	388	312	312
query95	391	335	302	302
query96	480	570	268	268
query97	2256	2331	2177	2177
query98	230	196	196	196
query99	1370	1390	1296	1296
Total cold run time: 254382 ms
Total hot run time: 172100 ms

hello-stephen · 2025-08-12T15:10:54Z

FE UT Coverage Report

Increment line coverage 64.20% (486/757) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2025-08-13T07:11:07Z

run buildall

doris-robot · 2025-08-13T07:46:24Z

TPC-H: Total hot run time: 34388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 495d6c01e4625a3bb83403c0bfbc891393b0a5b0, data reload: false
------ Round 1 ----------------------------------
q1	18027	5086	5026	5026
q2	1302	270	180	180
q3	8375	1240	734	734
q4	10183	1013	515	515
q5	7493	2447	2366	2366
q6	177	163	129	129
q7	929	747	610	610
q8	9304	1471	1107	1107
q9	7029	5175	5131	5131
q10	6971	2363	1990	1990
q11	470	284	279	279
q12	348	344	219	219
q13	17753	3637	3032	3032
q14	241	232	220	220
q15	558	479	486	479
q16	845	830	792	792
q17	621	864	357	357
q18	7314	7246	7120	7120
q19	1092	969	540	540
q20	344	333	220	220
q21	3932	2532	2351	2351
q22	1102	1040	991	991
Total cold run time: 104410 ms
Total hot run time: 34388 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5158	5095	5131	5095
q2	229	315	215	215
q3	2188	2671	2291	2291
q4	1367	1763	1361	1361
q5	4230	4574	4551	4551
q6	205	169	132	132
q7	2043	1999	1809	1809
q8	2589	2603	2589	2589
q9	7223	7364	7409	7364
q10	3096	3375	2863	2863
q11	567	532	485	485
q12	663	778	631	631
q13	3681	3892	3243	3243
q14	292	299	288	288
q15	524	484	481	481
q16	883	895	846	846
q17	1242	1581	1364	1364
q18	8100	7767	7686	7686
q19	834	796	867	796
q20	2070	2192	1927	1927
q21	5105	4715	4310	4310
q22	1070	1054	1015	1015
Total cold run time: 53359 ms
Total hot run time: 51342 ms

doris-robot · 2025-08-13T07:58:01Z

TPC-DS: Total hot run time: 186091 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 495d6c01e4625a3bb83403c0bfbc891393b0a5b0, data reload: false
query1	1486	377	408	377
query2	6811	1772	1779	1772
query3	3037	323	212	212
query4	26090	23704	23022	23022
query5	2958	590	502	502
query6	311	219	207	207
query7	4609	505	303	303
query8	272	242	226	226
query9	8598	2842	2857	2842
query10	502	352	290	290
query11	15885	14916	15068	14916
query12	158	112	107	107
query13	1662	545	420	420
query14	10273	5673	5655	5655
query15	197	190	169	169
query16	7262	636	471	471
query17	1447	730	597	597
query18	2013	415	311	311
query19	195	188	168	168
query20	125	117	113	113
query21	208	130	108	108
query22	4251	4291	4275	4275
query23	34173	33240	33223	33223
query24	7669	2346	2354	2346
query25	469	462	409	409
query26	1210	267	162	162
query27	2628	496	355	355
query28	4332	2283	2231	2231
query29	623	574	442	442
query30	279	218	191	191
query31	844	811	701	701
query32	83	76	72	72
query33	531	382	330	330
query34	787	841	496	496
query35	777	802	737	737
query36	955	1009	937	937
query37	124	106	83	83
query38	3975	3996	3927	3927
query39	1459	1414	1421	1414
query40	220	129	120	120
query41	58	62	55	55
query42	120	107	108	107
query43	513	488	473	473
query44	1320	845	850	845
query45	173	187	166	166
query46	868	1008	631	631
query47	1785	1789	1718	1718
query48	380	417	311	311
query49	715	502	379	379
query50	638	696	384	384
query51	3863	3963	3888	3888
query52	106	106	103	103
query53	223	256	186	186
query54	608	586	559	559
query55	84	79	83	79
query56	308	309	301	301
query57	1155	1190	1126	1126
query58	277	276	269	269
query59	2615	2729	2593	2593
query60	342	331	324	324
query61	126	120	128	120
query62	786	715	660	660
query63	222	187	184	184
query64	3235	1030	706	706
query65	4089	3992	3986	3986
query66	1120	416	324	324
query67	17212	16993	16694	16694
query68	9570	907	572	572
query69	503	317	280	280
query70	1218	1157	1093	1093
query71	417	320	309	309
query72	5405	4681	4770	4681
query73	603	609	355	355
query74	8942	8991	8882	8882
query75	3626	2976	2509	2509
query76	3437	1171	748	748
query77	806	402	315	315
query78	9552	9727	8956	8956
query79	2165	831	581	581
query80	582	546	464	464
query81	476	253	222	222
query82	384	139	101	101
query83	247	250	232	232
query84	248	107	89	89
query85	784	378	363	363
query86	410	289	291	289
query87	4282	4227	4182	4182
query88	3515	2170	2223	2170
query89	392	320	278	278
query90	1937	223	211	211
query91	147	142	111	111
query92	83	70	67	67
query93	1629	961	639	639
query94	686	400	312	312
query95	400	308	307	307
query96	484	573	270	270
query97	2636	2684	2599	2599
query98	239	211	204	204
query99	1357	1396	1290	1290
Total cold run time: 269591 ms
Total hot run time: 186091 ms

feiniaofeiafei · 2025-08-13T12:56:44Z

run buildall

doris-robot · 2025-08-13T13:16:27Z

TPC-H: Total hot run time: 34455 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 936f1d9d1938d02794965e81984f31498400fe77, data reload: false
------ Round 1 ----------------------------------
q1	17582	5270	5136	5136
q2	1921	283	182	182
q3	10314	1256	707	707
q4	10219	1039	504	504
q5	7538	2509	2337	2337
q6	177	158	130	130
q7	912	737	593	593
q8	9295	1269	1093	1093
q9	6784	5119	5078	5078
q10	6939	2384	1984	1984
q11	491	289	269	269
q12	340	352	220	220
q13	17777	3661	3052	3052
q14	233	246	209	209
q15	558	513	486	486
q16	835	848	807	807
q17	625	841	384	384
q18	7483	7241	7097	7097
q19	1276	952	609	609
q20	347	334	222	222
q21	3777	2525	2375	2375
q22	1035	1038	981	981
Total cold run time: 106458 ms
Total hot run time: 34455 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5184	5144	5289	5144
q2	238	320	220	220
q3	2182	2667	2276	2276
q4	1344	1811	1345	1345
q5	4217	4605	4580	4580
q6	213	193	132	132
q7	1987	1969	1787	1787
q8	2677	2578	2536	2536
q9	7362	7321	7367	7321
q10	3080	3297	3117	3117
q11	570	524	522	522
q12	677	783	618	618
q13	3439	3911	3385	3385
q14	291	299	270	270
q15	523	513	471	471
q16	876	893	846	846
q17	1262	1565	1433	1433
q18	8034	7967	7599	7599
q19	849	869	869	869
q20	1988	1937	1785	1785
q21	4772	4343	4347	4343
q22	1053	1019	998	998
Total cold run time: 52818 ms
Total hot run time: 51597 ms

doris-robot · 2025-08-13T13:28:02Z

TPC-DS: Total hot run time: 186143 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 936f1d9d1938d02794965e81984f31498400fe77, data reload: false
query1	982	382	406	382
query2	6533	1811	1750	1750
query3	6739	214	212	212
query4	26131	23253	23119	23119
query5	4356	636	521	521
query6	297	212	217	212
query7	4639	499	291	291
query8	270	211	219	211
query9	8576	2833	2853	2833
query10	488	326	295	295
query11	15901	14997	14866	14866
query12	151	112	102	102
query13	1657	533	440	440
query14	8495	5769	5682	5682
query15	197	191	164	164
query16	7141	659	446	446
query17	1025	740	600	600
query18	1992	420	336	336
query19	195	191	166	166
query20	123	115	119	115
query21	221	137	115	115
query22	4131	4263	4029	4029
query23	34111	33365	33402	33365
query24	8172	2376	2393	2376
query25	619	466	412	412
query26	1235	267	160	160
query27	2757	495	339	339
query28	4363	2304	2254	2254
query29	761	548	446	446
query30	288	219	189	189
query31	872	796	727	727
query32	86	75	73	73
query33	568	372	332	332
query34	793	828	503	503
query35	825	844	735	735
query36	964	1022	920	920
query37	125	112	94	94
query38	4054	4092	4085	4085
query39	1476	1416	1413	1413
query40	217	123	114	114
query41	58	56	53	53
query42	116	115	109	109
query43	516	493	470	470
query44	1329	854	847	847
query45	175	175	172	172
query46	854	1008	638	638
query47	1764	1790	1704	1704
query48	372	407	318	318
query49	703	502	393	393
query50	640	693	387	387
query51	3980	3840	3789	3789
query52	119	112	108	108
query53	231	248	185	185
query54	590	583	536	536
query55	91	85	85	85
query56	305	319	301	301
query57	1214	1189	1111	1111
query58	268	263	260	260
query59	2626	2737	2619	2619
query60	336	339	319	319
query61	129	123	127	123
query62	820	741	678	678
query63	233	201	187	187
query64	4294	1004	699	699
query65	4135	4025	3978	3978
query66	1158	411	329	329
query67	17126	17239	16821	16821
query68	7930	903	561	561
query69	478	320	293	293
query70	1265	1139	1139	1139
query71	455	318	304	304
query72	5393	4787	4795	4787
query73	693	625	365	365
query74	9150	9087	8616	8616
query75	3649	2952	2508	2508
query76	3644	1141	770	770
query77	799	393	342	342
query78	9587	9746	8859	8859
query79	2383	813	590	590
query80	648	525	450	450
query81	495	252	217	217
query82	211	133	104	104
query83	260	247	234	234
query84	285	100	86	86
query85	779	455	345	345
query86	384	308	305	305
query87	4288	4224	4127	4127
query88	3235	2219	2208	2208
query89	452	312	281	281
query90	1978	222	211	211
query91	143	137	110	110
query92	84	68	71	68
query93	2461	973	637	637
query94	620	417	306	306
query95	400	316	306	306
query96	491	571	278	278
query97	2621	2637	2592	2592
query98	251	218	205	205
query99	1329	1393	1296	1296
Total cold run time: 273549 ms
Total hot run time: 186143 ms

doris-robot · 2025-08-13T13:33:29Z

ClickBench: Total hot run time: 32.94 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 936f1d9d1938d02794965e81984f31498400fe77, data reload: false
query1	0.03	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.10
query5	0.26	0.25	0.26
query6	1.16	0.65	0.63
query7	0.03	0.02	0.02
query8	0.05	0.03	0.03
query9	0.46	0.40	0.38
query10	0.63	0.62	0.61
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.60	0.61
query14	1.03	1.01	1.04
query15	0.86	0.88	0.88
query16	0.38	0.40	0.39
query17	1.06	1.04	1.05
query18	0.21	0.19	0.20
query19	1.90	1.82	1.82
query20	0.01	0.01	0.02
query21	15.41	0.94	0.56
query22	0.77	1.22	0.84
query23	14.75	1.40	0.61
query24	6.64	1.18	1.09
query25	0.48	0.25	0.12
query26	0.59	0.16	0.13
query27	0.05	0.05	0.06
query28	9.53	0.92	0.43
query29	12.53	3.96	3.28
query30	3.07	3.04	2.96
query31	2.84	0.56	0.38
query32	3.25	0.55	0.47
query33	3.07	3.12	3.07
query34	15.95	5.44	4.91
query35	4.88	4.94	4.94
query36	0.71	0.52	0.50
query37	0.10	0.07	0.07
query38	0.06	0.05	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 105.9 s
Total hot run time: 32.94 s

hello-stephen · 2025-08-13T13:58:57Z

FE UT Coverage Report

Increment line coverage 60.98% (461/756) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2025-08-14T12:29:10Z

run buildall

doris-robot · 2025-08-14T13:01:03Z

TPC-H: Total hot run time: 34269 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 152774b4c0af1a6793209e35f61410ab57426fbc, data reload: false
------ Round 1 ----------------------------------
q1	17561	5293	5055	5055
q2	1935	309	184	184
q3	10280	1264	741	741
q4	10210	1006	526	526
q5	7505	2341	2313	2313
q6	177	159	130	130
q7	900	743	596	596
q8	9299	1283	1093	1093
q9	6906	5105	5030	5030
q10	6954	2396	2005	2005
q11	478	286	269	269
q12	346	341	226	226
q13	17775	3639	3043	3043
q14	229	239	224	224
q15	550	479	490	479
q16	819	830	761	761
q17	610	847	358	358
q18	7336	7101	7106	7101
q19	1090	935	554	554
q20	346	326	217	217
q21	4110	3143	2383	2383
q22	1101	981	985	981
Total cold run time: 106517 ms
Total hot run time: 34269 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5131	5086	5092	5086
q2	243	322	224	224
q3	2182	2632	2312	2312
q4	1361	1782	1311	1311
q5	4180	4581	4411	4411
q6	224	174	132	132
q7	2035	1998	1800	1800
q8	2668	2472	2476	2472
q9	7341	7145	7220	7145
q10	3300	3294	2963	2963
q11	586	522	522	522
q12	678	750	601	601
q13	3487	4177	3500	3500
q14	278	294	289	289
q15	509	477	485	477
q16	863	892	864	864
q17	1252	1573	1600	1573
q18	7638	7711	7478	7478
q19	836	866	861	861
q20	2091	2051	1975	1975
q21	4590	4312	4189	4189
q22	1107	1028	1006	1006
Total cold run time: 52580 ms
Total hot run time: 51191 ms

doris-robot · 2025-08-14T13:12:39Z

TPC-DS: Total hot run time: 186447 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 152774b4c0af1a6793209e35f61410ab57426fbc, data reload: false
query1	997	377	430	377
query2	6534	1734	1746	1734
query3	6744	222	211	211
query4	26074	23783	23123	23123
query5	4366	598	493	493
query6	305	223	221	221
query7	4628	504	290	290
query8	276	234	230	230
query9	8597	2881	2855	2855
query10	469	337	274	274
query11	15944	15020	14789	14789
query12	157	116	104	104
query13	1669	544	416	416
query14	8564	5708	5733	5708
query15	200	192	175	175
query16	7345	659	492	492
query17	1207	718	589	589
query18	1994	425	328	328
query19	192	197	186	186
query20	116	113	123	113
query21	216	129	114	114
query22	4200	4302	4045	4045
query23	34055	33318	33264	33264
query24	8073	2343	2337	2337
query25	545	479	409	409
query26	1234	272	153	153
query27	2748	515	343	343
query28	4409	2278	2244	2244
query29	788	547	460	460
query30	280	220	185	185
query31	887	778	717	717
query32	81	70	74	70
query33	558	376	334	334
query34	778	823	521	521
query35	798	844	755	755
query36	966	1016	923	923
query37	117	104	86	86
query38	4114	4023	4016	4016
query39	1487	1418	1407	1407
query40	220	124	116	116
query41	60	55	54	54
query42	120	108	112	108
query43	510	488	464	464
query44	1369	859	866	859
query45	172	172	166	166
query46	859	1010	646	646
query47	1764	1796	1720	1720
query48	372	409	304	304
query49	716	485	378	378
query50	638	687	395	395
query51	3855	3839	3847	3839
query52	107	107	99	99
query53	234	270	190	190
query54	592	579	520	520
query55	84	90	90	90
query56	310	303	322	303
query57	1181	1190	1115	1115
query58	304	274	288	274
query59	2617	2772	2525	2525
query60	344	341	326	326
query61	126	121	122	121
query62	828	718	672	672
query63	222	190	194	190
query64	4398	991	731	731
query65	4080	4019	4015	4015
query66	1169	423	380	380
query67	17226	16991	16918	16918
query68	8027	884	564	564
query69	474	328	289	289
query70	1224	1116	1108	1108
query71	467	332	321	321
query72	5359	4792	4819	4792
query73	689	618	353	353
query74	9039	9021	8997	8997
query75	3612	3011	2535	2535
query76	3512	1130	745	745
query77	818	395	320	320
query78	9595	9764	8849	8849
query79	2485	791	590	590
query80	593	535	461	461
query81	482	253	221	221
query82	474	137	102	102
query83	244	254	232	232
query84	254	104	80	80
query85	786	370	335	335
query86	394	322	309	309
query87	4326	4212	4157	4157
query88	3629	2283	2233	2233
query89	373	310	298	298
query90	1839	216	217	216
query91	141	142	111	111
query92	89	69	67	67
query93	1915	1004	639	639
query94	667	399	295	295
query95	403	318	304	304
query96	476	573	275	275
query97	2663	2658	2555	2555
query98	238	219	213	213
query99	1427	1379	1272	1272
Total cold run time: 273903 ms
Total hot run time: 186447 ms

doris-robot · 2025-08-14T13:18:08Z

ClickBench: Total hot run time: 32.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 152774b4c0af1a6793209e35f61410ab57426fbc, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.05
query3	0.24	0.08	0.07
query4	1.62	0.11	0.11
query5	0.29	0.26	0.24
query6	1.19	0.64	0.63
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.52	0.50
query10	0.63	0.63	0.63
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.64	0.60	0.61
query14	1.03	1.02	1.04
query15	0.86	0.84	0.85
query16	0.41	0.39	0.38
query17	1.01	1.06	1.07
query18	0.22	0.19	0.20
query19	1.92	1.81	1.86
query20	0.01	0.02	0.02
query21	15.49	0.89	0.55
query22	0.80	1.11	0.76
query23	14.86	1.36	0.66
query24	6.18	1.25	0.81
query25	0.50	0.18	0.14
query26	0.56	0.15	0.13
query27	0.07	0.06	0.05
query28	9.80	0.93	0.41
query29	12.57	3.91	3.23
query30	3.04	3.02	2.99
query31	2.82	0.59	0.37
query32	3.24	0.57	0.49
query33	3.06	3.04	3.16
query34	16.10	5.44	4.82
query35	4.91	4.88	4.86
query36	0.71	0.51	0.49
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.15	0.13
query41	0.09	0.02	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.38 s
Total hot run time: 32.57 s

hello-stephen · 2025-08-14T13:35:11Z

FE UT Coverage Report

Increment line coverage 61.70% (480/778) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2025-09-05T02:19:29Z

run buildall

doris-robot · 2025-09-05T02:39:16Z

TPC-H: Total hot run time: 34763 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit df8951b2d9aade1bc934d4d79887aa126b64c0ae, data reload: false
------ Round 1 ----------------------------------
q1	17615	5242	5118	5118
q2	2017	344	207	207
q3	10224	1291	739	739
q4	10235	1023	554	554
q5	7597	2502	2380	2380
q6	188	170	137	137
q7	918	798	623	623
q8	9346	1374	1084	1084
q9	7156	5129	5463	5129
q10	6918	2427	2001	2001
q11	499	313	280	280
q12	350	363	216	216
q13	17781	3706	3007	3007
q14	230	241	209	209
q15	583	490	501	490
q16	985	1012	957	957
q17	603	874	356	356
q18	7500	7086	7153	7086
q19	1237	992	583	583
q20	354	339	240	240
q21	3799	2528	2376	2376
q22	1098	1053	991	991
Total cold run time: 107233 ms
Total hot run time: 34763 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5198	5052	5142	5052
q2	260	334	227	227
q3	2205	2710	2309	2309
q4	1357	1792	1336	1336
q5	4216	4509	4702	4509
q6	226	177	134	134
q7	2154	1955	1844	1844
q8	2711	2702	2809	2702
q9	7417	7506	7434	7434
q10	3178	3320	2888	2888
q11	576	517	517	517
q12	727	783	672	672
q13	3553	3991	3418	3418
q14	297	298	286	286
q15	536	488	474	474
q16	1065	1124	1053	1053
q17	1205	1596	1419	1419
q18	7837	7644	7294	7294
q19	787	804	920	804
q20	1930	1998	1808	1808
q21	4725	4428	4390	4390
q22	1077	1057	1022	1022
Total cold run time: 53237 ms
Total hot run time: 51592 ms

doris-robot · 2025-09-05T02:51:05Z

TPC-DS: Total hot run time: 190027 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit df8951b2d9aade1bc934d4d79887aa126b64c0ae, data reload: false
query1	1057	448	403	403
query2	6561	1722	1761	1722
query3	6754	231	228	228
query4	26413	23276	23393	23276
query5	4495	672	506	506
query6	346	260	235	235
query7	4655	517	309	309
query8	316	282	248	248
query9	8643	2970	2914	2914
query10	517	366	307	307
query11	15772	15128	15227	15128
query12	194	127	125	125
query13	1738	605	456	456
query14	11681	9179	9215	9179
query15	220	196	186	186
query16	7693	674	505	505
query17	1347	777	647	647
query18	2058	447	373	373
query19	264	201	172	172
query20	141	124	120	120
query21	217	136	117	117
query22	4144	4212	4016	4016
query23	34133	33120	33054	33054
query24	8129	2405	2426	2405
query25	566	505	446	446
query26	1249	288	171	171
query27	2689	518	378	378
query28	4332	2342	2302	2302
query29	795	617	498	498
query30	297	226	197	197
query31	910	820	741	741
query32	92	83	78	78
query33	601	412	371	371
query34	805	866	514	514
query35	821	834	763	763
query36	962	1023	937	937
query37	129	117	96	96
query38	3532	3534	3522	3522
query39	1526	1460	1418	1418
query40	232	132	128	128
query41	63	69	63	63
query42	132	119	122	119
query43	520	525	493	493
query44	1353	872	869	869
query45	185	181	174	174
query46	862	1113	670	670
query47	1806	1817	1778	1778
query48	391	446	348	348
query49	747	519	410	410
query50	662	696	415	415
query51	3984	4010	3841	3841
query52	121	120	108	108
query53	249	268	212	212
query54	613	615	555	555
query55	103	90	93	90
query56	370	332	335	332
query57	1186	1198	1153	1153
query58	309	290	285	285
query59	2609	2663	2527	2527
query60	360	356	348	348
query61	197	160	163	160
query62	823	750	677	677
query63	234	200	200	200
query64	4420	1150	826	826
query65	4052	3954	3971	3954
query66	1092	447	349	349
query67	15794	15481	15139	15139
query68	9306	944	596	596
query69	489	341	299	299
query70	1413	1348	1280	1280
query71	595	360	328	328
query72	5605	4938	4815	4815
query73	718	588	364	364
query74	8991	9071	9008	9008
query75	4479	3291	2822	2822
query76	3721	1174	754	754
query77	828	409	337	337
query78	9789	9759	8884	8884
query79	2777	818	589	589
query80	678	585	544	544
query81	484	264	227	227
query82	466	160	137	137
query83	288	266	255	255
query84	307	124	104	104
query85	885	474	511	474
query86	352	317	323	317
query87	3770	3718	3650	3650
query88	3165	2246	2247	2246
query89	418	324	304	304
query90	1966	244	226	226
query91	165	170	137	137
query92	93	77	76	76
query93	1265	1013	658	658
query94	694	422	337	337
query95	419	336	325	325
query96	493	593	280	280
query97	2962	3020	2872	2872
query98	260	229	220	220
query99	1487	1412	1276	1276
Total cold run time: 279591 ms
Total hot run time: 190027 ms

doris-robot · 2025-09-05T02:56:46Z

ClickBench: Total hot run time: 32.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit df8951b2d9aade1bc934d4d79887aa126b64c0ae, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.06	0.06
query3	0.25	0.09	0.08
query4	1.61	0.12	0.12
query5	0.27	0.27	0.26
query6	1.17	0.66	0.66
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.62	0.53	0.54
query10	0.58	0.57	0.58
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.63	0.64	0.62
query14	1.05	1.03	1.03
query15	0.91	0.86	0.86
query16	0.41	0.40	0.40
query17	1.06	1.04	1.13
query18	0.21	0.20	0.20
query19	1.92	1.90	1.83
query20	0.01	0.01	0.02
query21	15.42	0.96	0.58
query22	0.75	1.26	0.73
query23	14.77	1.41	0.64
query24	7.25	1.21	0.67
query25	0.49	0.32	0.22
query26	0.72	0.16	0.13
query27	0.06	0.06	0.05
query28	9.72	0.97	0.43
query29	12.66	3.91	3.29
query30	3.11	3.05	2.96
query31	2.83	0.61	0.40
query32	3.25	0.57	0.48
query33	3.16	3.08	3.15
query34	16.07	5.48	4.90
query35	4.96	4.90	4.95
query36	0.68	0.52	0.50
query37	0.11	0.08	0.08
query38	0.07	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.04
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 107.71 s
Total hot run time: 32.95 s

hello-stephen · 2025-09-05T03:36:29Z

FE UT Coverage Report

Increment line coverage 78.01% (628/805) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-09-05T06:20:16Z

FE Regression Coverage Report

Increment line coverage 89.07% (717/805) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-09-05T07:08:35Z

PR approved by at least one committer and no changes requested.

morrySnow · 2025-09-05T16:27:52Z

+ default Statistics getStats() {
+ throw new IllegalStateException("Not support getStats for " + getClass().getName());
+ }


why add a default impl?

### What problem does this PR solve? #### Current Issues: 1. When the children of an agg satisfy a distribution, redundant PhysicalAggregates are generated. 2. count(distinct a, b) generates a three-stage gather: the first two stages deduplicate a and b, the third stage gathers the data, and then counts. (This modification avoids a direct gather by adding a local agg before the gather.) 3. The original code did not use statistical information to determine the different Aggregate strategies. This refactor incorporates statistical information as a basis for selection: 3.1 Utilizes statistical information in the logic for determining whether to use multi_distinct or CTE splitting: When the ndv of the group by key is low and the ndv of the distinct key is high, multi_distinct is advantageous. 3.2 In the single distinct scenario, statistical information is also used to determine whether to use a three-stage or four-stage Aggregate. When the ndv of the group by key is low, a four-stage Aggregate is used. #### REWRITE PHASE ##### DistinctAggStrategySelector Add the DistinctAggStrategySelector rule to handle scenarios with multiple distincts. Use statistical information to determine when to use CTE splitting or multidistinct. After this rule is implemented, only single distinct scenarios need to be considered. ##### DistinctAggregateRewriter Add the DistinctAggregateSplitter rule to handle scenarios with a single distinct. Determine whether to split or use multi-distinct. Distinguish between scenarios with and without group by. No group by: Not processed here. With group by: If splitting is possible and statistical information indicates that multi-stage splitting is more optimal, split. So, after this is complete, cases with distinct without group by: No processing, move to the cascades phase. Cases with distinct with group by: 1. distinct is rewritten to multi-distinct, 2. it is split, leaving distinct. 3. if it is neither rewritten to multi-distinct nor split, then splitting is performed directly into two, three, or four stages in the cascades phase. #### CASCADES STAGE ##### SplitAgg Add a SplitAgg rule to handle scenarios without distinct. 1. Split the Agg into two layers: local and global. This allows you to avoid splitting the Agg into two stages, for example, in group_concat scenarios. (If not, shuffle first, then agg.) 2. This also implements a single-stage Agg. ##### SplitAggMultiPhaseWithoutGbyKey ADD implementation rule: SplitAggMultiPhaseWithoutGbyKey only process agg without group by key, and with one distinct function provide there three rewrite: ```sql select count(distinct a) from t splitToThreePhase: agg(count(a); distinct global) +--gather +--agg(count(a); distinct local) +--agg(group by a; global) +--hashShuffle(a) splitToFourPhase: agg(count(a); distinct global) +--gather +--agg(count(a); distinct local) +--agg(group by a; global) +--hashShuffle(a) +--agg(group by a; local) twoPhaseAggregateWithFinalMultiDistinct: agg(sum0(c1)) +--gather +--agg(multi_distinct(a) as c1) +--hashShuffle(a) ``` splitToThreePhase and twoPhaseAggregateWithFinalMultiDistinct provide plan when scan(or other plan) satisfy distributed expr ##### SplitAggMultiPhase ADD implementation rule: SplitAggMultiPhase only process agg with group by key, and with one distinct function select count(distinct a) group by b (deduplicated agg hashShuffle by group by key b) ```sql splitToTwoPlusOnePhase: agg(group by b, count(a); distinct global) +--agg(group by a,b; global) +--hashShuffle(b) +--agg(group by a,b; local) agg(group by b, count(a); distinct global) +--agg(group by a,b; global) +--hashShuffle(b) splitToTwoPlusTwoPhase: agg(group by b, count(a); distinct global) +--hashShuffle(b) +--agg(group by b, count(a); distinct local) +--agg(group by a,b; global) +--hashShuffle(a,b) +--agg(group by a,b; local) agg(group by b, count(a); distinct global) +--hashShuffle(b) +--agg(group by b, count(a); distinct local) +--agg(group by a,b; global) +--hashShuffle(a,b) splitToOnePlusTwoPhase: (deduplicated agg hashShuffle by distinct key a) agg(group by b, count(a); distinct global) +--hashShuffle(b) +--agg(group by b, count(a); distinct local) +--agg(group by a,b; global) +--hashShuffle(a) ``` The second plan of splitToTwoPlusOnePhase and splitToTwoPlusTwoPhase provide plan when scan(or other plan) satisfy group by distributed expr; splitToOnePlusTwoPhase provide plan when scan(or other plan) satisfy distinct key distribution. #### RequestPropertyDeriver Implement visitPhysicalHashAggregate. The main logic is: If the request received by the AGG is hash-distributed with the distribution column set S1, and the request sent by the AGG is also hash-distributed with the distribution column set S2, and S = S2.intersect(S1), and the NDV of S1 is calculated, and if the NDV of S1 is found to be no less than a threshold, then the AGG sends the child a hash distribution based on S1. Otherwise, the AGG sends the child a hash distribution based on S. When directly generating a three-stage/four-stage AGG, the partitionby columns are specified. If the partitionby columns are not empty, the child is hash-distributed based on the partitionby columns. #### ChildrenPropertiesRegulator Disabled one-stage AGG: AGG-Distribute Special handling for one-stage AGG with CTE: AGG-Distribute-CTE. If the Distribute option is Gather, the option is disabled. If the Distribute option is HashShuffle, the option is determined based on statistics. If no statistics are available, the option is enabled. If statistics are available but the shuffle column is skewed, the option is disabled.

… have statistics (#55855) Related PR: #54079 Problem Summary: Using the multi-distinct strategy instead of multi-stage/CTE splitting strategy for AGG in scenarios without statistical information, consistent with the default strategy before refactor(#54079), can achieve better performance in scenarios without statistical information.

…epeat (#55907) Related PR: #54079 Problem Summary: There may be some issues in scenarios where the grouping() function is present. When merging a project, ```sql -- Upper level project output: projections : "GROUPING_PREFIX_publish_date#36 originExpression=Grouping(publish_date#6) AS `dim_207`#32" --Lower level project output: projections : "GROUP PING_PREVIX_publish_date # 79 AS ` GROUP PING_PREVIX_publish_date ` # 36" ``` In theory, it should be rewritten as `GROUPPING_PREVIX_publish_date# 79 AS dim_207# 32`, but there is a problem here: The map generated by the lower level project is: key : `GROUPING_PREFIX_publish_date#36 , value: GROUPING_PREFIX_publish_date#79 ` Dealing with upper level projects `GROUPING_PREFIX_publish_date#36 originExpression=Grouping(publish_date#6) AS dim_207#32` In the upper level project, the 'virtual slot' is used, and the 'key' in the map is' slot '. Although the ExprId is the same, it cannot be hit, resulting in an error. So temporarily disable the use of CTE splitting to handle grouping+multiple counts (distinctions) scenarios.

…d count distinct multi expr exists same time (#56271) ### What problem does this PR solve? Related PR: #54079 Problem Summary: 1. Added a check for multi_distinct_count(a,b) in the MultiDistinctCount constructor to prevent the use of multiple columns. Because BE doesn't report an error in this case, it only uses the first argument of the multi_distinct function, resulting in incorrect results. 2. In scenarios without a group by key, when multi_distinct_func and count(distinct a,b) appear together, the original code converts count(distinct a,b) to multi_distinct_count(a), resulting in incorrect results. The correct approach is to use multi-stage splitting when count distinct multi_expr appears. 3. Removed the mustUseMultiDistinct flag. This flag is useless.

### What problem does this PR solve? Related PR: #54079 Problem Summary: NormalizeAggregate treated reordered equivalent multi-column COUNT DISTINCT expressions as different aggregate functions, which could make the multi-distinct strategy construct a join from only one aggregate and fail with an index-out-of-bounds exception. It also retained duplicate arguments even though downstream COUNT DISTINCT processing treats them as a set, and the CountIf conversion indexed the original argument list using the deduplicated size. Canonicalize multi-column distinct arguments as an ordered set during normalization and make CountIf consume that deduplicated list directly. ### Release note Fix planning and execution of equivalent multi-column COUNT DISTINCT expressions with reordered or duplicate arguments.

…65206) ### What problem does this PR solve? Related PR: apache#54079 Problem Summary: NormalizeAggregate treated reordered equivalent multi-column COUNT DISTINCT expressions as different aggregate functions, which could make the multi-distinct strategy construct a join from only one aggregate and fail with an index-out-of-bounds exception. It also retained duplicate arguments even though downstream COUNT DISTINCT processing treats them as a set, and the CountIf conversion indexed the original argument list using the deduplicated size. Canonicalize multi-column distinct arguments as an ordered set during normalization and make CountIf consume that deduplicated list directly. ### Release note Fix planning and execution of equivalent multi-column COUNT DISTINCT expressions with reordered or duplicate arguments.

morrySnow marked this pull request as draft July 30, 2025 08:03

morrySnow reviewed Jul 30, 2025
View reviewed changes

feiniaofeiafei force-pushed the agg_strategy branch from cd9c2fc to f9fee5bCompare August 1, 2025 11:03

feiniaofeiafei force-pushed the agg_strategy branch from 68d260d to a6255feCompare August 12, 2025 14:04

feiniaofeiafei dismissed 924060929’s stale review via 4e330ccSeptember 5, 2025 02:18

feiniaofeiafei force-pushed the agg_strategy branch from 6b77923 to 4e330ccCompare September 5, 2025 02:18

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 5, 2025

add code
df8951b

feiniaofeiafei force-pushed the agg_strategy branch from 4e330cc to df8951bCompare September 5, 2025 02:19

924060929 approved these changes Sep 5, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2025

morrySnow approved these changes Sep 5, 2025
View reviewed changes

morrySnow merged commit db41a84 into apache:masterSep 5, 2025
26 of 28 checks passed

This was referenced Sep 11, 2025
[fix](nereids) not use cte split multi distinct when agg has source repeat #55907
Merged
[fix](nereids) change default agg strategy to multi_distinct when not have statistics #55855
Merged

feiniaofeiafei mentioned this pull request Sep 22, 2025
[fix] (agg_strategy) fix result wrong when the multi_distinct_func and count distinct multi expr exists same time #56271
Merged
16 tasks

morrySnow added dev/4.1.x and removed dev/4.1.x labels Mar 12, 2026

feiniaofeiafei mentioned this pull request Jul 8, 2026
[fix](agg) Normalize equivalent multi-column distinct counts #65206
Merged
16 tasks

		// rewriteJobs.addAll(jobs(topic("split multi distinct",
		// custom(RuleType.SPLIT_MULTI_DISTINCT, () -> SplitMultiDistinct.INSTANCE))));

		public static final List<Class<? extends AggregateFunction>> finalMultiDistinctSupportOtherFunc =
		ImmutableList.of(Count.class, Sum.class, Min.class, Max.class);

Uh oh!

Conversation

feiniaofeiafei commented Jul 30, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Current Issues:

REWRITE PHASE

DistinctAggStrategySelector

DistinctAggregateRewriter

CASCADES STAGE

SplitAgg

SplitAggMultiPhaseWithoutGbyKey

SplitAggMultiPhase

RequestPropertyDeriver

ChildrenPropertiesRegulator

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 30, 2025

Uh oh!

feiniaofeiafei commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

feiniaofeiafei commented Jul 31, 2025

Uh oh!

feiniaofeiafei commented Aug 1, 2025

Uh oh!

doris-robot commented Aug 1, 2025

Uh oh!

doris-robot commented Aug 1, 2025

Uh oh!

feiniaofeiafei commented Aug 11, 2025

Uh oh!

doris-robot commented Aug 11, 2025

Uh oh!

doris-robot commented Aug 11, 2025

Uh oh!

feiniaofeiafei commented Aug 12, 2025

Uh oh!

doris-robot commented Aug 12, 2025

Uh oh!

doris-robot commented Aug 12, 2025

Uh oh!

hello-stephen commented Aug 12, 2025

FE UT Coverage Report

Uh oh!

feiniaofeiafei commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

feiniaofeiafei commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

feiniaofeiafei commented Jul 30, 2025•
edited
Loading