[feature](agg) add variable agg_shuffle_use_parent_key to control the shuffle key choosing by feiniaofeiafei · Pull Request #59876 · apache/doris

feiniaofeiafei · 2026-01-14T09:25:40Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
add var agg_shuffle_use_parent_key, default value is true, when set false, can make agg shuffle by all gby key instead of parent shuffle key.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-01-14T09:25:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

feiniaofeiafei · 2026-01-14T09:26:26Z

run buildall

doris-robot · 2026-01-14T09:58:43Z

TPC-H: Total hot run time: 32020 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b33ba3094c40dc4d99ed2ffefd51843849a03f2a, data reload: false
------ Round 1 ----------------------------------
q1	17615	4312	4053	4053
q2	2050	354	236	236
q3	10152	1293	729	729
q4	10228	885	304	304
q5	7567	2096	1839	1839
q6	184	177	145	145
q7	919	803	662	662
q8	9286	1382	1141	1141
q9	4892	4738	4547	4547
q10	6731	1781	1380	1380
q11	548	306	286	286
q12	703	742	608	608
q13	17771	3873	3077	3077
q14	293	293	272	272
q15	566	507	509	507
q16	685	703	650	650
q17	661	852	451	451
q18	6635	6470	6913	6470
q19	1295	1035	717	717
q20	421	379	275	275
q21	3319	2646	2686	2646
q22	1126	1096	1025	1025
Total cold run time: 103647 ms
Total hot run time: 32020 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4493	4246	4191	4191
q2	341	427	336	336
q3	2425	2697	2396	2396
q4	1440	1923	1489	1489
q5	4484	4316	4418	4316
q6	214	167	133	133
q7	2016	1921	1732	1732
q8	2529	2573	2392	2392
q9	7067	7376	7230	7230
q10	2558	2743	2306	2306
q11	581	491	454	454
q12	752	764	614	614
q13	3661	4295	3123	3123
q14	275	287	263	263
q15	518	482	489	482
q16	602	663	614	614
q17	1116	1351	1340	1340
q18	7416	7294	7237	7237
q19	838	808	800	800
q20	1862	1952	1788	1788
q21	4520	4235	4092	4092
q22	1059	1018	969	969
Total cold run time: 50767 ms
Total hot run time: 48297 ms

doris-robot · 2026-01-14T10:09:29Z

TPC-DS: Total hot run time: 173253 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b33ba3094c40dc4d99ed2ffefd51843849a03f2a, data reload: false
query5	4409	627	514	514
query6	335	238	205	205
query7	4217	451	254	254
query8	335	243	237	237
query9	8678	2909	2847	2847
query10	493	381	369	369
query11	15300	15101	14741	14741
query12	172	116	117	116
query13	1244	478	369	369
query14	6350	3110	2754	2754
query14_1	2725	2680	2749	2680
query15	204	191	176	176
query16	1007	524	436	436
query17	1072	652	541	541
query18	2582	420	324	324
query19	219	214	187	187
query20	123	110	109	109
query21	208	138	117	117
query22	3884	4129	4021	4021
query23	16078	15707	15336	15336
query23_1	15778	15492	15339	15339
query24	7115	1547	1122	1122
query24_1	1180	1157	1151	1151
query25	523	440	374	374
query26	1244	267	160	160
query27	2765	447	267	267
query28	4570	2143	2121	2121
query29	785	529	435	435
query30	300	237	207	207
query31	773	632	541	541
query32	86	78	77	77
query33	547	365	307	307
query34	900	888	517	517
query35	728	760	709	709
query36	896	861	839	839
query37	133	100	84	84
query38	2715	2726	2616	2616
query39	766	738	730	730
query39_1	696	726	706	706
query40	221	135	118	118
query41	83	65	62	62
query42	103	101	102	101
query43	480	458	402	402
query44	1362	736	732	732
query45	187	180	179	179
query46	820	954	561	561
query47	1383	1394	1389	1389
query48	316	322	246	246
query49	627	427	343	343
query50	624	285	200	200
query51	3840	3883	3816	3816
query52	105	109	95	95
query53	289	317	272	272
query54	303	290	295	290
query55	82	82	76	76
query56	308	310	328	310
query57	1019	1013	929	929
query58	271	262	267	262
query59	2145	2055	2173	2055
query60	346	341	329	329
query61	165	157	152	152
query62	392	346	302	302
query63	304	263	262	262
query64	4850	1296	1002	1002
query65	3812	3747	3770	3747
query66	1410	428	332	332
query67	14869	15019	14676	14676
query68	2797	1028	749	749
query69	467	377	344	344
query70	998	946	949	946
query71	338	333	306	306
query72	5825	3586	3605	3586
query73	591	717	308	308
query74	8754	8791	8556	8556
query75	2754	2791	2443	2443
query76	2430	1071	675	675
query77	365	380	314	314
query78	9756	9791	9144	9144
query79	2226	933	573	573
query80	1740	574	503	503
query81	550	268	231	231
query82	974	146	111	111
query83	367	262	241	241
query84	247	112	99	99
query85	916	514	443	443
query86	401	327	289	289
query87	2898	2857	2762	2762
query88	3525	2600	2570	2570
query89	383	346	336	336
query90	1948	179	167	167
query91	170	173	139	139
query92	75	73	70	70
query93	1145	925	533	533
query94	647	315	304	304
query95	612	393	310	310
query96	647	519	227	227
query97	2337	2385	2298	2298
query98	210	202	196	196
query99	598	564	499	499
Total cold run time: 249661 ms
Total hot run time: 173253 ms

doris-robot · 2026-01-14T10:14:30Z

ClickBench: Total hot run time: 26.79 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b33ba3094c40dc4d99ed2ffefd51843849a03f2a, data reload: false
query1	0.06	0.05	0.04
query2	0.10	0.06	0.05
query3	0.26	0.09	0.09
query4	1.60	0.12	0.11
query5	0.28	0.25	0.27
query6	1.14	0.67	0.65
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.57	0.51	0.51
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.58
query14	0.95	0.95	0.95
query15	0.80	0.79	0.78
query16	0.42	0.40	0.38
query17	1.05	0.98	1.03
query18	0.23	0.22	0.21
query19	1.84	1.81	1.74
query20	0.02	0.02	0.01
query21	15.46	0.27	0.13
query22	4.90	0.06	0.05
query23	15.69	0.29	0.11
query24	1.51	0.81	0.35
query25	0.10	0.09	0.07
query26	0.14	0.15	0.13
query27	0.06	0.07	0.06
query28	4.37	1.09	0.91
query29	12.52	3.90	3.16
query30	0.27	0.14	0.12
query31	2.81	0.63	0.40
query32	3.24	0.56	0.45
query33	2.94	2.96	3.05
query34	16.13	5.11	4.45
query35	4.52	4.43	4.44
query36	0.64	0.50	0.49
query37	0.11	0.06	0.07
query38	0.07	0.04	0.04
query39	0.05	0.04	0.03
query40	0.17	0.14	0.14
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.72 s
Total hot run time: 26.79 s

hello-stephen · 2026-01-14T12:38:59Z

FE Regression Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2026-01-21T02:25:57Z

run external

hello-stephen · 2026-01-21T04:29:06Z

FE Regression Coverage Report

Increment line coverage 20.00% (1/5) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2026-01-21T08:01:53Z

run buildall

hello-stephen · 2026-01-21T09:07:55Z

FE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2026-01-21T10:23:57Z

run complile

feiniaofeiafei · 2026-01-28T06:44:58Z

run compile

feiniaofeiafei · 2026-01-28T09:21:40Z

run nonConcurrent

feiniaofeiafei · 2026-01-28T09:21:49Z

run performance

doris-robot · 2026-01-28T09:47:01Z

TPC-H: Total hot run time: 32097 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 956a345eb4bbcf822e2af4e9b959e0ec0820caab, data reload: false
------ Round 1 ----------------------------------
q1	17662	5317	5100	5100
q2	2002	311	212	212
q3	10221	1325	733	733
q4	10227	897	321	321
q5	7501	2196	1876	1876
q6	197	181	149	149
q7	910	748	631	631
q8	9283	1414	1061	1061
q9	5135	4860	4812	4812
q10	6776	1942	1566	1566
q11	517	310	276	276
q12	336	382	229	229
q13	17801	4057	3206	3206
q14	236	247	222	222
q15	865	828	819	819
q16	677	678	625	625
q17	624	750	534	534
q18	6912	6525	7447	6525
q19	1253	1045	673	673
q20	427	354	245	245
q21	2996	2317	1998	1998
q22	386	319	284	284
Total cold run time: 102944 ms
Total hot run time: 32097 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5678	5428	5596	5428
q2	255	334	261	261
q3	2395	2840	2467	2467
q4	1450	2111	1403	1403
q5	4654	4557	4814	4557
q6	218	177	136	136
q7	2006	1919	1798	1798
q8	2655	2394	2448	2394
q9	7860	7635	7478	7478
q10	2865	3137	2566	2566
q11	547	469	463	463
q12	695	766	601	601
q13	4000	4283	3625	3625
q14	356	309	301	301
q15	852	813	794	794
q16	656	744	666	666
q17	1136	1255	1246	1246
q18	7621	7493	7387	7387
q19	834	812	810	810
q20	1965	2053	1918	1918
q21	4500	4149	4110	4110
q22	576	544	557	544
Total cold run time: 53774 ms
Total hot run time: 50953 ms

doris-robot · 2026-01-28T10:03:48Z

ClickBench: Total hot run time: 28.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 956a345eb4bbcf822e2af4e9b959e0ec0820caab, data reload: false
query1	0.05	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.13	0.12
query5	0.26	0.24	0.27
query6	1.17	0.69	0.68
query7	0.04	0.03	0.03
query8	0.05	0.03	0.04
query9	0.56	0.51	0.50
query10	0.54	0.54	0.54
query11	0.14	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.62	0.61
query14	1.07	1.05	1.04
query15	0.88	0.87	0.88
query16	0.39	0.42	0.38
query17	1.12	1.10	1.05
query18	0.22	0.20	0.20
query19	2.07	1.94	1.97
query20	0.02	0.02	0.01
query21	15.42	0.26	0.15
query22	5.08	0.05	0.05
query23	15.85	0.30	0.10
query24	1.58	0.25	0.18
query25	0.06	0.08	0.09
query26	0.15	0.14	0.13
query27	0.09	0.05	0.05
query28	4.11	1.15	0.97
query29	12.54	3.91	3.21
query30	0.28	0.13	0.14
query31	2.79	0.64	0.40
query32	3.25	0.59	0.50
query33	3.26	3.35	3.29
query34	15.93	5.38	4.81
query35	4.72	4.78	4.80
query36	0.65	0.51	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.19	0.17	0.15
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 97.66 s
Total hot run time: 28.31 s

hello-stephen · 2026-01-28T11:11:38Z

FE Regression Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

github-actions · 2026-01-28T12:59:00Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-01-28T12:59:03Z

PR approved by anyone and no changes requested.

feiniaofeiafei · 2026-01-29T06:14:54Z

run buildall

doris-robot · 2026-01-29T06:37:00Z

TPC-H: Total hot run time: 31931 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3985da1b832f95726e677ae7bfa445059c0b68c5, data reload: false
------ Round 1 ----------------------------------
q1	17639	5257	5097	5097
q2	2091	311	194	194
q3	10144	1354	809	809
q4	10204	867	333	333
q5	7568	2153	1886	1886
q6	194	182	149	149
q7	963	720	627	627
q8	9271	1384	1040	1040
q9	5109	4828	4835	4828
q10	6795	1973	1579	1579
q11	511	291	283	283
q12	334	380	228	228
q13	17771	4066	3252	3252
q14	233	234	217	217
q15	889	818	835	818
q16	686	676	636	636
q17	625	830	448	448
q18	6786	6565	6528	6528
q19	1239	993	614	614
q20	382	348	232	232
q21	2617	2042	1854	1854
q22	360	314	279	279
Total cold run time: 102411 ms
Total hot run time: 31931 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5314	5275	5304	5275
q2	260	344	244	244
q3	2140	2680	2299	2299
q4	1376	1740	1273	1273
q5	4283	4221	4201	4201
q6	218	181	141	141
q7	2166	2192	1833	1833
q8	2577	2442	2392	2392
q9	7518	7566	7537	7537
q10	2849	3098	2642	2642
q11	550	471	441	441
q12	694	738	606	606
q13	3857	4519	3451	3451
q14	303	311	316	311
q15	866	851	833	833
q16	689	749	680	680
q17	1174	1390	1400	1390
q18	8335	8080	7998	7998
q19	892	844	880	844
q20	2083	2161	2000	2000
q21	4980	4451	4200	4200
q22	578	563	502	502
Total cold run time: 53702 ms
Total hot run time: 51093 ms

doris-robot · 2026-01-29T06:53:44Z

ClickBench: Total hot run time: 28.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3985da1b832f95726e677ae7bfa445059c0b68c5, data reload: false
query1	0.05	0.04	0.04
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.12
query5	0.26	0.25	0.25
query6	1.16	0.68	0.67
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.57	0.51	0.49
query10	0.54	0.54	0.55
query11	0.15	0.09	0.10
query12	0.13	0.10	0.10
query13	0.64	0.62	0.62
query14	1.05	1.05	1.07
query15	0.88	0.87	0.87
query16	0.38	0.40	0.38
query17	1.14	1.13	1.14
query18	0.22	0.22	0.21
query19	2.12	2.02	2.08
query20	0.03	0.01	0.01
query21	15.39	0.28	0.15
query22	5.16	0.06	0.05
query23	16.02	0.30	0.11
query24	1.04	0.67	0.71
query25	0.09	0.06	0.08
query26	0.14	0.14	0.13
query27	0.06	0.06	0.07
query28	4.34	1.16	0.96
query29	12.55	3.92	3.16
query30	0.28	0.13	0.11
query31	2.82	0.63	0.40
query32	3.23	0.60	0.50
query33	3.25	3.28	3.30
query34	16.35	5.38	4.67
query35	4.80	4.78	4.80
query36	0.66	0.52	0.49
query37	0.11	0.07	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.19	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.13 s
Total hot run time: 28.74 s

hello-stephen · 2026-01-29T07:31:55Z

FE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-29T10:08:46Z

FE Regression Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2026-01-30T02:06:26Z

run external

feiniaofeiafei · 2026-01-30T02:06:32Z

run p0

feiniaofeiafei · 2026-01-30T02:06:41Z

run nonConcurrent

hello-stephen · 2026-01-30T04:19:32Z

FE Regression Coverage Report

Increment line coverage 40.00% (2/5) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-30T04:22:25Z

FE Regression Coverage Report

Increment line coverage 40.00% (2/5) 🎉
Increment coverage report
Complete coverage report

feiniaofeiafei · 2026-01-30T06:01:23Z

run nonConcurrent

hello-stephen · 2026-01-30T08:28:01Z

FE Regression Coverage Report

Increment line coverage 40.00% (2/5) 🎉
Increment coverage report
Complete coverage report

… shuffle key choosing (apache#59876) Problem Summary: add var agg_shuffle_use_parent_key, default value is true, when set false, can make agg shuffle by all gby key instead of parent shuffle key.

…eAggregation (#59116) (#61301) picked from #59116 #60045 #59876 #60091 #60811 --------- Co-authored-by: yujun <yujun@selectdb.com>

englefly previously approved these changes Jan 28, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 28, 2026

github-actionsBot added the reviewed label Jan 28, 2026

feiniaofeiafei dismissed englefly’s stale review via 2ec0d2eJanuary 29, 2026 04:14

feiniaofeiafei force-pushed the pick_agg_shuffle_var branch from 956a345 to 2ec0d2eCompare January 29, 2026 04:14

feiniaofeiafei added 3 commits January 29, 2026 12:15

add variable agg_shuffle_use_parent_key
390c923

add ut
87d7a01

add description
3985da1

feiniaofeiafei force-pushed the pick_agg_shuffle_var branch from 2ec0d2e to 3985da1Compare January 29, 2026 04:15

morrySnow approved these changes Jan 29, 2026
View reviewed changes

englefly approved these changes Jan 30, 2026
View reviewed changes

924060929 merged commit 0dcf271 into apache:masterJan 30, 2026
30 checks passed

morrySnow added the dev/4.1.x label Mar 12, 2026

feiniaofeiafei mentioned this pull request Mar 13, 2026
branch-4.1:[enhance](aggregate)Add rewrite rule DecomposeRepeatWithPreAggregation (#59116) #61301
Merged

yiguolei pushed a commit that referenced this pull request Mar 13, 2026
branch-4.1:[enhance](aggregate)Add rewrite rule DecomposeRepeatWithPr…
f3ac2d6
…eAggregation (#59116) (#61301) picked from #59116 #60045 #59876 #60091 #60811 --------- Co-authored-by: yujun <yujun@selectdb.com>

yiguolei added dev/4.1.0-merged and removed dev/4.1.x labels Mar 13, 2026

yiguolei pushed a commit that referenced this pull request Mar 16, 2026
branch-4.1:[enhance](aggregate)Add rewrite rule DecomposeRepeatWithPr…
fb8b3a2
…eAggregation (#59116) (#61301) picked from #59116 #60045 #59876 #60091 #60811 --------- Co-authored-by: yujun <yujun@selectdb.com>

Uh oh!

Conversation

feiniaofeiafei commented Jan 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 14, 2026

Uh oh!

feiniaofeiafei commented Jan 14, 2026

Uh oh!

doris-robot commented Jan 14, 2026

Uh oh!

doris-robot commented Jan 14, 2026

Uh oh!

doris-robot commented Jan 14, 2026

Uh oh!

hello-stephen commented Jan 14, 2026

FE Regression Coverage Report

Uh oh!

feiniaofeiafei commented Jan 21, 2026

Uh oh!

hello-stephen commented Jan 21, 2026

FE Regression Coverage Report

Uh oh!

feiniaofeiafei commented Jan 21, 2026

Uh oh!

hello-stephen commented Jan 21, 2026

FE UT Coverage Report

Uh oh!

feiniaofeiafei commented Jan 21, 2026

Uh oh!

feiniaofeiafei commented Jan 28, 2026

Uh oh!

feiniaofeiafei commented Jan 28, 2026

Uh oh!

feiniaofeiafei commented Jan 28, 2026

Uh oh!

doris-robot commented Jan 28, 2026

Uh oh!

doris-robot commented Jan 28, 2026

Uh oh!

hello-stephen commented Jan 28, 2026

FE Regression Coverage Report

Uh oh!

github-actionsBot commented Jan 28, 2026

Uh oh!

github-actionsBot commented Jan 28, 2026

Uh oh!

feiniaofeiafei commented Jan 29, 2026

Uh oh!

doris-robot commented Jan 29, 2026

Uh oh!

doris-robot commented Jan 29, 2026

Uh oh!

hello-stephen commented Jan 29, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jan 29, 2026

FE Regression Coverage Report

Uh oh!

feiniaofeiafei commented Jan 30, 2026

Uh oh!

feiniaofeiafei commented Jan 30, 2026

Uh oh!

feiniaofeiafei commented Jan 30, 2026

Uh oh!

hello-stephen commented Jan 30, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Jan 30, 2026

FE Regression Coverage Report

Uh oh!

feiniaofeiafei commented Jan 30, 2026

Uh oh!

hello-stephen commented Jan 30, 2026

FE Regression Coverage Report

Uh oh!

feiniaofeiafei commented Jan 14, 2026•
edited
Loading