[opt](sort) avoid copying whole blocks during merge by Mryange · Pull Request #63429 · apache/doris

Mryange · 2026-05-20T03:48:45Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: MergeSorterState used the generic copy-based merge path even when the current top sorted run could return its whole remaining block before any other run. This adds a direct whole-block fast path guarded by a total-order check, avoiding unnecessary insert_range_from work in inner merge.

What is changed?

Add MergeSortCursor::totally_less_or_equals() to detect when the current run is wholly before the next child.
Return the current block directly from MergeSorterState::_merge_sort_read_impl() when the whole-block condition is satisfied.
Add focused BE unit tests for exact-batch and smaller-than-batch whole-block fast-path cases.

Release note

None

hello-stephen · 2026-05-20T03:49:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: MergeSorterState used the generic row-copy merge path even when the current top run could return its whole remaining block before any other run. This adds a direct whole-block path guarded by a total-order check. ### Release note None ### Check List (For Author) - Test: Unit Test - BE UT: sh run-be-ut.sh --run --filter='MergeSorterStateTest.*' - Behavior changed: No - Does this need documentation: No

Mryange · 2026-05-20T04:43:30Z

/review

Mryange · 2026-05-20T06:41:19Z

run buildall

hello-stephen · 2026-05-20T07:51:09Z

TPC-H: Total hot run time: 31977 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2e02388121cfb6d7a7c08872a514c5843f0c0cf, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17805	4087	4124	4087
q2	q3	10770	1393	805	805
q4	4697	474	355	355
q5	7667	2304	2145	2145
q6	246	189	146	146
q7	999	765	635	635
q8	9489	1711	1707	1707
q9	5199	4903	4926	4903
q10	6409	2058	1844	1844
q11	449	280	246	246
q12	644	429	302	302
q13	18174	3405	2798	2798
q14	265	264	232	232
q15	q16	824	765	711	711
q17	939	973	1005	973
q18	7125	5954	5518	5518
q19	1289	1247	1190	1190
q20	563	448	308	308
q21	6225	2858	2748	2748
q22	539	367	324	324
Total cold run time: 100317 ms
Total hot run time: 31977 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4792	4763	4626	4626
q2	q3	4903	5219	4639	4639
q4	2110	2216	1423	1423
q5	4980	4738	4739	4738
q6	243	187	143	143
q7	1854	1759	1558	1558
q8	2521	2204	2169	2169
q9	7579	7243	7257	7243
q10	4506	4383	4010	4010
q11	534	379	353	353
q12	718	720	518	518
q13	3083	3408	2788	2788
q14	271	292	249	249
q15	q16	677	680	615	615
q17	1269	1255	1261	1255
q18	7226	6715	6789	6715
q19	1120	1091	1085	1085
q20	2220	2208	1922	1922
q21	5407	4640	4510	4510
q22	522	463	408	408
Total cold run time: 56535 ms
Total hot run time: 50967 ms

hello-stephen · 2026-05-20T08:02:11Z

TPC-DS: Total hot run time: 169378 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2e02388121cfb6d7a7c08872a514c5843f0c0cf, data reload: false
query5	4334	647	507	507
query6	324	225	193	193
query7	4214	587	301	301
query8	370	237	229	229
query9	8849	4068	4064	4064
query10	462	353	296	296
query11	5804	2555	2294	2294
query12	192	135	134	134
query13	1301	610	444	444
query14	5922	5351	5072	5072
query14_1	4374	4347	4364	4347
query15	209	205	178	178
query16	971	453	419	419
query17	945	710	591	591
query18	2438	494	355	355
query19	212	214	160	160
query20	136	138	139	138
query21	219	143	125	125
query22	13556	13592	13317	13317
query23	17215	16431	16057	16057
query23_1	16096	16164	16236	16164
query24	7518	1770	1306	1306
query24_1	1326	1323	1344	1323
query25	584	505	454	454
query26	1329	332	180	180
query27	2660	571	339	339
query28	4489	1990	1958	1958
query29	1027	657	548	548
query30	308	238	205	205
query31	1112	1073	944	944
query32	90	85	78	78
query33	553	368	314	314
query34	1175	1156	647	647
query35	766	778	679	679
query36	1341	1289	1175	1175
query37	149	109	97	97
query38	3236	3145	3087	3087
query39	923	925	897	897
query39_1	887	903	928	903
query40	234	153	131	131
query41	74	70	68	68
query42	120	116	115	115
query43	328	340	297	297
query44 query45	218	203	196	196
query46	1101	1225	706	706
query47	2353	2340	2195	2195
query48	427	427	300	300
query49	660	508	411	411
query50	943	358	264	264
query51	4396	4297	4248	4248
query52	108	111	98	98
query53	270	282	211	211
query54	322	297	271	271
query55	95	95	123	95
query56	296	319	316	316
query57	1409	1402	1285	1285
query58	301	271	278	271
query59	1500	1606	1381	1381
query60	327	329	307	307
query61	161	153	160	153
query62	669	616	565	565
query63	251	209	212	209
query64	2439	828	665	665
query65 query66	1736	466	364	364
query67	30078	30032	29775	29775
query68 query69	464	336	306	306
query70	1023	1000	989	989
query71	312	286	278	278
query72	3021	2768	2422	2422
query73	817	746	423	423
query74	5076	4902	4742	4742
query75	2681	2597	2267	2267
query76	2305	1166	813	813
query77	400	423	314	314
query78	12147	12113	11616	11616
query79	1394	1001	739	739
query80	656	546	463	463
query81	458	283	241	241
query82	720	163	126	126
query83	363	272	249	249
query84	303	140	113	113
query85	902	537	453	453
query86	395	327	344	327
query87	3383	3358	3195	3195
query88	3534	2691	2662	2662
query89	437	389	338	338
query90	1937	181	190	181
query91	180	166	143	143
query92	81	74	75	74
query93	1418	1421	869	869
query94	536	366	325	325
query95	673	393	446	393
query96	1043	849	343	343
query97	2701	2758	2589	2589
query98	240	226	230	226
query99	1120	1095	962	962
Total cold run time: 252042 ms
Total hot run time: 169378 ms

Mryange · 2026-05-20T10:23:39Z

run buildall

Mryange · 2026-05-20T11:04:19Z

run buildall

hello-stephen · 2026-05-20T11:20:18Z

TPC-H: Total hot run time: 31585 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd5aa44d93be74f07b9de43d1d335283958205b2, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17630	3931	3874	3874
q2	q3	10772	1382	831	831
q4	4681	475	347	347
q5	7556	2301	2112	2112
q6	244	181	143	143
q7	943	786	641	641
q8	9389	1747	1706	1706
q9	5134	4931	4934	4931
q10	6422	2094	1843	1843
q11	449	267	240	240
q12	627	427	296	296
q13	18167	3418	2755	2755
q14	268	260	236	236
q15	q16	816	771	710	710
q17	997	985	1013	985
q18	6883	5696	5538	5538
q19	1281	1296	1096	1096
q20	527	450	337	337
q21	6207	2846	2645	2645
q22	485	372	319	319
Total cold run time: 99478 ms
Total hot run time: 31585 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	5211	4588	4541	4541
q2	q3	4871	5273	4705	4705
q4	2134	2184	1437	1437
q5	4887	4660	4601	4601
q6	245	186	132	132
q7	1948	1718	1515	1515
q8	2382	2083	2066	2066
q9	7809	7210	7234	7210
q10	4527	4401	3957	3957
q11	523	381	351	351
q12	718	731	515	515
q13	3083	3302	2844	2844
q14	276	292	244	244
q15	q16	695	693	618	618
q17	1271	1237	1239	1237
q18	7498	6804	6728	6728
q19	1137	1121	1124	1121
q20	2217	2216	1941	1941
q21	5327	4652	4558	4558
q22	523	493	409	409
Total cold run time: 57282 ms
Total hot run time: 50730 ms

hello-stephen · 2026-05-20T11:31:25Z

TPC-DS: Total hot run time: 171135 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd5aa44d93be74f07b9de43d1d335283958205b2, data reload: false
query5	4353	656	528	528
query6	337	250	208	208
query7	4316	565	321	321
query8	340	244	229	229
query9	8831	4024	4060	4024
query10	445	344	298	298
query11	5814	2402	2251	2251
query12	197	130	128	128
query13	1305	599	458	458
query14	5963	5323	5040	5040
query14_1	4370	4363	4331	4331
query15	208	202	184	184
query16	985	444	424	424
query17	952	714	590	590
query18	2448	486	353	353
query19	210	209	167	167
query20	138	133	129	129
query21	217	141	120	120
query22	13590	13590	13467	13467
query23	17204	16322	15937	15937
query23_1	16226	16104	16269	16104
query24	7474	1791	1319	1319
query24_1	1326	1327	1327	1327
query25	596	504	452	452
query26	1342	334	178	178
query27	2688	583	353	353
query28	4528	2005	1956	1956
query29	1041	655	543	543
query30	308	237	202	202
query31	1120	1088	941	941
query32	94	83	78	78
query33	555	363	315	315
query34	1202	1170	629	629
query35	773	778	693	693
query36	1339	1338	1222	1222
query37	162	109	92	92
query38	3191	3129	3060	3060
query39	938	927	891	891
query39_1	884	908	869	869
query40	237	157	140	140
query41	73	78	70	70
query42	115	112	110	110
query43	341	336	287	287
query44 query45	217	207	201	201
query46	1136	1260	721	721
query47	2343	2315	2232	2232
query48	406	434	315	315
query49	671	511	404	404
query50	990	378	260	260
query51	4306	4308	4221	4221
query52	110	117	102	102
query53	258	300	218	218
query54	336	293	284	284
query55	103	103	105	103
query56	345	328	332	328
query57	1424	1415	1337	1337
query58	309	283	291	283
query59	1558	1607	1412	1412
query60	365	320	307	307
query61	164	159	159	159
query62	675	627	571	571
query63	247	202	221	202
query64	2434	821	656	656
query65 query66	1709	502	350	350
query67	30036	29966	29811	29811
query68 query69	466	343	296	296
query70	1041	1033	941	941
query71	315	273	268	268
query72	3062	2739	2506	2506
query73	814	733	447	447
query74	5071	4912	4705	4705
query75	2679	2618	2269	2269
query76	2287	1144	772	772
query77	418	429	335	335
query78	12249	12070	11747	11747
query79	1509	1041	744	744
query80	1322	561	465	465
query81	528	276	241	241
query82	958	160	124	124
query83	366	282	255	255
query84	303	146	117	117
query85	946	557	466	466
query86	466	351	327	327
query87	3429	3356	3218	3218
query88	3605	2704	2647	2647
query89	458	392	342	342
query90	1970	179	187	179
query91	185	181	139	139
query92	79	81	71	71
query93	1662	1559	945	945
query94	728	369	330	330
query95	694	479	344	344
query96	1080	749	352	352
query97	2679	2696	2587	2587
query98	243	233	234	233
query99	1139	1107	987	987
Total cold run time: 254527 ms
Total hot run time: 171135 ms

hello-stephen · 2026-05-20T11:51:29Z

BE UT Coverage Report

Increment line coverage 100.00% (19/19) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.53% (20735/38737)
Line Coverage	37.17% (196161/527675)
Region Coverage	33.50% (153732/458891)
Branch Coverage	34.53% (67018/194084)

hello-stephen · 2026-05-20T12:26:46Z

BE UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.53% (20735/38737)
Line Coverage	37.17% (196160/527672)
Region Coverage	33.50% (153705/458887)
Branch Coverage	34.52% (67005/194078)

hello-stephen · 2026-05-20T13:58:05Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.70% (27961/37941)
Line Coverage	57.66% (303491/526321)
Region Coverage	54.99% (254754/463310)
Branch Coverage	56.42% (109917/194805)

hello-stephen · 2026-05-20T14:21:47Z

TPC-H: Total hot run time: 30985 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e5bf1c023a168cc02408549b2b21b85af444242, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17747	3902	4023	3902
q2	q3	10810	1438	853	853
q4	4723	476	368	368
q5	7881	2312	2084	2084
q6	357	180	141	141
q7	951	801	636	636
q8	9367	1774	1615	1615
q9	7067	5041	4924	4924
q10	6434	2158	1788	1788
q11	444	280	246	246
q12	694	420	308	308
q13	18244	3355	2753	2753
q14	264	257	240	240
q15	q16	817	789	711	711
q17	1020	945	999	945
q18	6888	5674	5410	5410
q19	1177	1334	1158	1158
q20	524	412	271	271
q21	5709	2549	2325	2325
q22	443	362	307	307
Total cold run time: 101561 ms
Total hot run time: 30985 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4192	4190	4262	4190
q2	q3	4532	4954	4321	4321
q4	2147	2265	1391	1391
q5	4435	4256	4978	4256
q6	249	190	147	147
q7	2009	1807	1729	1729
q8	2452	2221	2209	2209
q9	7866	7856	7776	7776
q10	4560	4551	4102	4102
q11	789	417	370	370
q12	723	738	530	530
q13	3383	3666	3039	3039
q14	304	290	280	280
q15	q16	739	759	631	631
q17	1424	1319	1322	1319
q18	8060	7321	6720	6720
q19	1126	1127	1069	1069
q20	2226	2205	1934	1934
q21	5344	4756	4512	4512
q22	546	485	405	405
Total cold run time: 57106 ms
Total hot run time: 50930 ms

hello-stephen · 2026-05-20T14:32:40Z

TPC-DS: Total hot run time: 169153 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e5bf1c023a168cc02408549b2b21b85af444242, data reload: false
query5	4333	657	511	511
query6	334	223	201	201
query7	4259	551	299	299
query8	342	244	232	232
query9	8820	4027	4053	4027
query10	454	372	307	307
query11	5867	2422	2178	2178
query12	194	133	130	130
query13	1295	633	421	421
query14	6038	5372	5092	5092
query14_1	4376	4379	4387	4379
query15	227	204	181	181
query16	1014	503	456	456
query17	1152	754	601	601
query18	2476	499	371	371
query19	222	219	168	168
query20	145	135	134	134
query21	214	140	121	121
query22	13581	13506	13417	13417
query23	17292	16300	15993	15993
query23_1	16126	16285	16221	16221
query24	7613	1758	1313	1313
query24_1	1335	1306	1298	1298
query25	585	519	453	453
query26	1309	323	171	171
query27	2708	526	346	346
query28	4441	1947	1939	1939
query29	970	646	518	518
query30	306	243	204	204
query31	1123	1078	946	946
query32	95	78	76	76
query33	556	369	312	312
query34	1179	1142	632	632
query35	778	827	682	682
query36	1352	1336	1230	1230
query37	152	96	87	87
query38	3212	3162	3071	3071
query39	920	944	912	912
query39_1	876	863	874	863
query40	233	147	130	130
query41	65	62	61	61
query42	113	109	110	109
query43	316	325	284	284
query44 query45	212	204	196	196
query46	1073	1231	698	698
query47	2335	2355	2210	2210
query48	403	414	291	291
query49	619	484	382	382
query50	976	345	253	253
query51	4328	4373	4192	4192
query52	109	104	96	96
query53	258	274	215	215
query54	304	275	253	253
query55	94	92	84	84
query56	298	310	302	302
query57	1424	1407	1334	1334
query58	291	274	258	258
query59	1540	1601	1356	1356
query60	322	337	304	304
query61	157	153	153	153
query62	663	616	556	556
query63	246	207	214	207
query64	2403	798	617	617
query65 query66	1701	490	360	360
query67	29966	29991	29808	29808
query68 query69	448	347	302	302
query70	1017	1036	1008	1008
query71	307	280	260	260
query72	3004	2769	2388	2388
query73	851	769	417	417
query74	5058	4951	4745	4745
query75	2662	2588	2277	2277
query76	2283	1151	762	762
query77	403	431	327	327
query78	12151	12098	11598	11598
query79	1490	1055	731	731
query80	976	552	460	460
query81	514	282	236	236
query82	1346	164	130	130
query83	351	280	258	258
query84	259	142	108	108
query85	915	545	460	460
query86	443	337	312	312
query87	3413	3447	3214	3214
query88	3509	2639	2609	2609
query89	445	383	346	346
query90	1834	180	180	180
query91	178	165	141	141
query92	83	83	73	73
query93	1590	1480	940	940
query94	643	366	315	315
query95	702	475	365	365
query96	1011	801	325	325
query97	2686	2713	2558	2558
query98	249	228	224	224
query99	1146	1112	957	957
Total cold run time: 253635 ms
Total hot run time: 169153 ms

Mryange · 2026-05-21T00:19:07Z

/review

github-actions

Review summary: no blocking issues found in the actual PR diff.

Critical checkpoint conclusions:

Goal/test coverage: the PR adds a whole-block fast path for merge-sort reads and adds focused BE unit coverage for block swapping and smaller-than-batch behavior. Existing sort operator/heap sorter tests were adjusted to consume multiple output batches, which matches the new behavior.
Scope: the actual GitHub PR contains only sort fast-path/test changes and is focused.
Concurrency/lifecycle: no new shared state, threads, locks, or special lifecycle concerns were introduced.
Configuration/compatibility/persistence: no new config items, storage formats, RPC/thrift/proto changes, or rolling-upgrade compatibility concerns in the actual PR diff.
Parallel paths: FullSorter/HeapSorter/TopNSorter all use MergeSorterState, so the optimization is applied at the common merge-sort read path.
Error handling/memory: no Status paths are discarded. The fast path transfers Block ownership by swap only when the current run can be returned whole, avoiding extra copies; it preserves the existing two-call EOS contract.
Performance: the optimization is reasonable and avoids per-row/per-column copying when one sorted block is wholly before the next run.
User focus: no additional user-provided review focus was present.

Residual risk: I did not run BE unit tests in this review environment; the conclusion is based on code inspection and the PR patch from GitHub.

HappenLee

LGTM

Problem Summary: `MergeSorterState` used the generic copy-based merge path even when the current top sorted run could return its whole remaining block before any other run. This adds a direct whole-block fast path guarded by a total-order check, avoiding unnecessary `insert_range_from` work in inner merge. ### What is changed? - Add `MergeSortCursor::totally_less_or_equals()` to detect when the current run is wholly before the next child. - Return the current block directly from `MergeSorterState::_merge_sort_read_impl()` when the whole-block condition is satisfied. - Add focused BE unit tests for exact-batch and smaller-than-batch whole-block fast-path cases.

…3429)" This reverts commit ea00d2f.

Mryange force-pushed the opt-merge-whole-block-fast-path branch from ca5da84 to a2e0238Compare May 20, 2026 03:55

fix beut
1e5bf1c

Mryange force-pushed the opt-merge-whole-block-fast-path branch from dd5aa44 to 1e5bf1cCompare May 20, 2026 11:03

github-actionsBot reviewed May 21, 2026
View reviewed changes

HappenLee approved these changes May 21, 2026
View reviewed changes

HappenLee merged commit 974f9bd into apache:masterMay 21, 2026
30 of 31 checks passed

Mryange mentioned this pull request Jun 15, 2026
[fix](be) Backport selected query engine fixes to branch-4.1 #64537
Merged
16 tasks

Mryange added a commit to Mryange/doris that referenced this pull request Jun 18, 2026
Revert "[opt](sort) avoid copying whole blocks during merge (apache#6…
4ed62a4
…3429)" This reverts commit ea00d2f.

Uh oh!

Conversation

Mryange commented May 20, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed?

Release note

Uh oh!

hello-stephen commented May 20, 2026

Uh oh!

Mryange commented May 20, 2026

Uh oh!

Mryange commented May 20, 2026

Uh oh!

hello-stephen commented May 20, 2026

Uh oh!

hello-stephen commented May 20, 2026

Uh oh!

Mryange commented May 20, 2026

Uh oh!

Mryange commented May 20, 2026

Uh oh!

hello-stephen commented May 20, 2026

Uh oh!

hello-stephen commented May 20, 2026

Uh oh!

hello-stephen commented May 20, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented May 20, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented May 20, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented May 20, 2026

Uh oh!

hello-stephen commented May 20, 2026

Uh oh!

Mryange commented May 21, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Mryange commented May 20, 2026•
edited
Loading