[fix](group commit) fix can not get a block queue by mymeiyi · Pull Request #63722 · apache/doris

mymeiyi · 2026-05-27T06:42:35Z

Problem Summary:
Under high-concurrency async stream load, group commit may fail to get a block queue when creating the group commit plan fragment fails or leaves pending load requests waiting for a queue that is no longer usable. This change tracks pending create-plan requests per table, adds a background worker to resubmit group commit plan creation, and introduces group_commit_create_plan_timeout_ms so waiting requests can be released after a bounded time.

hello-stephen · 2026-05-27T06:42:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR addresses failures in group-commit stream load where the BE could not obtain a usable LoadBlockQueue under concurrency, by improving error handling around FE RPCs and adding resubmission/timeout logic for creating group-commit plan fragments. It also adds a new high-concurrency regression test intended to reproduce/guard the issue.

Changes:

Improve RPC-status handling when aborting/committing group-commit transactions (avoid masking transport errors as FE result statuses).
Add a background “create plan” worker + resubmission/timeout mechanism to reduce “can not get a block queue” occurrences during plan creation.
Add a high-concurrency async group-commit stream load regression test.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
regression-test/suites/load_p0/stream_load/test_group_commit_stream_load_high_concurrency_async.groovy	New regression test stressing async group-commit under high concurrency.
be/src/load/group_commit/wal/wal_table.cpp	Return/LOG transport (RPC) errors distinctly from FE rollback result status.
be/src/load/group_commit/group_commit_mgr.h	Add new create-plan worker plumbing and make `_need_commit` atomic.
be/src/load/group_commit/group_commit_mgr.cpp	Implement create-plan resubmission worker, timeout handling, and improved commit/rollback RPC error handling.
be/src/common/config.h	Declare new config `group_commit_create_plan_timeout_ms`.
be/src/common/config.cpp	Define default for `group_commit_create_plan_timeout_ms`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mymeiyi · 2026-05-27T07:03:48Z

/review

github-actions

I found one blocking issue in the new group-commit plan creation worker. The new background thread does not initialize Doris thread context before executing BE code, which violates the BE runtime memory-tracking convention. Existing Copilot comments already covered the timeout comment mismatch, the add-load-id error message, and the p0 test size concern, so I did not duplicate those.

Checklist conclusions:

Goal/test: The PR targets high-concurrency async group commit plan creation and adds a regression case, but the new worker thread setup is incomplete.
Scope: The implementation is focused on group commit queue creation/retry, though it introduces a new manager thread.
Concurrency: The new worker/condition-variable path is relevant; lock ordering looks consistent, but the worker thread lacks required thread-context initialization.
Lifecycle: GroupCommitMgr::stop() joins the new worker before shutting down the pool; no additional lifecycle blocker found.
Config: Adds a mutable timeout config with default/comment now aligned.
Compatibility: No protocol/storage-format incompatibility found.
Parallel paths: WAL rollback RPC handling was updated consistently in the touched paths.
Tests: A stress regression test was added; existing review already noted the p0 size/flakiness risk.
Observability: Existing logs are mostly sufficient for retries/failures.
Transaction/persistence/data correctness: No additional transaction visibility or WAL recovery issue found in the changed logic.
Performance: No additional performance blocker beyond the existing high-concurrency p0 test concern.
User focus: No additional user-provided review focus was specified.

mymeiyi · 2026-05-27T09:03:49Z

run buildall

hello-stephen · 2026-05-27T11:20:22Z

TPC-H: Total hot run time: 31323 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 250bbcbb34580f8acafaa87e43bc633533c2c37c, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17833	4105	4072	4072
q2	q3	10769	1426	824	824
q4	4723	476	351	351
q5	8563	2357	2098	2098
q6	372	180	136	136
q7	962	774	668	668
q8	9593	1628	1494	1494
q9	7018	4959	4928	4928
q10	6484	2221	1904	1904
q11	442	291	243	243
q12	690	422	294	294
q13	18225	3444	2805	2805
q14	266	262	239	239
q15	q16	812	783	712	712
q17	1007	966	1028	966
q18	6843	5853	5521	5521
q19	1180	1183	1103	1103
q20	515	405	257	257
q21	5692	2668	2402	2402
q22	447	352	306	306
Total cold run time: 102436 ms
Total hot run time: 31323 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4356	4258	4251	4251
q2	q3	4552	4964	4324	4324
q4	2106	2231	1408	1408
q5	4441	4343	5024	4343
q6	267	191	144	144
q7	1926	1889	1589	1589
q8	2528	2147	2116	2116
q9	8138	7954	8069	7954
q10	4893	4761	4311	4311
q11	559	423	381	381
q12	761	762	551	551
q13	3256	3655	3029	3029
q14	314	319	265	265
q15	q16	719	755	703	703
q17	1390	1333	1326	1326
q18	7900	7369	6742	6742
q19	1125	1107	1100	1100
q20	2238	2250	1951	1951
q21	5289	4597	4417	4417
q22	536	481	407	407
Total cold run time: 57294 ms
Total hot run time: 51312 ms

hello-stephen · 2026-05-27T11:31:24Z

TPC-DS: Total hot run time: 172487 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 250bbcbb34580f8acafaa87e43bc633533c2c37c, data reload: false
query5	4337	671	525	525
query6	334	219	199	199
query7	4239	544	308	308
query8	324	229	220	220
query9	8793	4082	4091	4082
query10	467	334	302	302
query11	5781	2828	2256	2256
query12	187	137	149	137
query13	1290	593	439	439
query14	6091	5452	5102	5102
query14_1	4456	4821	4457	4457
query15	218	206	185	185
query16	1022	460	434	434
query17	1159	717	590	590
query18	2513	481	357	357
query19	217	204	166	166
query20	140	140	137	137
query21	217	136	118	118
query22	13744	13526	13493	13493
query23	17465	16552	16349	16349
query23_1	16301	16371	16362	16362
query24	7438	1816	1346	1346
query24_1	1292	1356	1308	1308
query25	585	507	455	455
query26	1303	338	188	188
query27	2643	588	347	347
query28	4461	2019	2020	2019
query29	1010	661	523	523
query30	309	245	199	199
query31	1160	1090	958	958
query32	90	79	80	79
query33	568	396	310	310
query34	1196	1170	652	652
query35	806	807	703	703
query36	1441	1398	1213	1213
query37	159	112	93	93
query38	3223	3154	3100	3100
query39	945	922	892	892
query39_1	878	887	907	887
query40	233	156	129	129
query41	71	69	68	68
query42	113	113	113	113
query43	332	335	300	300
query44 query45	236	208	200	200
query46	1088	1249	763	763
query47	2380	2369	2261	2261
query48	410	431	294	294
query49	641	510	406	406
query50	980	363	270	270
query51	4346	4283	4258	4258
query52	114	114	97	97
query53	261	292	211	211
query54	333	289	269	269
query55	95	99	89	89
query56	326	313	319	313
query57	1432	1434	1353	1353
query58	318	286	288	286
query59	1625	1674	1486	1486
query60	372	338	323	323
query61	161	163	156	156
query62	697	649	597	597
query63	246	203	217	203
query64	2414	803	641	641
query65 query66	1709	484	351	351
query67	29772	29088	29612	29088
query68 query69	459	350	306	306
query70	1081	1022	970	970
query71	300	274	275	274
query72	3068	2767	2433	2433
query73	891	741	444	444
query74	5139	4976	4791	4791
query75	2697	2585	2278	2278
query76	2289	1175	794	794
query77	413	422	335	335
query78	12556	12478	11942	11942
query79	1501	1074	788	788
query80	1221	545	500	500
query81	506	281	240	240
query82	1401	160	124	124
query83	363	299	258	258
query84	270	142	113	113
query85	946	530	471	471
query86	449	340	338	338
query87	3415	3395	3228	3228
query88	3660	2736	2745	2736
query89	458	391	348	348
query90	1827	190	194	190
query91	184	172	145	145
query92	85	79	76	76
query93	1534	1415	849	849
query94	620	367	292	292
query95	693	381	363	363
query96	1075	798	355	355
query97	2783	2760	2606	2606
query98	251	228	226	226
query99	1179	1134	1027	1027
Total cold run time: 255883 ms
Total hot run time: 172487 ms

hello-stephen · 2026-05-27T14:04:07Z

BE UT Coverage Report

Increment line coverage 0.00% (0/251) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.90% (20945/38861)
Line Coverage	37.45% (198483/530005)
Region Coverage	33.73% (155482/460967)
Branch Coverage	34.74% (67741/194967)

hello-stephen · 2026-05-27T14:11:37Z

BE Regression && UT Coverage Report

Increment line coverage 58.17% (146/251) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.92% (28117/38035)
Line Coverage	57.84% (305456/528150)
Region Coverage	54.98% (255639/464993)
Branch Coverage	56.49% (110406/195446)

mymeiyi · 2026-05-28T11:24:52Z

run buildall

hello-stephen · 2026-05-28T13:36:07Z

TPC-H: Total hot run time: 31954 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3e7b8175a0b8ad3b8cbb2e7e5e4d0c63be602c45, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17893	4091	4057	4057
q2	q3	10812	1400	818	818
q4	4699	488	345	345
q5	7805	2258	2185	2185
q6	381	177	137	137
q7	951	809	647	647
q8	9608	1878	1672	1672
q9	7079	4944	4944	4944
q10	6448	2258	1960	1960
q11	436	282	251	251
q12	689	439	307	307
q13	18230	3348	2827	2827
q14	271	257	238	238
q15	q16	829	778	717	717
q17	997	941	966	941
q18	7125	5790	5621	5621
q19	1292	1413	1098	1098
q20	530	433	263	263
q21	5949	2707	2603	2603
q22	456	369	323	323
Total cold run time: 102480 ms
Total hot run time: 31954 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4969	4975	5003	4975
q2	q3	4991	5315	4617	4617
q4	2151	2267	1451	1451
q5	4971	4686	4705	4686
q6	239	180	127	127
q7	1871	1800	1448	1448
q8	2275	1985	1950	1950
q9	7412	7354	7439	7354
q10	4792	4702	4218	4218
q11	530	390	359	359
q12	733	747	534	534
q13	3057	3429	2747	2747
q14	288	278	251	251
q15	q16	681	711	610	610
q17	1296	1284	1279	1279
q18	7358	6869	6885	6869
q19	1084	1159	1101	1101
q20	2236	2243	1953	1953
q21	5329	4632	4512	4512
q22	527	449	410	410
Total cold run time: 56790 ms
Total hot run time: 51451 ms

hello-stephen · 2026-05-28T13:47:22Z

TPC-DS: Total hot run time: 172648 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3e7b8175a0b8ad3b8cbb2e7e5e4d0c63be602c45, data reload: false
query5	4351	662	541	541
query6	348	210	203	203
query7	4221	606	301	301
query8	326	234	224	224
query9	8817	4169	4109	4109
query10	454	349	310	310
query11	5815	2935	2272	2272
query12	187	132	125	125
query13	1276	637	437	437
query14	6132	5497	5181	5181
query14_1	4495	4539	4494	4494
query15	210	201	181	181
query16	1036	475	446	446
query17	1125	712	597	597
query18	2717	491	355	355
query19	227	199	155	155
query20	139	132	135	132
query21	214	138	119	119
query22	13789	13702	13529	13529
query23	17535	16694	16196	16196
query23_1	16374	16275	16342	16275
query24	7567	1783	1332	1332
query24_1	1348	1333	1339	1333
query25	561	466	414	414
query26	1305	317	167	167
query27	2752	557	342	342
query28	4434	2044	1998	1998
query29	996	628	494	494
query30	311	247	197	197
query31	1161	1112	959	959
query32	89	73	73	73
query33	531	346	301	301
query34	1205	1165	628	628
query35	776	815	688	688
query36	1442	1412	1273	1273
query37	166	107	90	90
query38	3237	3165	3097	3097
query39	943	912	903	903
query39_1	875	871	871	871
query40	233	151	129	129
query41	67	67	63	63
query42	113	106	107	106
query43	336	335	299	299
query44 query45	215	211	198	198
query46	1128	1225	745	745
query47	2391	2345	2213	2213
query48	413	404	306	306
query49	622	494	388	388
query50	1003	354	252	252
query51	4414	4313	4253	4253
query52	105	116	104	104
query53	266	288	209	209
query54	308	264	254	254
query55	94	91	88	88
query56	295	301	322	301
query57	1439	1397	1328	1328
query58	305	275	278	275
query59	1602	1695	1478	1478
query60	318	324	313	313
query61	155	146	152	146
query62	689	667	565	565
query63	253	201	211	201
query64	2426	853	700	700
query65 query66	1686	503	386	386
query67	29887	29852	29057	29057
query68 query69	480	366	321	321
query70	1064	1000	1029	1000
query71	320	282	269	269
query72	3199	2988	2586	2586
query73	853	753	468	468
query74	5151	4992	4822	4822
query75	2696	2617	2284	2284
query76	2274	1166	805	805
query77	406	431	346	346
query78	12603	12560	12056	12056
query79	1520	1023	759	759
query80	1320	544	465	465
query81	530	286	241	241
query82	1056	165	123	123
query83	355	276	249	249
query84	261	143	113	113
query85	957	541	448	448
query86	452	342	334	334
query87	3486	3402	3267	3267
query88	3695	2781	2760	2760
query89	466	389	344	344
query90	1941	186	196	186
query91	180	213	136	136
query92	82	80	74	74
query93	1643	1443	835	835
query94	754	365	327	327
query95	679	395	461	395
query96	1135	816	327	327
query97	2764	2745	2592	2592
query98	240	230	232	230
query99	1165	1148	1039	1039
Total cold run time: 256996 ms
Total hot run time: 172648 ms

hello-stephen · 2026-05-28T14:02:44Z

BE UT Coverage Report

Increment line coverage 0.00% (0/256) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.94% (20963/38863)
Line Coverage	37.50% (198761/530072)
Region Coverage	33.77% (155718/461057)
Branch Coverage	34.78% (67823/194997)

hello-stephen · 2026-05-28T15:20:16Z

BE Regression && UT Coverage Report

Increment line coverage 57.03% (146/256) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.86% (28112/38062)
Line Coverage	57.76% (305411/528716)
Region Coverage	55.04% (256218/465472)
Branch Coverage	56.46% (110514/195723)

mymeiyi · 2026-06-04T10:40:01Z

run buildall

hello-stephen · 2026-06-04T14:13:13Z

TPC-H: Total hot run time: 29365 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0afd596e5b21461bb4c083d36e3bd96adc98fa4a, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17753	4074	4014	4014
q2	q3	10764	1508	838	838
q4	4691	474	349	349
q5	7558	902	586	586
q6	181	172	137	137
q7	764	848	661	661
q8	9358	1700	1590	1590
q9	5789	4535	4553	4535
q10	6810	1836	1512	1512
q11	441	283	249	249
q12	635	421	292	292
q13	18091	3478	2743	2743
q14	271	263	239	239
q15	q16	827	781	713	713
q17	998	954	1054	954
q18	6975	5820	5571	5571
q19	1309	1239	1110	1110
q20	524	415	267	267
q21	6424	2839	2678	2678
q22	475	403	327	327
Total cold run time: 100638 ms
Total hot run time: 29365 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	5154	4726	4967	4726
q2	q3	4908	5319	4718	4718
q4	2127	2215	1376	1376
q5	4844	4935	4712	4712
q6	226	179	126	126
q7	1876	1768	1645	1645
q8	2505	2156	2156	2156
q9	8170	7687	7400	7400
q10	4754	4668	4198	4198
q11	550	410	380	380
q12	726	740	529	529
q13	3073	3367	2847	2847
q14	271	285	262	262
q15	q16	694	711	625	625
q17	1290	1271	1269	1269
q18	7572	6768	6699	6699
q19	1114	1118	1142	1118
q20	2229	2218	1946	1946
q21	5319	4631	4510	4510
q22	519	466	403	403
Total cold run time: 57921 ms
Total hot run time: 51645 ms

hello-stephen · 2026-06-04T14:21:32Z

BE UT Coverage Report

Increment line coverage 0.00% (0/265) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.91% (21052/39048)
Line Coverage	37.57% (200156/532752)
Region Coverage	33.64% (156915/466410)
Branch Coverage	34.62% (68681/198404)

mymeiyi · 2026-06-29T06:28:40Z

run cloud_p0

hello-stephen · 2026-06-29T06:29:39Z

TPC-H: Total hot run time: 29370 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0afd596e5b21461bb4c083d36e3bd96adc98fa4a, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17620	4131	4035	4035
q2	2023	323	194	194
q3	10285	1437	809	809
q4	4679	470	334	334
q5	7571	870	571	571
q6	180	174	136	136
q7	785	846	626	626
q8	9319	1546	1640	1546
q9	5593	4572	4507	4507
q10	6769	1815	1540	1540
q11	464	283	256	256
q12	635	424	285	285
q13	18104	3455	2802	2802
q14	260	265	252	252
q15	q16	786	787	706	706
q17	963	981	1012	981
q18	7121	5881	5577	5577
q19	1297	1280	1106	1106
q20	499	394	267	267
q21	5870	2582	2534	2534
q22	424	364	306	306
Total cold run time: 101247 ms
Total hot run time: 29370 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4378	4261	4296	4261
q2	334	346	233	233
q3	4558	4973	4413	4413
q4	2067	2186	1389	1389
q5	4433	4339	4309	4309
q6	229	178	128	128
q7	1726	1631	2002	1631
q8	2635	2191	2221	2191
q9	8309	8399	8093	8093
q10	4821	4822	4278	4278
q11	562	427	375	375
q12	742	769	568	568
q13	3262	3643	2996	2996
q14	294	293	271	271
q15	q16	733	716	650	650
q17	1357	1342	1304	1304
q18	7964	7304	7183	7183
q19	1173	1162	1145	1145
q20	2230	2216	1940	1940
q21	5307	4609	4488	4488
q22	517	450	412	412
Total cold run time: 57631 ms
Total hot run time: 52258 ms

hello-stephen · 2026-06-29T06:40:32Z

TPC-DS: Total hot run time: 170933 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0afd596e5b21461bb4c083d36e3bd96adc98fa4a, data reload: false
query5	4310	631	473	473
query6	444	188	165	165
query7	4814	573	304	304
query8	340	182	171	171
query9	8763	4002	4013	4002
query10	446	309	256	256
query11	5898	2358	2131	2131
query12	161	104	105	104
query13	1309	616	426	426
query14	6266	5314	4944	4944
query14_1	4256	4272	4268	4268
query15	209	203	185	185
query16	1041	464	435	435
query17	1104	711	570	570
query18	2501	474	340	340
query19	195	181	139	139
query20	112	103	112	103
query21	216	142	113	113
query22	13656	13692	13513	13513
query23	17376	16579	16148	16148
query23_1	16142	16187	16263	16187
query24	7456	1842	1309	1309
query24_1	1339	1257	1302	1257
query25	562	447	391	391
query26	1295	351	176	176
query27	2643	597	327	327
query28	4417	2053	2036	2036
query29	1088	623	511	511
query30	314	232	205	205
query31	1110	1086	967	967
query32	118	65	60	60
query33	516	347	258	258
query34	1214	1126	657	657
query35	781	800	664	664
query36	1418	1394	1210	1210
query37	155	107	95	95
query38	1902	1733	1657	1657
query39	932	925	915	915
query39_1	878	873	881	873
query40	226	128	107	107
query41	71	73	68	68
query42	91	93	88	88
query43	326	323	278	278
query44	1442	794	786	786
query45	207	193	189	189
query46	1094	1197	710	710
query47	2369	2370	2225	2225
query48	417	443	310	310
query49	587	433	316	316
query50	998	373	308	308
query51	4422	4370	4348	4348
query52	82	81	67	67
query53	248	268	187	187
query54	267	222	197	197
query55	74	72	67	67
query56	232	221	209	209
query57	1470	1400	1336	1336
query58	242	207	216	207
query59	1572	1647	1424	1424
query60	276	245	229	229
query61	151	150	147	147
query62	691	637	586	586
query63	234	192	192	192
query64	2543	749	611	611
query65	4764	4845	4763	4763
query66	1796	461	335	335
query67	28942	28803	28505	28505
query68	3197	1606	909	909
query69	409	295	262	262
query70	1084	927	966	927
query71	287	230	217	217
query72	2945	2596	2334	2334
query73	833	749	446	446
query74	5125	4991	4786	4786
query75	2594	2572	2151	2151
query76	2311	1198	815	815
query77	378	385	291	291
query78	12615	12490	11967	11967
query79	1398	1132	771	771
query80	590	454	368	368
query81	449	279	243	243
query82	582	162	121	121
query83	354	270	247	247
query84	264	147	110	110
query85	848	522	411	411
query86	378	301	274	274
query87	1838	1848	1775	1775
query88	3686	2778	2736	2736
query89	433	386	329	329
query90	2019	173	178	173
query91	170	171	136	136
query92	64	59	55	55
query93	1543	1495	885	885
query94	532	351	292	292
query95	696	476	349	349
query96	1049	741	347	347
query97	2672	2685	2568	2568
query98	216	207	196	196
query99	1181	1178	1028	1028
Total cold run time: 256707 ms
Total hot run time: 170933 ms

hello-stephen · 2026-06-29T06:45:28Z

ClickBench: Total hot run time: 25.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0afd596e5b21461bb4c083d36e3bd96adc98fa4a, data reload: false
query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.26	0.14	0.13
query4	1.61	0.13	0.14
query5	0.25	0.24	0.23
query6	1.27	1.07	1.05
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.32	0.31
query10	0.56	0.59	0.58
query11	0.19	0.14	0.14
query12	0.18	0.15	0.15
query13	0.48	0.48	0.48
query14	1.01	1.01	1.00
query15	0.61	0.59	0.60
query16	0.31	0.31	0.31
query17	1.14	1.15	1.11
query18	0.23	0.21	0.22
query19	1.98	1.96	1.96
query20	0.02	0.01	0.01
query21	15.45	0.24	0.13
query22	4.82	0.05	0.05
query23	16.12	0.31	0.12
query24	3.07	0.43	0.32
query25	0.12	0.06	0.05
query26	0.74	0.20	0.15
query27	0.04	0.04	0.04
query28	3.52	0.90	0.54
query29	12.66	4.35	3.47
query30	0.27	0.14	0.15
query31	2.77	0.60	0.30
query32	3.22	0.61	0.48
query33	3.23	3.21	3.23
query34	15.61	4.19	3.54
query35	3.57	3.49	3.54
query36	0.56	0.42	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.98 s
Total hot run time: 25.3 s

mymeiyi · 2026-06-29T06:47:22Z

/review

github-actions

I found two issues that should be fixed before this lands.

Critical checkpoint conclusions: the PR's goal of reducing group-commit create-plan blocking is only partially met because the new timeout is skipped while a create-plan task is already queued or running. The new mutable config is added, but its runtime enforcement has a gap. The changed group-commit sink filtering/block handoff and WAL rollback status path looked consistent after review. Test coverage was added, but the new p0 stream-load test mutates persistent global root user state from a normal concurrent suite. I did not run the full Doris build or regression suite in this review runner.

Subagent conclusions: TSC-1 became MAIN-1 and is submitted as an inline test-isolation comment. TSC-2 became MAIN-2 and is submitted as an inline timeout-enforcement comment. OPT-1 was dismissed with code evidence because the empty add_rows path preserves the all-filtered 0-row behavior and checked column implementations treat empty ranges as no-ops. Convergence round 3 ended with both live subagents reporting NO_NEW_VALUABLE_FINDINGS for this final two-comment set.

mymeiyi · 2026-06-29T09:20:14Z

run cloud_p0

mymeiyi · 2026-06-29T11:20:22Z

run buildall

hello-stephen · 2026-06-29T12:55:02Z

TPC-H: Total hot run time: 29462 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0afd596e5b21461bb4c083d36e3bd96adc98fa4a, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17644	4049	4049	4049
q2	2031	316	198	198
q3	10294	1442	842	842
q4	4690	473	340	340
q5	7525	871	578	578
q6	181	178	143	143
q7	811	856	626	626
q8	9357	1821	1651	1651
q9	5762	4604	4500	4500
q10	6778	1808	1529	1529
q11	461	286	248	248
q12	637	425	283	283
q13	18079	3580	2788	2788
q14	269	263	237	237
q15	q16	792	780	707	707
q17	1048	906	985	906
q18	7493	5968	5593	5593
q19	1169	1346	1103	1103
q20	503	406	268	268
q21	5671	2587	2564	2564
q22	450	359	309	309
Total cold run time: 101645 ms
Total hot run time: 29462 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4359	4316	4364	4316
q2	334	362	225	225
q3	4651	5009	4432	4432
q4	2080	2183	1373	1373
q5	4579	4390	4371	4371
q6	241	178	130	130
q7	1807	2241	1741	1741
q8	2679	2332	2168	2168
q9	8241	8482	8211	8211
q10	4880	4744	4294	4294
q11	619	439	391	391
q12	745	779	552	552
q13	3363	3732	2948	2948
q14	305	291	291	291
q15	q16	742	738	634	634
q17	1412	1355	1363	1355
q18	8253	7402	7300	7300
q19	1200	1174	1127	1127
q20	2236	2226	1979	1979
q21	5376	4739	4539	4539
q22	519	454	396	396
Total cold run time: 58621 ms
Total hot run time: 52773 ms

hello-stephen · 2026-06-29T13:05:55Z

TPC-DS: Total hot run time: 172557 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0afd596e5b21461bb4c083d36e3bd96adc98fa4a, data reload: false
query5	4319	624	474	474
query6	443	199	168	168
query7	4868	542	316	316
query8	334	179	169	169
query9	8784	4124	4136	4124
query10	451	305	250	250
query11	5919	2318	2130	2130
query12	160	101	97	97
query13	1258	665	482	482
query14	6212	5276	4956	4956
query14_1	4288	4309	4276	4276
query15	223	204	179	179
query16	1021	469	442	442
query17	951	744	579	579
query18	2429	488	350	350
query19	209	192	156	156
query20	111	109	115	109
query21	220	140	114	114
query22	13628	13640	13528	13528
query23	17488	16576	16227	16227
query23_1	16224	16336	16357	16336
query24	7455	1761	1335	1335
query24_1	1334	1351	1296	1296
query25	559	490	355	355
query26	1301	330	169	169
query27	2681	549	333	333
query28	4477	2054	2033	2033
query29	1045	603	475	475
query30	313	229	195	195
query31	1126	1063	952	952
query32	103	64	58	58
query33	507	317	238	238
query34	1203	1132	645	645
query35	762	787	662	662
query36	1399	1449	1191	1191
query37	154	106	89	89
query38	1883	1713	1669	1669
query39	921	936	884	884
query39_1	865	878	862	862
query40	217	120	99	99
query41	66	63	61	61
query42	87	85	86	85
query43	318	327	277	277
query44	1421	795	785	785
query45	197	188	176	176
query46	1042	1199	779	779
query47	2280	2363	2226	2226
query48	411	432	291	291
query49	583	424	306	306
query50	998	356	267	267
query51	4409	4421	4330	4330
query52	83	82	69	69
query53	250	264	185	185
query54	275	217	193	193
query55	80	74	65	65
query56	242	227	224	224
query57	1438	1408	1285	1285
query58	233	195	201	195
query59	1586	1682	1436	1436
query60	282	245	219	219
query61	149	180	149	149
query62	694	649	571	571
query63	224	183	192	183
query64	2501	743	591	591
query65	4874	4739	4771	4739
query66	1792	467	332	332
query67	29911	29648	29560	29560
query68	3030	1613	951	951
query69	402	300	260	260
query70	1106	984	964	964
query71	292	233	216	216
query72	2829	2854	2324	2324
query73	866	795	399	399
query74	5114	4954	4783	4783
query75	2573	2534	2184	2184
query76	2303	1204	795	795
query77	349	379	284	284
query78	12347	12280	11845	11845
query79	1281	1148	790	790
query80	524	478	390	390
query81	444	282	243	243
query82	240	155	120	120
query83	272	267	247	247
query84	289	141	113	113
query85	821	533	424	424
query86	316	313	290	290
query87	1832	1836	1772	1772
query88	3723	2791	2782	2782
query89	418	388	334	334
query90	2140	185	178	178
query91	172	154	129	129
query92	64	60	60	60
query93	1427	1457	890	890
query94	549	357	320	320
query95	675	463	356	356
query96	1140	852	322	322
query97	2691	2747	2538	2538
query98	203	203	199	199
query99	1176	1190	1041	1041
Total cold run time: 255984 ms
Total hot run time: 172557 ms

hello-stephen · 2026-06-29T13:10:51Z

ClickBench: Total hot run time: 25.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0afd596e5b21461bb4c083d36e3bd96adc98fa4a, data reload: false
query1	0.01	0.00	0.00
query2	0.10	0.05	0.05
query3	0.26	0.13	0.16
query4	1.62	0.14	0.14
query5	0.25	0.22	0.22
query6	1.26	1.08	1.09
query7	0.04	0.01	0.01
query8	0.05	0.03	0.04
query9	0.37	0.32	0.31
query10	0.56	0.53	0.52
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.47	0.48	0.48
query14	1.00	1.01	1.01
query15	0.62	0.59	0.62
query16	0.32	0.32	0.31
query17	1.10	1.16	1.11
query18	0.21	0.22	0.21
query19	2.09	2.03	1.92
query20	0.01	0.01	0.01
query21	15.44	0.20	0.15
query22	5.00	0.05	0.06
query23	16.14	0.31	0.11
query24	2.98	0.42	0.29
query25	0.13	0.05	0.04
query26	0.74	0.20	0.17
query27	0.04	0.04	0.03
query28	3.49	0.88	0.52
query29	12.48	4.30	3.48
query30	0.27	0.15	0.17
query31	2.76	0.59	0.32
query32	3.24	0.60	0.48
query33	3.23	3.19	3.22
query34	15.71	4.21	3.50
query35	3.52	3.53	3.52
query36	0.55	0.45	0.43
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.16	0.16
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 96.96 s
Total hot run time: 25.22 s

hello-stephen · 2026-06-29T14:40:34Z

BE Regression && UT Coverage Report

Increment line coverage 10.94% (29/265) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	64.36% (24719/38410)
Line Coverage	48.08% (257015/534596)
Region Coverage	44.86% (212439/473610)
Branch Coverage	45.92% (92213/200795)

hello-stephen · 2026-06-29T14:43:12Z

BE Regression && UT Coverage Report

Increment line coverage 10.94% (29/265) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	64.36% (24719/38410)
Line Coverage	48.08% (257015/534596)
Region Coverage	44.86% (212439/473610)
Branch Coverage	45.92% (92213/200795)

hello-stephen · 2026-06-29T14:48:02Z

BE Regression && UT Coverage Report

Increment line coverage 10.94% (29/265) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	64.37% (24723/38410)
Line Coverage	48.08% (257010/534596)
Region Coverage	44.86% (212445/473610)
Branch Coverage	45.93% (92218/200795)

github-actions · 2026-07-01T03:34:24Z

PR approved by at least one committer and no changes requested.

dimiss

Under high-concurrency async stream load, group commit may fail to get a block queue when creating the group commit plan fragment fails or leaves pending load requests waiting for a queue that is no longer usable. This change tracks pending create-plan requests per table, adds a background worker to resubmit group commit plan creation, and introduces group_commit_create_plan_timeout_ms so waiting requests can be released after a bounded time.

…65078) Cherry-picked from #63722 Co-authored-by: meiyi <meiyi@selectdb.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#63722 Problem Summary: Group Commit cancellation reconstructed BlockData from the current Block and could subtract a changed byte size from the shared queue memory counter, causing size_t underflow and persistent backpressure. ### Release note Fix Group Commit requests becoming blocked after a canceled load corrupts queue memory accounting. ### Check List (For Author) - Test: Unit Test added; not run per request. Static checks passed. - Behavior changed: No. Corrects queue memory accounting during cancellation. - Does this need documentation: No

CopilotAI review requested due to automatic review settings May 27, 2026 06:42

mymeiyi requested review from gavinchou and liaoxin01 as code owners May 27, 2026 06:42

Copilotstarted reviewing on behalf of mymeiyi May 27, 2026 06:42 View session

CopilotAI reviewed May 27, 2026
View reviewed changes

Comment thread...-test/suites/load_p0/stream_load/test_group_commit_stream_load_high_concurrency_async.groovy Outdated
Comment threadbe/src/load/group_commit/group_commit_mgr.cpp Outdated
Comment threadbe/src/common/config.cpp Outdated
Comment threadbe/src/common/config.h Outdated

mymeiyi changed the title ~~[fix](group commit) fix error: can not get a block queue~~[fix](group commit) fix can not get a block queueMay 27, 2026

github-actionsBot requested changes May 27, 2026
View reviewed changes

Comment threadbe/src/load/group_commit/group_commit_mgr.cpp

mymeiyi force-pushed the fix-group-commit-get-queue branch 3 times, most recently from 47e1354 to 250bbcbCompare May 27, 2026 08:42

mymeiyi force-pushed the fix-group-commit-get-queue branch from 5c6c86a to 3e7b817Compare May 28, 2026 10:32

mymeiyi added 3 commits June 4, 2026 17:57

[fix](group commit) fix error: can not get a block queue
4618039

fix case
cddb074

fix sink
0afd596

mymeiyi force-pushed the fix-group-commit-get-queue branch from 3e7b817 to 0afd596Compare June 4, 2026 10:38

github-actionsBot previously requested changes Jun 29, 2026
View reviewed changes

Comment thread...-test/suites/load_p0/stream_load/test_group_commit_stream_load_high_concurrency_async.groovy
Comment threadbe/src/load/group_commit/group_commit_mgr.cpp

gavinchou reviewed Jun 30, 2026
View reviewed changes

Comment threadbe/src/load/group_commit/group_commit_mgr.cpp

gavinchou reviewed Jun 30, 2026
View reviewed changes

Comment threadbe/src/load/group_commit/group_commit_mgr.cpp

gavinchou approved these changes Jul 1, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 1, 2026

mymeiyi added the dev/4.1.x label Jul 1, 2026

mymeiyi merged commit 7b2c4b9 into apache:masterJul 1, 2026
35 of 36 checks passed

github-actionsBot mentioned this pull request Jul 1, 2026
branch-4.1: [fix](group commit) fix can not get a block queue #63722 #65078
Merged

yiguolei pushed a commit that referenced this pull request Jul 2, 2026
branch-4.1: [fix](group commit) fix can not get a block queue#63722(#…
61322e4
…65078) Cherry-picked from #63722 Co-authored-by: meiyi <meiyi@selectdb.com>

yiguolei added dev/4.1.3-merged and removed dev/4.1.x labels Jul 2, 2026

yiguolei mentioned this pull request Jul 7, 2026
4.1.3 Release Note #65306
Closed

wenzhenghu mentioned this pull request Sep 3, 2026
[fix](be) Fix group commit queue memory accounting on cancel HYDCP/hy-doris#95
Merged

Uh oh!

Conversation

mymeiyi commented May 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented May 27, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mymeiyi commented May 27, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mymeiyi commented May 27, 2026

Uh oh!

hello-stephen commented May 27, 2026

Uh oh!

hello-stephen commented May 27, 2026

Uh oh!

hello-stephen commented May 27, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented May 27, 2026

BE Regression && UT Coverage Report

Uh oh!

mymeiyi commented May 28, 2026

Uh oh!

hello-stephen commented May 28, 2026

Uh oh!

hello-stephen commented May 28, 2026

Uh oh!

hello-stephen commented May 28, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented May 28, 2026

BE Regression && UT Coverage Report

Uh oh!

mymeiyi commented Jun 4, 2026

Uh oh!

hello-stephen commented Jun 4, 2026

Uh oh!

hello-stephen commented Jun 4, 2026

BE UT Coverage Report

Uh oh!

mymeiyi commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

mymeiyi commented Jun 29, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mymeiyi commented Jun 29, 2026

Uh oh!

mymeiyi commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

Uh oh!

hello-stephen commented Jun 29, 2026

BE Regression && UT Coverage Report

mymeiyi commented May 27, 2026•
edited
Loading