[fix](load)fix VNodeChannel close_wait hang by koarz · Pull Request #58024 · apache/doris

koarz · 2025-11-14T07:29:24Z

During streamload execution, close_wait waits for either add_batches_finished or cancel. When writing to a single replica, encountering a slave node=nullptr condition causes an early return. If no RPC callback is set and no cancel occurs at this point, close_wait becomes deadlock.

Thearas · 2025-11-14T07:29:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

koarz · 2025-11-14T07:29:33Z

run buildall

koarz · 2025-11-14T07:44:16Z

run buildall

doris-robot · 2025-11-14T08:14:00Z

TPC-H: Total hot run time: 34043 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 57cfcbaca895d78f404dbaaf75e15774bf45747e, data reload: false
------ Round 1 ----------------------------------
q1	17584	5123	5011	5011
q2	2083	323	205	205
q3	10234	1258	703	703
q4	10223	872	377	377
q5	7543	2390	2315	2315
q6	176	164	133	133
q7	914	735	618	618
q8	9348	1305	1126	1126
q9	6823	5145	5159	5145
q10	6857	2225	1830	1830
q11	503	304	279	279
q12	357	371	229	229
q13	17787	3604	2979	2979
q14	220	231	215	215
q15	567	516	517	516
q16	971	987	944	944
q17	583	874	349	349
q18	7405	7029	7052	7029
q19	1358	936	568	568
q20	350	339	232	232
q21	3993	3175	2271	2271
q22	1080	1050	969	969
Total cold run time: 106959 ms
Total hot run time: 34043 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5166	5317	5085	5085
q2	247	329	261	261
q3	2124	2628	2263	2263
q4	1349	1770	1323	1323
q5	4201	4366	4411	4366
q6	213	168	128	128
q7	2073	1958	1837	1837
q8	2649	2489	2702	2489
q9	7345	7364	7317	7317
q10	3057	3200	2892	2892
q11	598	521	503	503
q12	669	762	580	580
q13	3696	3778	3205	3205
q14	319	315	282	282
q15	552	515	511	511
q16	1119	1243	1051	1051
q17	1143	1576	1377	1377
q18	7958	7715	7644	7644
q19	794	813	853	813
q20	1949	2076	1901	1901
q21	4722	4330	4352	4330
q22	1124	1052	986	986
Total cold run time: 53067 ms
Total hot run time: 51144 ms

doris-robot · 2025-11-14T08:25:40Z

TPC-DS: Total hot run time: 187459 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 57cfcbaca895d78f404dbaaf75e15774bf45747e, data reload: false
query1	1026	411	390	390
query2	6567	1691	1707	1691
query3	6755	233	227	227
query4	26207	23804	23570	23570
query5	4682	613	467	467
query6	349	243	233	233
query7	4652	484	294	294
query8	304	254	262	254
query9	8726	2553	2601	2553
query10	501	340	296	296
query11	15371	15035	14884	14884
query12	175	119	113	113
query13	1692	577	465	465
query14	10604	9193	9049	9049
query15	194	192	165	165
query16	7519	658	524	524
query17	1210	749	593	593
query18	2020	415	320	320
query19	208	205	172	172
query20	142	127	130	127
query21	213	130	110	110
query22	4213	4180	4024	4024
query23	33635	33016	32847	32847
query24	8455	2359	2386	2359
query25	599	527	492	492
query26	1252	278	163	163
query27	2771	491	358	358
query28	4414	2209	2186	2186
query29	842	645	521	521
query30	307	226	199	199
query31	915	810	706	706
query32	83	73	76	73
query33	603	382	328	328
query34	809	839	533	533
query35	834	837	743	743
query36	951	998	913	913
query37	119	112	88	88
query38	3553	3456	3444	3444
query39	1471	1405	1441	1405
query40	230	133	121	121
query41	71	67	64	64
query42	131	110	112	110
query43	499	502	476	476
query44	1221	748	743	743
query45	190	181	173	173
query46	877	985	646	646
query47	1793	1814	1738	1738
query48	393	421	339	339
query49	800	507	411	411
query50	651	683	407	407
query51	4009	3839	3910	3839
query52	112	111	112	111
query53	237	270	205	205
query54	324	303	299	299
query55	85	91	90	90
query56	339	338	312	312
query57	1195	1190	1128	1128
query58	298	287	279	279
query59	2549	2663	2487	2487
query60	361	405	323	323
query61	164	161	163	161
query62	797	723	669	669
query63	221	193	199	193
query64	4523	1152	848	848
query65	4030	3930	3944	3930
query66	1132	435	330	330
query67	15223	15101	14847	14847
query68	8462	934	592	592
query69	492	319	286	286
query70	1353	1297	1293	1293
query71	480	336	316	316
query72	6137	4874	4826	4826
query73	678	610	367	367
query74	9046	9122	8814	8814
query75	3957	3324	2826	2826
query76	3779	1145	734	734
query77	819	390	304	304
query78	9501	9520	8858	8858
query79	2230	831	617	617
query80	669	578	500	500
query81	506	254	240	240
query82	480	168	135	135
query83	270	262	257	257
query84	255	114	88	88
query85	958	487	446	446
query86	399	300	284	284
query87	3744	3722	3636	3636
query88	3731	2262	2239	2239
query89	397	326	301	301
query90	1889	218	212	212
query91	170	172	147	147
query92	81	72	63	63
query93	1779	989	641	641
query94	709	456	339	339
query95	403	312	319	312
query96	489	585	277	277
query97	2948	2966	2855	2855
query98	244	215	208	208
query99	1407	1385	1288	1288
Total cold run time: 275860 ms
Total hot run time: 187459 ms

doris-robot · 2025-11-14T08:30:47Z

ClickBench: Total hot run time: 27.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 57cfcbaca895d78f404dbaaf75e15774bf45747e, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.27	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.25
query6	1.18	0.66	0.64
query7	0.04	0.03	0.03
query8	0.06	0.04	0.05
query9	0.60	0.53	0.51
query10	0.57	0.58	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	1.02	0.99	1.00
query15	0.86	0.83	0.85
query16	0.39	0.40	0.38
query17	1.03	1.01	1.02
query18	0.21	0.19	0.20
query19	1.94	1.78	1.85
query20	0.02	0.01	0.01
query21	15.44	0.20	0.13
query22	5.10	0.07	0.05
query23	15.68	0.28	0.10
query24	2.36	1.44	0.30
query25	0.07	0.08	0.06
query26	0.14	0.14	0.13
query27	0.07	0.06	0.06
query28	4.24	1.14	0.93
query29	12.59	3.92	3.22
query30	0.28	0.14	0.11
query31	2.82	0.61	0.37
query32	3.23	0.56	0.48
query33	3.04	3.11	3.19
query34	15.85	5.25	4.61
query35	4.54	4.59	4.53
query36	0.66	0.49	0.49
query37	0.09	0.06	0.07
query38	0.06	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.13	0.14
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.79 s
Total hot run time: 27.35 s

koarz · 2025-11-14T08:47:57Z

run buildall

doris-robot · 2025-11-14T09:10:13Z

TPC-H: Total hot run time: 34157 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56baf53c63fa1c5f78ae3df9efeddceaa5df0132, data reload: false
------ Round 1 ----------------------------------
q1	17624	5219	5080	5080
q2	2043	310	219	219
q3	10272	1264	714	714
q4	10227	875	370	370
q5	7531	2391	2321	2321
q6	184	176	137	137
q7	929	768	621	621
q8	9348	1272	1142	1142
q9	6950	5146	5103	5103
q10	6872	2214	1798	1798
q11	490	312	276	276
q12	347	363	220	220
q13	17772	3671	3085	3085
q14	228	231	206	206
q15	582	509	520	509
q16	1021	1004	947	947
q17	591	861	369	369
q18	7809	7075	6988	6988
q19	1629	958	547	547
q20	360	348	225	225
q21	3893	2511	2289	2289
q22	1068	1020	991	991
Total cold run time: 107770 ms
Total hot run time: 34157 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5258	5123	5161	5123
q2	241	355	240	240
q3	2150	2630	2312	2312
q4	1337	1764	1304	1304
q5	4203	4432	4421	4421
q6	206	177	127	127
q7	1977	1989	1863	1863
q8	2647	2530	2507	2507
q9	7348	7339	7298	7298
q10	3089	3331	2775	2775
q11	582	523	519	519
q12	679	767	632	632
q13	3461	3930	3318	3318
q14	279	312	297	297
q15	540	536	493	493
q16	1081	1105	1064	1064
q17	1173	1607	1373	1373
q18	7883	7679	7942	7679
q19	784	796	841	796
q20	1929	1973	1813	1813
q21	4704	4341	4288	4288
q22	1088	1072	1025	1025
Total cold run time: 52639 ms
Total hot run time: 51267 ms

doris-robot · 2025-11-14T09:21:51Z

TPC-DS: Total hot run time: 188658 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56baf53c63fa1c5f78ae3df9efeddceaa5df0132, data reload: false
query1	1021	396	404	396
query2	6562	1681	1713	1681
query3	6758	225	219	219
query4	26500	23765	23526	23526
query5	4355	626	482	482
query6	344	253	234	234
query7	4649	505	302	302
query8	339	262	254	254
query9	8744	2581	2588	2581
query10	468	349	302	302
query11	15660	15103	14866	14866
query12	188	120	114	114
query13	1688	593	456	456
query14	11479	9216	9152	9152
query15	196	189	170	170
query16	7674	686	524	524
query17	1224	764	621	621
query18	2031	421	320	320
query19	213	204	181	181
query20	129	122	120	120
query21	212	128	112	112
query22	4016	4192	4044	4044
query23	34135	33160	33009	33009
query24	8504	2373	2414	2373
query25	604	516	460	460
query26	1226	272	162	162
query27	2758	515	364	364
query28	4414	2207	2191	2191
query29	790	616	502	502
query30	295	223	196	196
query31	910	801	739	739
query32	85	80	83	80
query33	599	384	331	331
query34	820	876	532	532
query35	838	833	738	738
query36	947	1000	906	906
query37	122	115	91	91
query38	3592	3503	3508	3503
query39	1485	1409	1422	1409
query40	222	126	119	119
query41	60	61	63	61
query42	139	118	108	108
query43	490	494	479	479
query44	1241	767	761	761
query45	183	186	174	174
query46	895	1001	637	637
query47	1734	1775	1702	1702
query48	398	458	322	322
query49	765	512	420	420
query50	647	695	415	415
query51	3949	3922	3897	3897
query52	108	106	105	105
query53	240	271	199	199
query54	311	305	276	276
query55	91	85	89	85
query56	347	323	358	323
query57	1188	1169	1102	1102
query58	293	280	281	280
query59	2582	2644	2507	2507
query60	345	349	333	333
query61	158	158	160	158
query62	789	724	717	717
query63	232	198	193	193
query64	4437	1161	847	847
query65	4033	3948	3950	3948
query66	1111	434	343	343
query67	15396	15219	15075	15075
query68	4617	945	636	636
query69	494	347	294	294
query70	1387	1279	1242	1242
query71	445	347	335	335
query72	6115	5196	5157	5157
query73	663	606	380	380
query74	9134	9081	8671	8671
query75	3321	3348	2852	2852
query76	3326	1157	763	763
query77	545	469	319	319
query78	9606	9902	8904	8904
query79	1760	855	616	616
query80	1671	560	511	511
query81	551	260	232	232
query82	404	168	128	128
query83	356	266	254	254
query84	249	118	102	102
query85	916	488	442	442
query86	382	315	300	300
query87	3753	3705	3602	3602
query88	2972	2396	2383	2383
query89	392	333	288	288
query90	1739	220	219	219
query91	168	175	135	135
query92	73	64	67	64
query93	1121	969	658	658
query94	728	430	337	337
query95	428	331	307	307
query96	501	603	303	303
query97	2930	2968	2879	2879
query98	232	216	207	207
query99	1339	1407	1309	1309
Total cold run time: 271276 ms
Total hot run time: 188658 ms

doris-robot · 2025-11-14T09:26:59Z

ClickBench: Total hot run time: 27.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 56baf53c63fa1c5f78ae3df9efeddceaa5df0132, data reload: false
query1	0.06	0.05	0.04
query2	0.09	0.06	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.27	0.25	0.25
query6	1.18	0.66	0.64
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.52	0.52
query10	0.58	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.61	0.60	0.61
query14	1.01	1.01	1.00
query15	0.85	0.83	0.84
query16	0.38	0.38	0.38
query17	1.00	1.02	1.02
query18	0.22	0.20	0.20
query19	1.94	1.87	1.78
query20	0.02	0.01	0.01
query21	15.47	0.22	0.13
query22	5.10	0.06	0.04
query23	15.71	0.26	0.11
query24	2.73	1.34	0.98
query25	0.08	0.07	0.06
query26	0.15	0.13	0.13
query27	0.07	0.06	0.05
query28	5.09	1.12	0.94
query29	12.60	3.89	3.31
query30	0.29	0.13	0.12
query31	2.82	0.57	0.39
query32	3.23	0.56	0.47
query33	3.07	3.00	3.10
query34	15.85	5.20	4.50
query35	4.61	4.55	4.57
query36	0.70	0.51	0.48
query37	0.09	0.07	0.07
query38	0.06	0.03	0.04
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 99.14 s
Total hot run time: 27.97 s

hello-stephen · 2025-11-14T09:58:05Z

BE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.81% (18285/34621)
Line Coverage	38.19% (166282/435450)
Region Coverage	33.17% (129245/389602)
Branch Coverage	33.91% (55468/163582)

hello-stephen · 2025-11-14T11:09:49Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.60% (24359/34021)
Line Coverage	58.07% (253236/436124)
Region Coverage	53.44% (211143/395113)
Branch Coverage	54.71% (90085/164657)

liaoxin01

LGTM

github-actions · 2025-11-24T12:48:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-24T12:48:47Z

PR approved by anyone and no changes requested.

gavinchou · 2025-11-29T16:23:16Z

skip check_coverage

github-actions · 2025-11-29T16:23:16Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-29T16:23:18Z

PR approved by anyone and no changes requested.

During streamload execution, close_wait waits for either add_batches_finished or cancel. When writing to a single replica, encountering a slave node=nullptr condition causes an early return. If no RPC callback is set and no cancel occurs at this point, close_wait becomes deadlock.

Cherry-picked from #58024 Co-authored-by: koarz <66543806+koarz@users.noreply.github.com>

During streamload execution, close_wait waits for either add_batches_finished or cancel. When writing to a single replica, encountering a slave node=nullptr condition causes an early return. If no RPC callback is set and no cancel occurs at this point, close_wait becomes deadlock.

pick: #58024 Co-authored-by: koarz <66543806+koarz@users.noreply.github.com>

Pick #58024 Co-authored-by: koarz <66543806+koarz@users.noreply.github.com>

[fix](load)fix VNodeChannel close_wait hang
bb33efe

1
57cfcba

2
56baf53

liaoxin01 approved these changes Nov 24, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 24, 2025

github-actionsBot added the reviewed label Nov 24, 2025

liaoxin01 added dev/3.1.x dev/4.0.x and removed approved Indicates a PR has been approved by one committer. reviewed labels Nov 24, 2025

gavinchou approved these changes Nov 29, 2025
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 29, 2025

liaoxin01 merged commit 77455ea into apache:masterDec 1, 2025
29 of 30 checks passed

github-actionsBot mentioned this pull request Dec 1, 2025
branch-3.1: [fix](load)fix VNodeChannel close_wait hang #58024 #58583
Merged

github-actionsBot mentioned this pull request Dec 1, 2025
branch-4.0: [fix](load)fix VNodeChannel close_wait hang #58024 #58584
Closed

koarz deleted the fix_vtable_writer branch December 2, 2025 02:59

morrySnow pushed a commit that referenced this pull request Dec 2, 2025
branch-3.1: [fix](load)fix VNodeChannel close_wait hang#58024(#58583)
bbed154
Cherry-picked from #58024 Co-authored-by: koarz <66543806+koarz@users.noreply.github.com>

morrySnow added dev/3.1.4-merged and removed dev/3.1.x labels Dec 2, 2025

wyxxxcat mentioned this pull request Apr 2, 2026
branch-4.0: [fix](load)fix VNodeChannel close_wait hang #58024 #62030
Merged

morningman pushed a commit that referenced this pull request Apr 3, 2026
branch-4.0: [fix](load)fix VNodeChannel close_wait hang#58024(#62030)
ad7204a
pick: #58024 Co-authored-by: koarz <66543806+koarz@users.noreply.github.com>

morningman added dev/4.0.6-merged and removed dev/4.0.x labels Apr 3, 2026

morningman mentioned this pull request Jun 3, 2026
4.0.6 Release Notes #64065
Closed

liaoxin01 mentioned this pull request Jul 6, 2026
branch-4.1: [fix](load)fix VNodeChannel close_wait hang #65241
Merged

yiguolei pushed a commit that referenced this pull request Jul 7, 2026
branch-4.1: [fix](load)fix VNodeChannel close_wait hang (#65241)
444e4cc
Pick #58024 Co-authored-by: koarz <66543806+koarz@users.noreply.github.com>

yiguolei added the dev/4.1.3-merged label Jul 7, 2026

Uh oh!

Conversation

koarz commented Nov 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thearas commented Nov 14, 2025

Uh oh!

koarz commented Nov 14, 2025

Uh oh!

koarz commented Nov 14, 2025

Uh oh!

doris-robot commented Nov 14, 2025

Uh oh!

doris-robot commented Nov 14, 2025

Uh oh!

doris-robot commented Nov 14, 2025

Uh oh!

koarz commented Nov 14, 2025

Uh oh!

doris-robot commented Nov 14, 2025

Uh oh!

doris-robot commented Nov 14, 2025

Uh oh!

doris-robot commented Nov 14, 2025

Uh oh!

hello-stephen commented Nov 14, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Nov 14, 2025

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 24, 2025

Uh oh!

github-actionsBot commented Nov 24, 2025

Uh oh!

gavinchou commented Nov 29, 2025

Uh oh!

github-actionsBot commented Nov 29, 2025

Uh oh!

github-actionsBot commented Nov 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

koarz commented Nov 14, 2025•
edited
Loading