[fix](blocking queue) introduce condition variable wait timeout to avoid blocking queue deadlock by sollhui · Pull Request #50906 · apache/doris

sollhui · 2025-05-14T09:39:28Z

What problem does this PR solve?

We found some routine load tasks are never finish, and these task hang for blocking queue is deadlock. We observed the stack using GDB and printed the values:

It is clear that both the get thread and put thread are waiting on the condition variable at the same time, causing a deadlock.

However, unfortunately, we have not located the problem yet. we choose introducing timeout to avoid blocking queue deadlock temporarily.If anyone encounters this problem or has any ideas, please feel free to discuss under this PR.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-05-14T09:39:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2025-05-14T09:39:44Z

run buildall

doris-robot · 2025-05-14T11:19:30Z

TPC-H: Total hot run time: 34162 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d875a14585304694076565442d8fa5900f59ff7, data reload: false
------ Round 1 ----------------------------------
q1	26096	5418	5248	5248
q2	2085	299	188	188
q3	10505	1283	727	727
q4	10239	1034	517	517
q5	7790	2451	2301	2301
q6	194	176	137	137
q7	921	763	640	640
q8	9306	1285	1040	1040
q9	6790	5086	5163	5086
q10	6812	2294	1904	1904
q11	473	291	274	274
q12	351	344	208	208
q13	17778	3716	3109	3109
q14	239	246	227	227
q15	553	481	479	479
q16	430	429	382	382
q17	576	853	353	353
q18	7513	7324	7197	7197
q19	1689	967	565	565
q20	337	341	226	226
q21	3802	3268	2412	2412
q22	1057	1002	942	942
Total cold run time: 115536 ms
Total hot run time: 34162 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5509	5426	5416	5416
q2	240	326	230	230
q3	2139	2659	2307	2307
q4	1348	1865	1503	1503
q5	4510	4437	4359	4359
q6	209	167	130	130
q7	2003	1911	1753	1753
q8	2595	2578	2541	2541
q9	7226	7118	7169	7118
q10	3015	3183	2771	2771
q11	596	513	492	492
q12	681	736	593	593
q13	3544	3802	3281	3281
q14	282	310	274	274
q15	525	482	466	466
q16	439	509	436	436
q17	1140	1594	1364	1364
q18	7657	7481	7393	7393
q19	859	781	939	781
q20	2021	2031	1927	1927
q21	5044	4717	4527	4527
q22	1090	999	1003	999
Total cold run time: 52672 ms
Total hot run time: 50661 ms

doris-robot · 2025-05-14T11:31:05Z

TPC-DS: Total hot run time: 186542 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d875a14585304694076565442d8fa5900f59ff7, data reload: false
query1	1003	474	485	474
query2	6569	1889	1880	1880
query3	6750	222	215	215
query4	26150	23653	22999	22999
query5	4305	613	449	449
query6	292	200	189	189
query7	4615	488	299	299
query8	302	251	246	246
query9	8605	2644	2657	2644
query10	489	337	287	287
query11	15542	15076	14911	14911
query12	166	123	106	106
query13	1658	536	423	423
query14	8813	6144	6202	6144
query15	198	185	179	179
query16	7153	637	493	493
query17	1089	711	545	545
query18	1965	393	298	298
query19	186	186	174	174
query20	121	116	118	116
query21	221	127	137	127
query22	4096	4179	3979	3979
query23	34018	33167	33274	33167
query24	8474	2372	2368	2368
query25	549	464	401	401
query26	1241	262	156	156
query27	2779	499	340	340
query28	4348	2139	2125	2125
query29	777	564	426	426
query30	286	216	184	184
query31	912	852	777	777
query32	75	64	66	64
query33	556	365	316	316
query34	794	835	534	534
query35	754	815	736	736
query36	956	963	869	869
query37	115	104	75	75
query38	4221	4223	4184	4184
query39	1477	1392	1401	1392
query40	213	120	106	106
query41	58	57	56	56
query42	131	107	109	107
query43	516	515	493	493
query44	1289	811	810	810
query45	178	170	166	166
query46	842	1010	630	630
query47	1765	1812	1698	1698
query48	402	436	321	321
query49	768	538	438	438
query50	648	667	405	405
query51	4134	4131	4136	4131
query52	112	106	102	102
query53	222	251	186	186
query54	593	588	515	515
query55	88	86	80	80
query56	312	297	282	282
query57	1164	1146	1110	1110
query58	277	262	256	256
query59	2624	2776	2545	2545
query60	353	342	330	330
query61	149	149	147	147
query62	813	753	659	659
query63	229	199	192	192
query64	4466	1116	678	678
query65	4353	4267	4250	4250
query66	1155	412	315	315
query67	15957	15706	15361	15361
query68	8000	885	529	529
query69	474	304	264	264
query70	1157	1073	1090	1073
query71	471	321	295	295
query72	5594	4728	4872	4728
query73	719	659	354	354
query74	8905	9173	8979	8979
query75	3766	3266	2708	2708
query76	3628	1172	756	756
query77	803	427	286	286
query78	10079	10135	9329	9329
query79	2145	808	577	577
query80	595	502	453	453
query81	508	252	223	223
query82	455	129	107	107
query83	257	257	244	244
query84	272	109	87	87
query85	793	353	310	310
query86	391	302	279	279
query87	4417	4508	4277	4277
query88	3425	2292	2309	2292
query89	384	320	287	287
query90	1837	213	208	208
query91	143	144	111	111
query92	80	66	56	56
query93	1595	953	587	587
query94	658	421	309	309
query95	370	292	282	282
query96	492	576	281	281
query97	3176	3242	3141	3141
query98	246	217	206	206
query99	1565	1413	1264	1264
Total cold run time: 273831 ms
Total hot run time: 186542 ms

doris-robot · 2025-05-14T11:36:15Z

ClickBench: Total hot run time: 29.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d875a14585304694076565442d8fa5900f59ff7, data reload: false
query1	0.04	0.03	0.03
query2	0.13	0.11	0.12
query3	0.25	0.20	0.18
query4	1.59	0.20	0.18
query5	0.60	0.59	0.59
query6	1.20	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.52	0.52
query10	0.57	0.58	0.58
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.59
query14	0.77	0.80	0.80
query15	0.88	0.85	0.86
query16	0.38	0.39	0.38
query17	1.08	1.05	1.07
query18	0.22	0.22	0.21
query19	1.90	1.87	1.81
query20	0.01	0.01	0.01
query21	15.44	0.88	0.53
query22	0.75	1.17	0.64
query23	15.08	1.34	0.61
query24	6.45	1.63	0.57
query25	0.46	0.27	0.06
query26	0.55	0.16	0.14
query27	0.05	0.05	0.05
query28	9.26	0.90	0.44
query29	12.55	4.08	3.33
query30	0.26	0.11	0.07
query31	2.82	0.60	0.38
query32	3.23	0.55	0.48
query33	3.04	3.11	3.06
query34	15.93	5.13	4.51
query35	4.54	4.59	4.56
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.12
query41	0.07	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 102.76 s
Total hot run time: 29.11 s

doris-robot · 2025-05-14T13:07:09Z

BE UT Coverage Report

Increment line coverage 54.55% (18/33) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.81% (14901/26700)
Line Coverage	44.60% (131882/295722)
Region Coverage	43.65% (66294/151892)
Branch Coverage	38.26% (33970/88788)

hello-stephen · 2025-05-14T16:07:03Z

BE Regression && UT Coverage Report

Increment line coverage 84.85% (28/33) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.32% (20835/26266)
Line Coverage	72.52% (214415/295658)
Region Coverage	70.71% (126150/178397)
Branch Coverage	64.42% (65317/101398)

sollhui · 2025-05-15T06:40:06Z

run buildall

sollhui · 2025-05-15T08:31:51Z

run buildall

dataroaring

LGTM

github-actions · 2025-05-15T08:45:04Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-15T08:45:06Z

PR approved by anyone and no changes requested.

dataroaring · 2025-05-18T14:39:12Z

run buildall

sollhui · 2025-05-26T07:56:55Z

run buildall

hello-stephen · 2025-05-26T08:07:16Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.25% (1113/1337)
Line Coverage	66.14% (18666/28224)
Region Coverage	65.75% (9257/14079)
Branch Coverage	55.60% (4985/8966)

sollhui · 2025-05-26T08:14:25Z

run buildall

sollhui · 2025-05-26T08:18:59Z

run buildall

sollhui · 2025-06-05T14:32:39Z

run buildall

sollhui · 2025-06-05T17:17:16Z

run buildall

sollhui · 2025-06-05T17:18:02Z

run buildall

doris-robot · 2025-06-05T19:02:37Z

TPC-H: Total hot run time: 33988 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 073649b8b789d2fc3dead10e132ad203cff52589, data reload: false
------ Round 1 ----------------------------------
q1	26562	5243	5068	5068
q2	1993	291	189	189
q3	10359	1282	744	744
q4	10248	998	540	540
q5	7556	2405	2344	2344
q6	186	164	131	131
q7	934	755	632	632
q8	9303	1323	1101	1101
q9	6786	5126	5044	5044
q10	6886	2310	1908	1908
q11	495	295	282	282
q12	358	375	221	221
q13	17771	3687	3098	3098
q14	241	222	217	217
q15	580	481	484	481
q16	449	441	381	381
q17	610	873	368	368
q18	7571	7230	7172	7172
q19	1684	972	572	572
q20	338	340	228	228
q21	3805	3145	2341	2341
q22	1038	996	926	926
Total cold run time: 115753 ms
Total hot run time: 33988 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5218	5100	5129	5100
q2	234	314	222	222
q3	2190	2679	2259	2259
q4	1370	1787	1367	1367
q5	4494	4398	4388	4388
q6	216	174	129	129
q7	2061	1938	1774	1774
q8	2617	2577	2607	2577
q9	7232	7148	7034	7034
q10	3000	3175	2749	2749
q11	589	513	488	488
q12	699	772	652	652
q13	3555	3892	3347	3347
q14	290	291	275	275
q15	536	498	487	487
q16	440	502	457	457
q17	1173	1518	1398	1398
q18	7828	7678	7539	7539
q19	840	942	1100	942
q20	1980	2000	1831	1831
q21	4842	4458	4393	4393
q22	1084	1082	997	997
Total cold run time: 52488 ms
Total hot run time: 50405 ms

doris-robot · 2025-06-05T19:14:48Z

TPC-DS: Total hot run time: 192627 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 073649b8b789d2fc3dead10e132ad203cff52589, data reload: false
query1	1423	1071	1074	1071
query2	6244	1841	1819	1819
query3	11034	4420	4426	4420
query4	53302	25410	23690	23690
query5	5156	540	476	476
query6	370	216	195	195
query7	4987	533	298	298
query8	295	235	235	235
query9	6098	2672	2658	2658
query10	441	315	276	276
query11	15087	15032	14792	14792
query12	162	108	109	108
query13	1091	516	416	416
query14	10153	6375	6229	6229
query15	211	203	176	176
query16	7126	666	496	496
query17	1068	728	602	602
query18	1566	419	298	298
query19	200	190	168	168
query20	125	131	117	117
query21	200	124	109	109
query22	4399	4395	4184	4184
query23	34138	33268	33455	33268
query24	6556	2437	2404	2404
query25	461	466	403	403
query26	687	286	155	155
query27	2241	525	348	348
query28	2997	2201	2163	2163
query29	572	570	428	428
query30	266	218	187	187
query31	888	820	795	795
query32	73	62	70	62
query33	466	356	315	315
query34	765	904	544	544
query35	806	839	749	749
query36	957	1012	908	908
query37	113	98	75	75
query38	4208	4317	4305	4305
query39	1508	1472	1447	1447
query40	211	123	109	109
query41	61	58	54	54
query42	134	114	108	108
query43	506	517	482	482
query44	1343	868	870	868
query45	183	176	179	176
query46	878	1044	657	657
query47	1837	1866	1778	1778
query48	411	442	335	335
query49	663	501	403	403
query50	666	691	403	403
query51	4260	4283	4165	4165
query52	107	109	105	105
query53	229	263	182	182
query54	577	587	505	505
query55	95	89	88	88
query56	300	305	299	299
query57	1160	1189	1102	1102
query58	285	266	264	264
query59	2641	2690	2685	2685
query60	324	311	293	293
query61	133	146	152	146
query62	705	710	681	681
query63	225	195	187	187
query64	1576	1135	661	661
query65	4207	4178	4189	4178
query66	730	392	305	305
query67	15662	15618	15401	15401
query68	6886	887	510	510
query69	532	322	324	322
query70	1197	1113	1152	1113
query71	517	317	296	296
query72	5669	4797	4933	4797
query73	1368	690	361	361
query74	8972	8971	9233	8971
query75	3852	3203	2719	2719
query76	4222	1202	762	762
query77	632	372	301	301
query78	10206	10009	9305	9305
query79	2605	834	571	571
query80	699	502	449	449
query81	479	258	221	221
query82	479	127	94	94
query83	352	255	237	237
query84	297	109	96	96
query85	777	362	392	362
query86	458	296	293	293
query87	4361	4514	4364	4364
query88	3241	2255	2254	2254
query89	399	315	292	292
query90	1800	210	205	205
query91	143	140	109	109
query92	73	62	53	53
query93	2274	947	565	565
query94	656	414	297	297
query95	362	283	282	282
query96	500	576	277	277
query97	2740	2774	2627	2627
query98	233	209	199	199
query99	1464	1407	1314	1314
Total cold run time: 296994 ms
Total hot run time: 192627 ms

doris-robot · 2025-06-05T19:19:57Z

ClickBench: Total hot run time: 28.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 073649b8b789d2fc3dead10e132ad203cff52589, data reload: false
query1	0.04	0.03	0.03
query2	0.12	0.11	0.11
query3	0.25	0.20	0.19
query4	1.59	0.19	0.20
query5	0.44	0.43	0.44
query6	1.45	0.66	0.65
query7	0.03	0.01	0.02
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.58	0.59	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.80	0.81	0.80
query15	0.89	0.85	0.86
query16	0.40	0.38	0.37
query17	1.04	1.07	1.04
query18	0.23	0.21	0.20
query19	1.87	1.84	1.80
query20	0.01	0.01	0.02
query21	15.40	0.92	0.55
query22	0.77	1.18	0.63
query23	14.98	1.37	0.60
query24	7.04	1.37	0.48
query25	0.50	0.26	0.07
query26	0.60	0.16	0.14
query27	0.05	0.05	0.05
query28	9.06	0.89	0.45
query29	12.56	3.98	3.34
query30	0.25	0.09	0.06
query31	2.82	0.60	0.38
query32	3.22	0.56	0.47
query33	3.03	3.20	3.06
query34	15.91	5.11	4.44
query35	4.49	4.51	4.52
query36	0.65	0.50	0.48
query37	0.09	0.07	0.07
query38	0.04	0.05	0.04
query39	0.03	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.12 s
Total hot run time: 28.64 s

sollhui · 2025-06-06T06:56:04Z

run buildall

…dlock

sollhui · 2025-06-09T07:54:29Z

run buildall

doris-robot · 2025-06-09T09:43:32Z

TPC-H: Total hot run time: 33907 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b549abc45551cea0b5fccdc4e2b9825f9bcc7c14, data reload: false
------ Round 1 ----------------------------------
q1	26570	5039	5072	5039
q2	1989	279	186	186
q3	10374	1265	699	699
q4	10225	1003	541	541
q5	7707	2430	2326	2326
q6	185	181	137	137
q7	927	758	619	619
q8	9331	1291	1041	1041
q9	6742	5148	5137	5137
q10	6830	2389	1942	1942
q11	499	286	278	278
q12	349	350	228	228
q13	17763	3719	3102	3102
q14	239	239	219	219
q15	572	477	487	477
q16	430	430	370	370
q17	608	862	368	368
q18	7691	7363	7062	7062
q19	1786	966	582	582
q20	337	345	225	225
q21	3686	3282	2335	2335
q22	1068	1014	994	994
Total cold run time: 115908 ms
Total hot run time: 33907 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5193	5075	5111	5075
q2	241	323	220	220
q3	2191	2678	2312	2312
q4	1335	1764	1459	1459
q5	4573	4439	4338	4338
q6	222	173	130	130
q7	2188	1922	1799	1799
q8	2616	2544	2501	2501
q9	7287	7193	7165	7165
q10	2974	3199	2747	2747
q11	601	529	499	499
q12	685	795	623	623
q13	3550	3954	3374	3374
q14	286	310	274	274
q15	539	482	473	473
q16	442	490	446	446
q17	1165	1570	1384	1384
q18	7783	7645	7515	7515
q19	851	858	946	858
q20	1997	2047	1874	1874
q21	4768	4383	4388	4383
q22	1046	1034	979	979
Total cold run time: 52533 ms
Total hot run time: 50428 ms

doris-robot · 2025-06-09T09:55:07Z

TPC-DS: Total hot run time: 186987 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b549abc45551cea0b5fccdc4e2b9825f9bcc7c14, data reload: false
query1	1011	479	490	479
query2	6575	1809	1809	1809
query3	6746	229	222	222
query4	26281	23803	23525	23525
query5	4392	656	471	471
query6	304	215	217	215
query7	4625	505	302	302
query8	270	226	213	213
query9	8622	2655	2623	2623
query10	496	348	270	270
query11	15450	15200	15017	15017
query12	159	111	105	105
query13	1663	535	421	421
query14	8763	6178	6157	6157
query15	199	207	170	170
query16	7210	632	461	461
query17	1175	718	553	553
query18	1989	389	299	299
query19	189	200	160	160
query20	120	124	113	113
query21	213	130	109	109
query22	4058	4219	4067	4067
query23	34097	33135	33200	33135
query24	7802	2403	2400	2400
query25	536	454	413	413
query26	1232	276	154	154
query27	2713	500	352	352
query28	4261	2138	2093	2093
query29	799	560	435	435
query30	284	215	188	188
query31	925	900	788	788
query32	83	69	72	69
query33	588	383	314	314
query34	828	878	531	531
query35	791	821	751	751
query36	967	1005	883	883
query37	120	102	79	79
query38	4192	4166	4127	4127
query39	1526	1411	1428	1411
query40	217	125	111	111
query41	69	66	68	66
query42	128	112	110	110
query43	505	480	466	466
query44	1326	841	831	831
query45	186	182	179	179
query46	855	1032	638	638
query47	1788	1813	1736	1736
query48	389	439	320	320
query49	784	485	407	407
query50	660	681	404	404
query51	4121	4261	4139	4139
query52	108	103	98	98
query53	221	253	182	182
query54	570	568	502	502
query55	83	88	90	88
query56	311	297	291	291
query57	1189	1200	1170	1170
query58	269	257	310	257
query59	2578	2623	2469	2469
query60	327	320	301	301
query61	130	127	125	125
query62	797	717	657	657
query63	227	194	187	187
query64	4395	1013	699	699
query65	4263	4159	4182	4159
query66	1162	463	326	326
query67	15826	15587	15384	15384
query68	8276	906	536	536
query69	474	311	262	262
query70	1241	1129	1140	1129
query71	481	328	313	313
query72	5912	4764	4771	4764
query73	709	615	367	367
query74	9198	8947	8840	8840
query75	3810	3210	2813	2813
query76	3664	1209	745	745
query77	796	370	286	286
query78	10037	10098	9365	9365
query79	2379	816	590	590
query80	620	525	450	450
query81	506	262	224	224
query82	664	130	102	102
query83	257	258	247	247
query84	254	111	93	93
query85	842	358	321	321
query86	389	305	283	283
query87	4458	4485	4429	4429
query88	3540	2339	2324	2324
query89	389	327	286	286
query90	1876	211	214	211
query91	143	160	117	117
query92	80	65	60	60
query93	1545	976	599	599
query94	668	419	316	316
query95	378	302	286	286
query96	503	569	282	282
query97	2736	2787	2671	2671
query98	234	210	208	208
query99	1432	1430	1282	1282
Total cold run time: 274229 ms
Total hot run time: 186987 ms

doris-robot · 2025-06-09T10:00:20Z

ClickBench: Total hot run time: 28.88 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b549abc45551cea0b5fccdc4e2b9825f9bcc7c14, data reload: false
query1	0.04	0.04	0.03
query2	0.14	0.10	0.11
query3	0.26	0.19	0.18
query4	1.59	0.20	0.10
query5	0.45	0.42	0.42
query6	1.19	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.52	0.52
query10	0.57	0.59	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.61	0.60
query14	0.79	0.81	0.82
query15	0.89	0.87	0.87
query16	0.40	0.38	0.37
query17	1.07	1.05	1.02
query18	0.23	0.21	0.21
query19	1.94	1.82	1.94
query20	0.01	0.00	0.01
query21	15.42	0.90	0.57
query22	0.77	1.32	0.71
query23	14.77	1.40	0.64
query24	6.60	1.59	0.67
query25	0.52	0.18	0.06
query26	0.65	0.16	0.13
query27	0.06	0.06	0.05
query28	9.17	0.95	0.44
query29	12.59	4.01	3.28
query30	0.24	0.09	0.07
query31	2.82	0.60	0.39
query32	3.22	0.55	0.46
query33	3.12	3.12	3.09
query34	15.84	5.16	4.48
query35	4.50	4.50	4.49
query36	0.66	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.58 s
Total hot run time: 28.88 s

doris-robot · 2025-06-09T14:36:53Z

BE UT Coverage Report

Increment line coverage 64.71% (11/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.40% (15071/26720)
Line Coverage	45.16% (134670/298197)
Region Coverage	44.26% (67769/153105)
Branch Coverage	38.82% (34750/89510)

hello-stephen · 2025-06-09T16:09:57Z

BE Regression && UT Coverage Report

Increment line coverage 76.47% (13/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.74% (20972/26299)
Line Coverage	72.74% (216804/298051)
Region Coverage	70.96% (127803/180100)
Branch Coverage	64.63% (66132/102322)

liaoxin01

LGTM

github-actions · 2025-06-10T01:57:51Z

PR approved by at least one committer and no changes requested.

dataroaring

LGTM

…oid blocking queue deadlock (#50906) ### What problem does this PR solve? We found some routine load tasks are never finish, and these task hang for **blocking queue** is deadlock. We observed the stack using GDB and printed the values: ![image](https://github.com/user-attachments/assets/729de36d-3bcc-46e7-b749-90e1c781a9fa) ![image](https://github.com/user-attachments/assets/f5e68040-d936-468d-bdb9-266136026b70) It is clear that both the get thread and put thread are waiting on the condition variable at the same time, causing a deadlock. However, unfortunately, we have not located the problem yet. we choose introducing timeout to avoid blocking queue deadlock temporarily.If anyone encounters this problem or has any ideas, please feel free to discuss under this PR.

…imeout to avoid blocking queue deadlock #50906 (#51596) Cherry-picked from #50906 Co-authored-by: hui lai <laihui@selectdb.com>

sollhui force-pushed the fix_load_task_hang branch from 2d875a1 to 183d248Compare May 15, 2025 03:45

sollhui changed the title ~~[fix](load) use non-blocking get to avoid load task hang~~[fix](blocking queue) introduce timeout to avoid blocking queue deadlockMay 15, 2025

sollhui force-pushed the fix_load_task_hang branch from 183d248 to c669056Compare May 15, 2025 08:31

dataroaring previously approved these changes May 15, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 15, 2025

github-actionsBot added the reviewed label May 15, 2025

dataroaring added dev/2.1.x dev/3.0.x labels May 18, 2025

dataroaring force-pushed the fix_load_task_hang branch from c669056 to 8662e67Compare May 18, 2025 14:39

yiguolei reviewed May 19, 2025
View reviewed changes

Comment threadbe/src/runtime/routine_load/data_consumer.cpp Outdated

sollhui dismissed dataroaring’s stale review via 82c6e43May 26, 2025 07:56

sollhui force-pushed the fix_load_task_hang branch from 8662e67 to 82c6e43Compare May 26, 2025 07:56

sollhui changed the title ~~[fix](blocking queue) introduce timeout to avoid blocking queue deadlock~~[fix](blocking queue) introduce condition variable timeout to avoid blocking queue deadlockMay 26, 2025

sollhui force-pushed the fix_load_task_hang branch from 82c6e43 to 6b32088Compare May 26, 2025 08:14

sollhui force-pushed the fix_load_task_hang branch from 7233b10 to 3cf402fCompare June 5, 2025 17:16

sollhui force-pushed the fix_load_task_hang branch from 3cf402f to 073649bCompare June 5, 2025 17:17

introduce condition variable wait timeout to avoid blocking queue dea…
b549abc
…dlock

sollhui force-pushed the fix_load_task_hang branch from 6a06163 to b549abcCompare June 9, 2025 07:54

liaoxin01 approved these changes Jun 10, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 10, 2025

dataroaring approved these changes Jun 10, 2025
View reviewed changes

dataroaring merged commit 46d2a8a into apache:masterJun 10, 2025

github-actionsBot mentioned this pull request Jun 10, 2025
branch-3.0: [fix](blocking queue) introduce condition variable wait timeout to avoid blocking queue deadlock #50906 #51596
Merged

github-actionsBot mentioned this pull request Jun 10, 2025
branch-2.1: [fix](blocking queue) introduce condition variable wait timeout to avoid blocking queue deadlock #50906 #51597
Closed

dataroaring pushed a commit that referenced this pull request Jun 11, 2025
branch-3.0: [fix](blocking queue) introduce condition variable wait t…
327c545
…imeout to avoid blocking queue deadlock #50906 (#51596) Cherry-picked from #50906 Co-authored-by: hui lai <laihui@selectdb.com>

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jun 11, 2025

morrySnow added the dev/3.1.0-merged label Jun 17, 2025

Uh oh!

Conversation

sollhui commented May 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented May 14, 2025

Uh oh!

sollhui commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 14, 2025

BE Regression && UT Coverage Report

Uh oh!

sollhui commented May 15, 2025

Uh oh!

sollhui commented May 15, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 15, 2025

Uh oh!

github-actionsBot commented May 15, 2025

Uh oh!

dataroaring commented May 18, 2025

Uh oh!

Uh oh!

sollhui commented May 26, 2025

Uh oh!

hello-stephen commented May 26, 2025

Cloud UT Coverage Report

Uh oh!

sollhui commented May 26, 2025

Uh oh!

sollhui commented May 26, 2025

Uh oh!

sollhui commented Jun 5, 2025

Uh oh!

sollhui commented Jun 5, 2025

Uh oh!

sollhui commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

sollhui commented Jun 6, 2025

Uh oh!

sollhui commented Jun 9, 2025

Uh oh!

doris-robot commented Jun 9, 2025

Uh oh!

doris-robot commented Jun 9, 2025

Uh oh!

doris-robot commented Jun 9, 2025

Uh oh!

doris-robot commented Jun 9, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 9, 2025

BE Regression && UT Coverage Report

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 10, 2025

sollhui commented May 14, 2025•
edited
Loading