[Fix](grace-exit) Fix threadpool hang on grace exit by zclllyybb · Pull Request #33132 · apache/doris

zclllyybb · 2024-04-01T13:08:42Z

Proposed changes

Issue Number: close #xxx

stack like:

Thread 641 (Thread 0x7f1d6e3ae700 (LWP 3912128) "WriteCooldownMe"):
#0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x614009733320) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x614009733358, cond=0x6140097332f8) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x6140097332f8, mutex=0x614009733358) at pthread_cond_wait.c:647
#3 0x000056102fd56b0c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865
#4 std::__condvar::wait (__m=..., this=<optimized out>) at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#5 std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#6 0x0000560ff926a0de in doris::BlockingPriorityQueue<doris::WorkThreadPool<true>::Task>::blocking_get (this=0x6140097332f0, out=0x7f1d69b0b020, timeout_ms=0) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/blocking_priority_queue.hpp:70
#7 0x0000560ff9264f19 in doris::WorkThreadPool<true>::work_thread (this=0x614009733240, thread_id=0) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/work_thread_pool.hpp:156
#8 0x0000560ff9267665 in std::__invoke_impl<void, void (doris::WorkThreadPool<true>::* const&)(int), doris::WorkThreadPool<true>*&, int&> (__f=@0x604042fb1c98: (void (doris::WorkThreadPool<true>::*)(doris::WorkThreadPool<true> * const, int)) 0x560ff9264d80 <doris::WorkThreadPool<true>::work_thread(int)>, __t=@0x604042fb1cb0: 0x614009733240, __args=@0x604042fb1ca8: 0) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74
#9 0x0000560ff9267527 in std::__invoke<void (doris::WorkThreadPool<true>::* const&)(int), doris::WorkThreadPool<true>*&, int&> (__fn=@0x604042fb1c98: (void (doris::WorkThreadPool<true>::*)(doris::WorkThreadPool<true> * const, int)) 0x560ff9264d80 <doris::WorkThreadPool<true>::work_thread(int)>, __args=@0x604042fb1ca8: 0, __args=@0x604042fb1ca8: 0) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96
#10 0x0000560ff92674e7 in std::_Mem_fn_base<void (doris::WorkThreadPool<true>::*)(int), true>::operator()<doris::WorkThreadPool<true>*&, int&> (this=0x604042fb1c98, __args=@0x604042fb1ca8: 0, __args=@0x604042fb1ca8: 0) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:131
#11 0x0000560ff92674a7 in std::__invoke_impl<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)>&, doris::WorkThreadPool<true>*&, int&> (__f=..., __args=@0x604042fb1ca8: 0, __args=@0x604042fb1ca8: 0) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#12 0x0000560ff92673a9 in std::__invoke_r<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)>&, doris::WorkThreadPool<true>*&, int&> (__fn=..., __args=@0x604042fb1ca8: 0, __args=@0x604042fb1ca8: 0) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111
#13 0x0000560ff92672d6 in std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x604042fb1c98, __args=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570
#14 0x0000560ff9267130 in std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)>::operator()<>() (this=0x604042fb1c98) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:629
#15 0x0000560ff9267037 in std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)>&&) (__f=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#16 0x0000560ff9266fd7 in std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)>&&) (__fn=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96
#17 0x0000560ff9266f9f in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x604042fb1c98) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253
#18 0x0000560ff9266f67 in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)> > >::operator()() (this=0x604042fb1c98) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260
#19 0x0000560ff9266eab in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<true>::*)(int)> (doris::WorkThreadPool<true>*, int)> > > >::_M_run() (this=0x604042fb1c90) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211
#20 0x000056102fdc5aa0 in std::execute_native_thread_routine (__p=0x604042fb1c90) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#21 0x00007f29ad6e9609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#22 0x00007f29ad996133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-01T13:08:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zclllyybb · 2024-04-01T13:09:05Z

run buildall

github-actions

clang-tidy made some suggestions

doris-robot · 2024-04-01T13:28:31Z

TeamCity be ut coverage result:
Function Coverage: 35.55% (8842/24871)
Line Coverage: 27.28% (72512/265775)
Region Coverage: 26.49% (37540/141725)
Branch Coverage: 23.29% (19141/82170)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9761e0dfc61465ea9574efd33a31af097d0a4966_9761e0dfc61465ea9574efd33a31af097d0a4966/report/index.html

doris-robot · 2024-04-01T13:44:30Z

TPC-H: Total hot run time: 38668 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9761e0dfc61465ea9574efd33a31af097d0a4966, data reload: false
------ Round 1 ----------------------------------
q1	17672	4079	4063	4063
q2	2010	187	185	185
q3	10482	1311	1412	1311
q4	10201	901	1045	901
q5	7470	2972	2903	2903
q6	215	131	130	130
q7	1101	608	618	608
q8	9403	2043	2027	2027
q9	6717	6162	6148	6148
q10	8439	3496	3513	3496
q11	414	236	233	233
q12	380	216	215	215
q13	17783	2889	2901	2889
q14	263	247	241	241
q15	532	491	478	478
q16	500	377	381	377
q17	955	921	920	920
q18	7182	6388	6361	6361
q19	1578	1524	1531	1524
q20	602	324	308	308
q21	3558	3048	3086	3048
q22	362	302	309	302
Total cold run time: 107819 ms
Total hot run time: 38668 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4058	4041	4112	4041
q2	340	225	225	225
q3	2970	2948	2921	2921
q4	1836	1829	1823	1823
q5	5229	5173	5219	5173
q6	209	127	125	125
q7	2222	1813	1808	1808
q8	3212	3284	3279	3279
q9	8457	8471	8511	8471
q10	3730	3931	3968	3931
q11	552	474	469	469
q12	765	607	596	596
q13	16583	3064	3076	3064
q14	318	285	285	285
q15	532	492	511	492
q16	494	454	473	454
q17	1741	1744	1729	1729
q18	8301	7752	7846	7752
q19	1686	1685	1683	1683
q20	2029	1853	1832	1832
q21	5000	4944	4889	4889
q22	492	433	458	433
Total cold run time: 70756 ms
Total hot run time: 55475 ms

doris-robot · 2024-04-01T13:56:08Z

TPC-DS: Total hot run time: 182727 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9761e0dfc61465ea9574efd33a31af097d0a4966, data reload: false
query1	1206	1116	1115	1115
query2	6278	1953	1960	1953
query3	6652	216	205	205
query4	23545	21383	21454	21383
query5	4228	406	418	406
query6	286	194	192	192
query7	4599	303	308	303
query8	234	179	180	179
query9	8458	2229	2231	2229
query10	462	249	265	249
query11	14896	14581	14531	14531
query12	140	96	95	95
query13	1649	374	376	374
query14	8470	6905	6905	6905
query15	214	186	176	176
query16	6822	289	269	269
query17	968	598	596	596
query18	1856	299	291	291
query19	210	163	165	163
query20	99	92	91	91
query21	197	138	135	135
query22	5017	4765	4802	4765
query23	33352	32644	32853	32644
query24	13226	3269	3272	3269
query25	718	420	436	420
query26	1806	174	179	174
query27	3326	377	389	377
query28	7253	1881	1891	1881
query29	1356	635	642	635
query30	304	162	163	162
query31	1010	778	789	778
query32	100	68	61	61
query33	715	263	255	255
query34	1279	530	534	530
query35	888	745	750	745
query36	996	884	884	884
query37	274	80	82	80
query38	3606	3618	3635	3618
query39	1686	1597	1626	1597
query40	254	148	152	148
query41	54	47	49	47
query42	116	115	112	112
query43	473	419	421	419
query44	1148	723	726	723
query45	314	284	269	269
query46	1099	839	844	839
query47	1987	1909	1892	1892
query48	404	317	315	315
query49	945	379	390	379
query50	822	419	426	419
query51	6995	6921	6892	6892
query52	126	112	103	103
query53	372	309	309	309
query54	313	239	250	239
query55	86	83	84	83
query56	261	243	250	243
query57	1316	1176	1171	1171
query58	259	234	236	234
query59	2979	2485	2392	2392
query60	252	229	239	229
query61	93	90	87	87
query62	658	452	470	452
query63	314	282	287	282
query64	5812	3039	3113	3039
query65	3102	3040	3017	3017
query66	1298	311	312	311
query67	15573	15128	14919	14919
query68	8394	566	582	566
query69	553	349	335	335
query70	1245	1115	1077	1077
query71	493	280	276	276
query72	6240	2581	2406	2406
query73	819	320	320	320
query74	6759	6413	6316	6316
query75	3332	2291	2288	2288
query76	4923	1081	1223	1081
query77	571	251	250	250
query78	10868	10200	10132	10132
query79	8437	529	535	529
query80	1210	430	430	430
query81	473	241	222	222
query82	712	102	105	102
query83	195	169	162	162
query84	267	90	91	90
query85	1304	293	282	282
query86	411	294	274	274
query87	3679	3518	3502	3502
query88	3639	2275	2262	2262
query89	559	372	371	371
query90	1923	176	192	176
query91	138	104	107	104
query92	64	57	50	50
query93	6780	532	529	529
query94	1012	197	203	197
query95	453	337	337	337
query96	612	264	270	264
query97	2710	2476	2500	2476
query98	238	220	217	217
query99	1324	825	842	825
Total cold run time: 296143 ms
Total hot run time: 182727 ms

doris-robot · 2024-04-01T14:01:33Z

ClickBench: Total hot run time: 30.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9761e0dfc61465ea9574efd33a31af097d0a4966, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.05	0.05
query4	1.68	0.07	0.07
query5	0.47	0.48	0.49
query6	1.14	0.66	0.65
query7	0.02	0.01	0.01
query8	0.06	0.04	0.05
query9	0.56	0.51	0.51
query10	0.56	0.57	0.55
query11	0.16	0.11	0.10
query12	0.14	0.12	0.12
query13	0.63	0.60	0.60
query14	0.78	0.78	0.79
query15	0.85	0.84	0.84
query16	0.35	0.36	0.35
query17	0.97	1.01	0.98
query18	0.26	0.25	0.24
query19	1.83	1.78	1.70
query20	0.02	0.01	0.01
query21	15.42	0.75	0.70
query22	3.11	4.95	2.31
query23	17.61	1.18	1.10
query24	1.40	0.22	0.24
query25	0.13	0.10	0.09
query26	0.26	0.17	0.18
query27	0.09	0.10	0.08
query28	13.86	0.95	0.95
query29	12.62	3.37	3.62
query30	0.26	0.06	0.06
query31	2.86	0.38	0.38
query32	3.28	0.48	0.48
query33	2.87	2.91	2.86
query34	15.50	4.35	4.35
query35	4.39	4.39	4.35
query36	0.67	0.47	0.47
query37	0.19	0.18	0.17
query38	0.18	0.16	0.16
query39	0.04	0.04	0.04
query40	0.19	0.14	0.16
query41	0.10	0.06	0.06
query42	0.06	0.06	0.05
query43	0.04	0.05	0.04
Total cold run time: 105.96 s
Total hot run time: 30.75 s

doris-robot · 2024-04-01T14:04:39Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9761e0dfc61465ea9574efd33a31af097d0a4966 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 16.0 seconds inserted 10000000 Rows, about 625K ops/s

yiguolei

Refactor WriteCooldownMetaExecutors

yiguolei

LGTM

yiguolei · 2024-04-02T03:11:44Z

run buildall

github-actions · 2024-04-02T03:12:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-02T03:12:04Z

PR approved by anyone and no changes requested.

HappenLee

LGTM

doris-robot · 2024-04-02T04:02:56Z

TPC-H: Total hot run time: 38948 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87461e24225aeaa719bf03a79052887465f4fead, data reload: false
------ Round 1 ----------------------------------
q1	17630	4141	4081	4081
q2	2019	192	186	186
q3	10559	1211	1428	1211
q4	10357	868	1022	868
q5	7493	3013	2953	2953
q6	222	137	136	136
q7	1131	634	617	617
q8	9407	2119	2079	2079
q9	6730	6237	6185	6185
q10	8425	3531	3516	3516
q11	416	241	251	241
q12	382	217	214	214
q13	17770	2893	2903	2893
q14	269	236	244	236
q15	529	482	484	482
q16	497	401	387	387
q17	979	911	912	911
q18	7873	6529	6490	6490
q19	1627	1538	1545	1538
q20	614	324	296	296
q21	3612	3118	3127	3118
q22	375	310	316	310
Total cold run time: 108916 ms
Total hot run time: 38948 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4043	4051	4054	4051
q2	334	224	226	224
q3	2971	2983	2986	2983
q4	1912	1871	1822	1822
q5	5253	5245	5223	5223
q6	211	126	126	126
q7	2262	1800	1801	1800
q8	3244	3312	3294	3294
q9	8526	8492	8493	8492
q10	3774	4049	4021	4021
q11	585	467	495	467
q12	762	648	631	631
q13	13675	3204	3093	3093
q14	322	285	273	273
q15	522	477	490	477
q16	516	438	425	425
q17	1792	1744	1745	1744
q18	8286	7811	7799	7799
q19	1738	1688	1676	1676
q20	1933	1775	1863	1775
q21	5252	4985	4904	4904
q22	495	443	452	443
Total cold run time: 68408 ms
Total hot run time: 55743 ms

doris-robot · 2024-04-02T04:14:38Z

TPC-DS: Total hot run time: 182116 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87461e24225aeaa719bf03a79052887465f4fead, data reload: false
query1	1198	364	1124	364
query2	6188	1982	1945	1945
query3	6651	221	208	208
query4	23911	21257	21563	21257
query5	4162	397	408	397
query6	273	195	185	185
query7	4597	319	304	304
query8	237	178	191	178
query9	8438	2290	2283	2283
query10	435	265	267	265
query11	15030	14516	14547	14516
query12	149	97	98	97
query13	1640	376	372	372
query14	8518	7009	7020	7009
query15	216	172	181	172
query16	6730	284	285	284
query17	954	607	584	584
query18	1832	290	295	290
query19	212	164	164	164
query20	99	92	94	92
query21	202	133	132	132
query22	4967	4835	4834	4834
query23	33815	32773	32848	32773
query24	12821	3273	3360	3273
query25	735	443	422	422
query26	1820	179	172	172
query27	3190	390	402	390
query28	7210	1936	1943	1936
query29	1275	619	613	613
query30	317	158	161	158
query31	1020	748	763	748
query32	102	64	62	62
query33	714	252	261	252
query34	1269	521	526	521
query35	859	735	755	735
query36	1054	871	869	869
query37	264	89	84	84
query38	3686	3574	3603	3574
query39	1660	1593	1612	1593
query40	242	142	150	142
query41	48	46	50	46
query42	120	117	115	115
query43	470	409	415	409
query44	1130	772	754	754
query45	283	263	278	263
query46	1102	828	832	828
query47	1995	1890	1902	1890
query48	403	339	332	332
query49	992	379	381	379
query50	829	410	434	410
query51	6869	6693	6761	6693
query52	126	102	104	102
query53	374	301	308	301
query54	321	249	251	249
query55	95	86	74	74
query56	238	233	229	229
query57	1310	1167	1202	1167
query58	253	237	240	237
query59	2602	2333	2492	2333
query60	259	246	242	242
query61	92	90	88	88
query62	660	478	447	447
query63	314	280	282	280
query64	5909	3339	3277	3277
query65	3081	3011	3042	3011
query66	1316	318	311	311
query67	15295	14839	14970	14839
query68	9582	597	606	597
query69	587	333	329	329
query70	1390	1043	1102	1043
query71	515	276	274	274
query72	6567	2576	2449	2449
query73	1598	332	331	331
query74	6850	6290	6293	6290
query75	3722	2298	2264	2264
query76	5456	1144	1232	1144
query77	568	245	251	245
query78	11061	10113	10177	10113
query79	9681	531	540	531
query80	1697	413	419	413
query81	505	221	221	221
query82	335	104	103	103
query83	204	162	163	162
query84	267	88	90	88
query85	962	296	294	294
query86	371	282	264	264
query87	3731	3475	3544	3475
query88	3916	2374	2387	2374
query89	565	385	384	384
query90	2031	179	178	178
query91	138	108	107	107
query92	71	54	49	49
query93	6550	526	540	526
query94	1322	197	201	197
query95	450	341	338	338
query96	613	273	274	273
query97	2652	2508	2436	2436
query98	230	220	216	216
query99	1195	853	855	853
Total cold run time: 300560 ms
Total hot run time: 182116 ms

doris-robot · 2024-04-02T04:20:03Z

ClickBench: Total hot run time: 30.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87461e24225aeaa719bf03a79052887465f4fead, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.05	0.05
query4	1.67	0.06	0.06
query5	0.47	0.48	0.48
query6	1.15	0.66	0.67
query7	0.01	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.51	0.50
query10	0.56	0.56	0.57
query11	0.14	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.60	0.60
query14	0.77	0.79	0.79
query15	0.87	0.83	0.83
query16	0.36	0.36	0.35
query17	0.95	0.98	0.96
query18	0.25	0.24	0.26
query19	1.78	1.69	1.71
query20	0.01	0.01	0.02
query21	15.40	0.77	0.66
query22	3.48	5.55	1.80
query23	17.62	1.34	1.11
query24	1.93	0.22	0.23
query25	0.13	0.09	0.09
query26	0.30	0.17	0.19
query27	0.09	0.09	0.08
query28	13.37	0.95	0.94
query29	12.58	3.51	3.34
query30	0.28	0.06	0.05
query31	2.85	0.38	0.39
query32	3.28	0.47	0.48
query33	2.88	2.83	2.86
query34	15.52	4.36	4.35
query35	4.39	4.39	4.39
query36	0.66	0.48	0.48
query37	0.21	0.18	0.18
query38	0.17	0.17	0.16
query39	0.04	0.04	0.03
query40	0.17	0.17	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.24 s
Total hot run time: 30.15 s

doris-robot · 2024-04-02T04:23:11Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 87461e24225aeaa719bf03a79052887465f4fead with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 15.3 seconds inserted 10000000 Rows, about 653K ops/s

zclllyybb · 2024-04-02T04:58:26Z

run buildall

doris-robot · 2024-04-02T05:59:48Z

TPC-H: Total hot run time: 39054 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11d0030df53dd5ca378b1f56f8355b7bdf4ab088, data reload: false
------ Round 1 ----------------------------------
q1	17671	4134	4095	4095
q2	2015	190	184	184
q3	10473	1257	1390	1257
q4	10205	892	1080	892
q5	7465	3021	2971	2971
q6	218	137	132	132
q7	1121	638	619	619
q8	9407	2074	2054	2054
q9	6625	6201	6165	6165
q10	8466	3526	3522	3522
q11	430	241	234	234
q12	380	220	215	215
q13	17783	2916	2913	2913
q14	282	243	242	242
q15	533	501	485	485
q16	531	382	380	380
q17	963	927	923	923
q18	7309	6485	6537	6485
q19	1591	1544	1540	1540
q20	619	332	315	315
q21	3530	3122	3139	3122
q22	367	309	314	309
Total cold run time: 107984 ms
Total hot run time: 39054 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4056	4048	4074	4048
q2	326	221	228	221
q3	2964	2975	2957	2957
q4	1923	1923	1836	1836
q5	5287	5249	5249	5249
q6	206	124	125	124
q7	2258	1816	1800	1800
q8	3238	3303	3310	3303
q9	8577	8530	8531	8530
q10	3806	4033	4035	4033
q11	554	479	456	456
q12	748	628	620	620
q13	15766	3040	3160	3040
q14	337	290	295	290
q15	540	499	479	479
q16	517	444	457	444
q17	1766	1729	1749	1729
q18	8300	7868	7693	7693
q19	1696	1698	1680	1680
q20	2092	1849	1733	1733
q21	5305	4884	5006	4884
q22	513	446	445	445
Total cold run time: 70775 ms
Total hot run time: 55594 ms

doris-robot · 2024-04-02T06:00:51Z

TeamCity be ut coverage result:
Function Coverage: 35.58% (8852/24879)
Line Coverage: 27.29% (72548/265846)
Region Coverage: 26.49% (37551/141759)
Branch Coverage: 23.29% (19139/82194)
Coverage Report: http://coverage.selectdb-in.cc/coverage/11d0030df53dd5ca378b1f56f8355b7bdf4ab088_11d0030df53dd5ca378b1f56f8355b7bdf4ab088/report/index.html

doris-robot · 2024-04-02T09:02:06Z

TPC-H: Total hot run time: 38684 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a338453bce9500e0f43bdd6579f288d344b6b6f4, data reload: false
------ Round 1 ----------------------------------
q1	17657	4193	4104	4104
q2	2002	186	182	182
q3	10475	1296	1417	1296
q4	10216	844	1006	844
q5	7526	2979	2933	2933
q6	218	135	132	132
q7	1094	629	593	593
q8	9418	2038	2061	2038
q9	6760	6210	6123	6123
q10	8463	3526	3495	3495
q11	414	253	230	230
q12	384	215	209	209
q13	17777	2920	2901	2901
q14	276	241	233	233
q15	530	483	471	471
q16	498	367	375	367
q17	964	887	922	887
q18	7275	6504	6383	6383
q19	1625	1534	1536	1534
q20	596	322	313	313
q21	3579	3113	3115	3113
q22	362	303	307	303
Total cold run time: 108109 ms
Total hot run time: 38684 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4196	4091	4106	4091
q2	331	223	219	219
q3	2976	2979	2921	2921
q4	1863	1836	1851	1836
q5	5257	5200	5220	5200
q6	208	125	126	125
q7	2258	1787	1821	1787
q8	3229	3278	3310	3278
q9	8488	8468	8475	8468
q10	3779	3988	3998	3988
q11	563	450	460	450
q12	741	599	580	580
q13	15546	3108	3066	3066
q14	307	271	274	271
q15	539	485	493	485
q16	507	458	444	444
q17	1797	1778	1743	1743
q18	8441	7749	7730	7730
q19	1800	1680	1686	1680
q20	2063	1845	1849	1845
q21	5240	4993	4879	4879
q22	495	460	449	449
Total cold run time: 70624 ms
Total hot run time: 55535 ms

doris-robot · 2024-04-02T09:08:37Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (8881/24913)
Line Coverage: 27.38% (72915/266317)
Region Coverage: 26.55% (37705/141991)
Branch Coverage: 23.35% (19222/82316)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a338453bce9500e0f43bdd6579f288d344b6b6f4_a338453bce9500e0f43bdd6579f288d344b6b6f4/report/index.html

doris-robot · 2024-04-02T09:13:48Z

TPC-DS: Total hot run time: 182599 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a338453bce9500e0f43bdd6579f288d344b6b6f4, data reload: false
query1	1241	1115	1111	1111
query2	6319	2058	1955	1955
query3	6656	210	209	209
query4	24646	21432	21448	21432
query5	4190	393	399	393
query6	271	194	185	185
query7	4603	302	309	302
query8	232	182	194	182
query9	8452	2271	2270	2270
query10	448	252	260	252
query11	14953	14488	14346	14346
query12	146	98	93	93
query13	1622	398	371	371
query14	8336	6909	6921	6909
query15	218	185	179	179
query16	6790	273	266	266
query17	986	586	585	585
query18	1839	287	274	274
query19	200	160	163	160
query20	96	90	92	90
query21	200	130	132	130
query22	5058	4891	4808	4808
query23	33639	32685	32772	32685
query24	12623	3248	3224	3224
query25	723	466	442	442
query26	1868	177	168	168
query27	3132	390	399	390
query28	7146	1903	1887	1887
query29	1404	614	619	614
query30	300	172	177	172
query31	1008	748	766	748
query32	100	64	61	61
query33	711	264	252	252
query34	1070	511	511	511
query35	890	737	736	736
query36	1010	894	872	872
query37	279	82	92	82
query38	3626	3539	3643	3539
query39	1667	1599	1577	1577
query40	252	142	138	138
query41	49	45	46	45
query42	115	118	114	114
query43	456	421	429	421
query44	1143	738	733	733
query45	294	272	273	272
query46	1130	824	822	822
query47	2017	1913	1897	1897
query48	401	318	331	318
query49	989	375	380	375
query50	829	415	412	412
query51	6990	6846	6885	6846
query52	109	106	105	105
query53	379	302	303	302
query54	327	257	256	256
query55	87	83	84	83
query56	263	238	243	238
query57	1284	1194	1193	1193
query58	254	233	242	233
query59	2666	2433	2368	2368
query60	254	240	235	235
query61	94	91	83	83
query62	673	460	442	442
query63	310	284	282	282
query64	5857	3318	3077	3077
query65	3055	3009	2988	2988
query66	1320	310	304	304
query67	15715	15126	15378	15126
query68	9070	590	592	590
query69	577	328	314	314
query70	1391	1114	1098	1098
query71	497	266	274	266
query72	6294	2557	2425	2425
query73	1556	328	331	328
query74	6667	6237	6414	6237
query75	3424	2297	2292	2292
query76	5176	1091	1199	1091
query77	568	249	241	241
query78	10824	10101	10128	10101
query79	11420	545	527	527
query80	1637	422	411	411
query81	528	238	244	238
query82	355	105	105	105
query83	214	164	164	164
query84	276	89	90	89
query85	1063	289	307	289
query86	352	308	283	283
query87	3744	3473	3524	3473
query88	3746	2344	2352	2344
query89	547	366	363	363
query90	2013	182	175	175
query91	133	106	104	104
query92	68	55	51	51
query93	5207	549	532	532
query94	1227	197	192	192
query95	441	327	331	327
query96	615	280	271	271
query97	2667	2490	2474	2474
query98	222	220	217	217
query99	1151	845	817	817
Total cold run time: 299680 ms
Total hot run time: 182599 ms

doris-robot · 2024-04-02T09:19:12Z

ClickBench: Total hot run time: 30.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a338453bce9500e0f43bdd6579f288d344b6b6f4, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.48	0.48	0.59
query6	1.14	0.65	0.65
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.57	0.51	0.52
query10	0.57	0.58	0.55
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.59	0.60
query14	0.77	0.78	0.78
query15	0.85	0.84	0.84
query16	0.35	0.36	0.36
query17	0.96	0.98	0.98
query18	0.25	0.24	0.24
query19	1.86	1.76	1.68
query20	0.02	0.01	0.01
query21	15.41	0.76	0.66
query22	3.29	4.55	1.80
query23	17.54	1.37	1.03
query24	1.80	0.27	0.22
query25	0.13	0.10	0.09
query26	0.30	0.18	0.18
query27	0.08	0.09	0.08
query28	13.38	0.94	0.98
query29	12.53	3.65	3.39
query30	0.26	0.06	0.06
query31	2.86	0.39	0.38
query32	3.29	0.47	0.46
query33	2.81	2.87	2.85
query34	15.48	4.31	4.33
query35	4.36	4.36	4.35
query36	0.67	0.48	0.47
query37	0.20	0.17	0.17
query38	0.17	0.16	0.15
query39	0.05	0.04	0.04
query40	0.20	0.14	0.16
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 105.83 s
Total hot run time: 30.04 s

doris-robot · 2024-04-02T09:22:20Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a338453bce9500e0f43bdd6579f288d344b6b6f4 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 16.6 seconds inserted 10000000 Rows, about 602K ops/s

zclllyybb · 2024-04-07T02:25:55Z

run buildall

github-actions · 2024-04-07T02:32:52Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-07T03:20:10Z

TeamCity be ut coverage result:
Function Coverage: 35.64% (8887/24934)
Line Coverage: 27.38% (72977/266503)
Region Coverage: 26.56% (37720/142044)
Branch Coverage: 23.36% (19227/82312)
Coverage Report: http://coverage.selectdb-in.cc/coverage/47880412ca4b85a0fc62985246b95209bafd8c77_47880412ca4b85a0fc62985246b95209bafd8c77/report/index.html

doris-robot · 2024-04-07T03:28:23Z

TPC-H: Total hot run time: 38657 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47880412ca4b85a0fc62985246b95209bafd8c77, data reload: false
------ Round 1 ----------------------------------
q1	17580	4122	4091	4091
q2	2017	188	180	180
q3	10473	1164	1298	1164
q4	10207	820	988	820
q5	7553	3001	2936	2936
q6	221	136	137	136
q7	1115	638	604	604
q8	9407	2030	2026	2026
q9	6680	6213	6184	6184
q10	8473	3502	3515	3502
q11	432	246	235	235
q12	377	205	204	204
q13	17778	2904	2889	2889
q14	262	241	235	235
q15	529	490	476	476
q16	512	382	371	371
q17	954	925	924	924
q18	7326	6540	6451	6451
q19	1582	1526	1544	1526
q20	566	310	329	310
q21	3505	3130	3098	3098
q22	351	295	313	295
Total cold run time: 107900 ms
Total hot run time: 38657 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4071	4059	4032	4032
q2	325	224	218	218
q3	2990	2953	2968	2953
q4	1873	1888	1843	1843
q5	5206	5213	5216	5213
q6	209	124	123	123
q7	2237	1787	1795	1787
q8	3219	3274	3277	3274
q9	8503	8441	8468	8441
q10	3776	3955	3993	3955
q11	550	464	456	456
q12	729	584	609	584
q13	15993	3058	3113	3058
q14	300	265	280	265
q15	531	524	497	497
q16	489	441	429	429
q17	1785	1760	1737	1737
q18	8293	7664	7833	7664
q19	1700	1683	1676	1676
q20	2038	1832	1813	1813
q21	5150	4922	4920	4920
q22	542	434	430	430
Total cold run time: 70509 ms
Total hot run time: 55368 ms

doris-robot · 2024-04-07T03:40:01Z

TPC-DS: Total hot run time: 181938 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47880412ca4b85a0fc62985246b95209bafd8c77, data reload: false
query1	1234	1123	1116	1116
query2	6352	2006	1823	1823
query3	6658	219	213	213
query4	25179	21404	21423	21404
query5	4192	414	393	393
query6	267	185	187	185
query7	4606	292	294	292
query8	228	175	170	170
query9	8464	2182	2189	2182
query10	438	232	248	232
query11	14915	14494	14641	14494
query12	135	90	87	87
query13	1641	383	382	382
query14	8480	7017	6804	6804
query15	206	166	177	166
query16	6869	272	275	272
query17	983	584	561	561
query18	1878	281	282	281
query19	196	157	167	157
query20	95	91	89	89
query21	198	126	129	126
query22	4927	4754	4799	4754
query23	33278	32723	32689	32689
query24	11308	3162	3205	3162
query25	704	415	405	405
query26	1853	171	166	166
query27	3164	376	364	364
query28	7380	1867	1864	1864
query29	1292	627	609	609
query30	307	170	169	169
query31	965	755	751	751
query32	101	57	56	56
query33	635	243	242	242
query34	1174	514	520	514
query35	836	729	736	729
query36	999	889	895	889
query37	274	71	72	71
query38	3525	3580	3600	3580
query39	1618	1610	1584	1584
query40	226	134	136	134
query41	49	48	46	46
query42	107	108	99	99
query43	459	434	439	434
query44	1194	748	756	748
query45	296	279	278	278
query46	1125	839	824	824
query47	2016	1884	1851	1851
query48	376	305	306	305
query49	924	373	374	373
query50	828	416	406	406
query51	6842	6715	6726	6715
query52	109	94	98	94
query53	357	282	293	282
query54	259	227	233	227
query55	84	74	75	74
query56	246	226	223	223
query57	1294	1178	1140	1140
query58	240	218	209	209
query59	2711	2687	2321	2321
query60	252	224	234	224
query61	91	88	87	87
query62	642	446	451	446
query63	304	276	278	276
query64	5802	3159	3135	3135
query65	3035	3010	3036	3010
query66	1290	325	307	307
query67	15738	14965	15013	14965
query68	9412	578	574	574
query69	573	309	307	307
query70	1448	1106	1076	1076
query71	492	272	274	272
query72	6341	2564	2414	2414
query73	1565	327	324	324
query74	6586	6384	6396	6384
query75	3680	2276	2273	2273
query76	5805	1221	1245	1221
query77	622	247	242	242
query78	10862	10299	10130	10130
query79	9601	544	535	535
query80	1390	426	435	426
query81	514	228	231	228
query82	276	92	94	92
query83	223	167	170	167
query84	268	88	92	88
query85	925	292	283	283
query86	371	299	287	287
query87	3681	3471	3490	3471
query88	3898	2350	2360	2350
query89	546	372	375	372
query90	2010	191	196	191
query91	138	105	109	105
query92	63	47	48	47
query93	6287	528	528	528
query94	1204	180	182	180
query95	447	322	323	322
query96	625	270	270	270
query97	2657	2491	2511	2491
query98	222	217	210	210
query99	1177	825	818	818
Total cold run time: 298359 ms
Total hot run time: 181938 ms

doris-robot · 2024-04-07T03:45:28Z

ClickBench: Total hot run time: 29.36 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47880412ca4b85a0fc62985246b95209bafd8c77, data reload: false
query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.24	0.04	0.04
query4	1.67	0.07	0.07
query5	0.48	0.48	0.47
query6	1.14	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.56	0.51	0.50
query10	0.58	0.57	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.59	0.59
query14	0.77	0.79	0.79
query15	0.87	0.85	0.84
query16	0.37	0.36	0.35
query17	1.03	0.98	0.98
query18	0.26	0.26	0.26
query19	1.81	1.72	1.70
query20	0.01	0.01	0.01
query21	15.45	0.67	0.64
query22	3.68	7.68	1.15
query23	18.00	1.28	1.24
query24	1.56	0.19	0.19
query25	0.16	0.08	0.08
query26	0.27	0.17	0.17
query27	0.09	0.08	0.08
query28	13.85	0.97	0.95
query29	12.59	3.25	3.23
query30	0.25	0.06	0.06
query31	2.87	0.39	0.41
query32	3.23	0.47	0.48
query33	2.87	2.89	2.86
query34	15.49	4.32	4.31
query35	4.37	4.35	4.34
query36	0.67	0.48	0.48
query37	0.18	0.15	0.15
query38	0.16	0.14	0.14
query39	0.04	0.03	0.03
query40	0.18	0.15	0.17
query41	0.09	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 107 s
Total hot run time: 29.36 s

doris-robot · 2024-04-07T03:48:35Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 47880412ca4b85a0fc62985246b95209bafd8c77 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 17.2 seconds inserted 10000000 Rows, about 581K ops/s

Fix threadpool hang on grace exit

github-actionsBot reviewed Apr 1, 2024
View reviewed changes

Comment threadbe/src/olap/tablet.cpp Outdated

yiguolei reviewed Apr 2, 2024
View reviewed changes

Comment threadbe/src/olap/tablet.cpp Outdated

yiguolei requested changes Apr 2, 2024
View reviewed changes

zclllyybb force-pushed the hang branch from 9761e0d to 87461e2Compare April 2, 2024 02:58

zclllyybb requested a review from yiguolei April 2, 2024 02:58

yiguolei previously approved these changes Apr 2, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2024

github-actionsBot added the reviewed label Apr 2, 2024

HappenLee previously approved these changes Apr 2, 2024
View reviewed changes

zclllyybb dismissed stale reviews from HappenLee and yiguolei via 11d0030April 2, 2024 04:58

github-actionsBot removed approved Indicates a PR has been approved by one committer. labels Apr 2, 2024

yiguolei previously approved these changes Apr 4, 2024
View reviewed changes

zclllyybb added 3 commits April 7, 2024 10:20

fix
f30227b

fix ut
2be738c

fix
4788041

zclllyybb dismissed yiguolei’s stale review via 4788041April 7, 2024 02:21

zclllyybb force-pushed the hang branch from a338453 to 4788041Compare April 7, 2024 02:21

BiteTheDDDDt approved these changes Apr 8, 2024
View reviewed changes

BiteTheDDDDt merged commit b7e44d8 into apache:masterApr 8, 2024

seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 10, 2024
[Fix](grace-exit) Fix threadpool hang on grace exit (apache#33132)
955c6fb
Fix threadpool hang on grace exit

yiguolei added the 2.1.0-conflict label Apr 10, 2024

zclllyybb deleted the hang branch April 10, 2024 13:08

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 11, 2024
[Fix](grace-exit) Fix threadpool hang on grace exit (apache#33132)
e632c91
Fix threadpool hang on grace exit

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 11, 2024
[Fix](grace-exit) Fix threadpool hang on grace exit (apache#33132)
2550ee3
Fix threadpool hang on grace exit

zclllyybb mentioned this pull request Apr 11, 2024
[branch-2.1](auto-partition) pick auto partition and some more prs #33523
Merged

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 11, 2024
[Fix](grace-exit) Fix threadpool hang on grace exit (apache#33132)
84fa189
Fix threadpool hang on grace exit

yiguolei added dev/2.1.3-merged and removed 2.1.0-conflict labels Apr 11, 2024

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[Fix](grace-exit) Fix threadpool hang on grace exit (apache#33132)
38a4afc
Fix threadpool hang on grace exit

Uh oh!

Conversation

zclllyybb commented Apr 1, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

zclllyybb commented Apr 1, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei commented Apr 2, 2024

Uh oh!

github-actionsBot commented Apr 2, 2024

Uh oh!

github-actionsBot commented Apr 2, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

zclllyybb commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

zclllyybb commented Apr 7, 2024

Uh oh!

github-actionsBot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

Reviewers

Assignees