[Fix](core) Fix wal mgr heap use after free when stop doris by Yukang-Lian · Pull Request #33131 · apache/doris

Yukang-Lian · 2024-04-01T13:02:09Z

Problem: When the process stops, there is a heap use after free error with the WAL manager.

Reason: During the startup process, if the storage engine does not initialize successfully and the main program directly returns 0, the WAL manager, which was created during initialization and started a thread to periodically check disk space, will encounter an issue. When the program exits and returns 0, local variables are destroyed first before the thread is properly terminated. If the thread attempts to access those local variables at this point, it leads to a heap use after free error.

Solution: Ensure that the thread for periodically checking disk space is only started after the storage engine has been successfully initialized.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-01T13:02:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Yukang-Lian · 2024-04-01T13:02:18Z

run buildall

github-actions · 2024-04-01T13:07:32Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-01T13:34:17Z

TPC-H: Total hot run time: 38934 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 204627b7d467f585926c92fcca40911373aae01b, data reload: false
------ Round 1 ----------------------------------
q1	17624	4373	4242	4242
q2	2012	191	171	171
q3	10635	1216	1408	1216
q4	10393	857	994	857
q5	7471	2987	2936	2936
q6	213	131	130	130
q7	1102	624	606	606
q8	9427	2058	2069	2058
q9	6700	6229	6162	6162
q10	8485	3535	3548	3535
q11	420	241	241	241
q12	393	219	213	213
q13	17775	2917	2883	2883
q14	278	241	232	232
q15	522	484	492	484
q16	530	390	378	378
q17	959	917	923	917
q18	7283	6411	6454	6411
q19	1646	1555	1553	1553
q20	601	310	323	310
q21	3551	3145	3107	3107
q22	373	300	292	292
Total cold run time: 108393 ms
Total hot run time: 38934 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4117	4047	4065	4047
q2	324	224	222	222
q3	2991	2945	2950	2945
q4	1869	1855	1859	1855
q5	5270	5193	5211	5193
q6	207	125	123	123
q7	2241	1808	1814	1808
q8	3236	3313	3288	3288
q9	8487	8493	8510	8493
q10	3755	3812	3868	3812
q11	559	441	457	441
q12	698	547	550	547
q13	16740	2872	2907	2872
q14	292	268	268	268
q15	517	469	481	469
q16	456	406	405	405
q17	1711	1678	1680	1678
q18	7674	7819	7597	7597
q19	1701	1675	1692	1675
q20	2082	1846	1828	1828
q21	5301	4994	4971	4971
q22	489	433	433	433
Total cold run time: 70717 ms
Total hot run time: 54970 ms

dataroaring

LGTM

github-actions · 2024-04-01T14:51:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-01T14:51:37Z

PR approved by anyone and no changes requested.

Yukang-Lian · 2024-04-02T07:15:30Z

run buildall

github-actions · 2024-04-02T07:20:29Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-02T07:53:20Z

TPC-H: Total hot run time: 39193 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da2877e439d53ef0bda0080cdcf530aaf01e43a6, data reload: false
------ Round 1 ----------------------------------
q1	19552	4263	4199	4199
q2	2470	196	191	191
q3	11601	1272	1469	1272
q4	10760	903	1050	903
q5	8376	3123	3046	3046
q6	225	141	139	139
q7	1191	698	644	644
q8	9418	2021	2061	2021
q9	6690	6222	6185	6185
q10	8478	3527	3505	3505
q11	429	241	229	229
q12	387	217	218	217
q13	17777	2902	2918	2902
q14	281	243	242	242
q15	531	484	489	484
q16	521	380	373	373
q17	988	885	918	885
q18	7387	6546	6504	6504
q19	1625	1567	1549	1549
q20	623	324	321	321
q21	3635	3135	3072	3072
q22	357	310	312	310
Total cold run time: 113302 ms
Total hot run time: 39193 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4129	4038	4055	4038
q2	328	216	221	216
q3	2988	2974	2966	2966
q4	1903	1826	1828	1826
q5	5254	5254	5240	5240
q6	209	126	126	126
q7	2256	1803	1829	1803
q8	3226	3303	3287	3287
q9	8677	8469	8476	8469
q10	3764	3832	3844	3832
q11	561	452	449	449
q12	744	563	534	534
q13	11601	2895	2921	2895
q14	286	262	261	261
q15	520	476	461	461
q16	449	404	399	399
q17	1727	1687	1686	1686
q18	7631	7413	7238	7238
q19	1649	1645	1664	1645
q20	1925	1714	1714	1714
q21	5039	4800	4844	4800
q22	504	421	421	421
Total cold run time: 65370 ms
Total hot run time: 54306 ms

doris-robot · 2024-04-02T08:05:00Z

TPC-DS: Total hot run time: 181459 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da2877e439d53ef0bda0080cdcf530aaf01e43a6, data reload: false
query1	1218	374	1123	374
query2	6497	1879	1816	1816
query3	6669	218	219	218
query4	24922	21627	21385	21385
query5	4158	402	409	402
query6	283	185	185	185
query7	4611	316	310	310
query8	230	179	176	176
query9	8496	2277	2263	2263
query10	570	258	276	258
query11	14864	14567	14572	14567
query12	146	96	100	96
query13	1634	379	372	372
query14	8632	6911	6906	6906
query15	214	178	186	178
query16	7160	289	272	272
query17	991	612	578	578
query18	1910	282	277	277
query19	203	161	161	161
query20	99	93	94	93
query21	198	128	124	124
query22	4974	4856	4754	4754
query23	33676	32660	32498	32498
query24	12676	3235	3166	3166
query25	722	403	415	403
query26	1915	169	161	161
query27	3003	323	339	323
query28	6699	1868	1851	1851
query29	1390	595	594	594
query30	307	163	167	163
query31	1012	738	727	727
query32	98	62	57	57
query33	715	264	252	252
query34	1056	489	512	489
query35	832	692	705	692
query36	997	843	872	843
query37	283	77	80	77
query38	3525	3417	3396	3396
query39	1569	1530	1515	1515
query40	299	133	137	133
query41	50	47	45	45
query42	109	104	104	104
query43	433	403	398	398
query44	1110	712	700	700
query45	277	255	253	253
query46	1069	790	782	782
query47	1901	1772	1792	1772
query48	368	306	304	304
query49	1168	366	360	360
query50	793	401	391	391
query51	6828	6810	6825	6810
query52	117	95	98	95
query53	359	290	286	286
query54	338	237	247	237
query55	88	77	81	77
query56	256	231	226	226
query57	1221	1132	1126	1126
query58	255	225	230	225
query59	2546	2505	2446	2446
query60	260	255	252	252
query61	114	109	109	109
query62	693	471	452	452
query63	308	287	289	287
query64	6464	3550	3311	3311
query65	3080	3021	3003	3003
query66	1444	334	333	333
query67	15614	15308	14912	14912
query68	7405	564	554	554
query69	541	325	315	315
query70	1205	1134	1092	1092
query71	508	272	263	263
query72	6315	2568	2424	2424
query73	791	323	332	323
query74	6677	6322	6208	6208
query75	3520	2316	2282	2282
query76	5053	1116	1197	1116
query77	615	250	249	249
query78	10777	10233	10016	10016
query79	8743	568	518	518
query80	1354	426	417	417
query81	513	247	238	238
query82	732	102	108	102
query83	205	159	163	159
query84	282	88	93	88
query85	1459	299	287	287
query86	433	298	287	287
query87	3684	3497	3485	3485
query88	4342	2359	2362	2359
query89	560	368	374	368
query90	2025	175	191	175
query91	132	103	103	103
query92	66	53	53	53
query93	6891	542	529	529
query94	1133	195	197	195
query95	1092	1093	1098	1093
query96	610	274	276	274
query97	2678	2447	2458	2447
query98	238	221	212	212
query99	1325	872	827	827
Total cold run time: 298620 ms
Total hot run time: 181459 ms

doris-robot · 2024-04-02T08:10:24Z

ClickBench: Total hot run time: 30.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da2877e439d53ef0bda0080cdcf530aaf01e43a6, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.24	0.05	0.05
query4	1.67	0.07	0.07
query5	0.47	0.49	0.49
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.51
query10	0.58	0.56	0.57
query11	0.15	0.11	0.11
query12	0.13	0.11	0.12
query13	0.60	0.59	0.59
query14	0.77	0.78	0.80
query15	0.86	0.83	0.86
query16	0.35	0.36	0.36
query17	1.02	0.97	1.00
query18	0.26	0.24	0.25
query19	1.84	1.75	1.79
query20	0.02	0.01	0.01
query21	15.41	0.78	0.75
query22	3.16	4.66	1.46
query23	17.63	1.24	1.13
query24	1.53	0.26	0.30
query25	0.14	0.08	0.07
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.65	1.03	0.96
query29	12.64	3.50	3.50
query30	0.26	0.06	0.06
query31	2.86	0.37	0.39
query32	3.27	0.48	0.47
query33	2.83	2.83	2.86
query34	15.51	4.35	4.35
query35	4.39	4.38	4.42
query36	0.66	0.47	0.47
query37	0.20	0.17	0.17
query38	0.17	0.16	0.16
query39	0.05	0.04	0.04
query40	0.18	0.15	0.17
query41	0.10	0.04	0.04
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 105.95 s
Total hot run time: 30.16 s

doris-robot · 2024-04-02T08:13:30Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit da2877e439d53ef0bda0080cdcf530aaf01e43a6 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 15.6 seconds inserted 10000000 Rows, about 641K ops/s

Yukang-Lian · 2024-04-04T14:13:51Z

run buildall

github-actions · 2024-04-04T14:18:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-04T14:32:14Z

TeamCity be ut coverage result:
Function Coverage: 35.61% (8879/24932)
Line Coverage: 27.34% (72872/266537)
Region Coverage: 26.52% (37682/142092)
Branch Coverage: 23.32% (19199/82336)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ca8109be3103ab4cf3c7abb54833e7d98d238758_ca8109be3103ab4cf3c7abb54833e7d98d238758/report/index.html

doris-robot · 2024-04-04T14:51:19Z

TPC-H: Total hot run time: 38350 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca8109be3103ab4cf3c7abb54833e7d98d238758, data reload: false
------ Round 1 ----------------------------------
q1	17703	4063	4037	4037
q2	2010	183	181	181
q3	10494	1135	1309	1135
q4	10205	812	945	812
q5	7509	2925	2931	2925
q6	218	135	131	131
q7	1085	612	612	612
q8	9394	2034	2008	2008
q9	6712	6176	6161	6161
q10	8451	3513	3477	3477
q11	416	231	229	229
q12	390	220	210	210
q13	17783	2947	2916	2916
q14	265	230	233	230
q15	524	472	492	472
q16	501	383	381	381
q17	948	912	889	889
q18	7146	6520	6348	6348
q19	3228	1513	1518	1513
q20	557	318	294	294
q21	3478	3091	3108	3091
q22	364	298	314	298
Total cold run time: 109381 ms
Total hot run time: 38350 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4076	4018	4030	4018
q2	326	216	217	216
q3	2945	2920	2936	2920
q4	1873	1860	1827	1827
q5	5217	5203	5184	5184
q6	205	121	122	121
q7	2205	1794	1766	1766
q8	3177	3247	3249	3247
q9	8434	8421	8421	8421
q10	3723	3978	4000	3978
q11	568	472	480	472
q12	749	603	575	575
q13	16825	3102	3095	3095
q14	301	269	274	269
q15	539	487	506	487
q16	490	441	441	441
q17	1769	1741	1684	1684
q18	8196	7637	7625	7625
q19	1672	1680	1669	1669
q20	2045	1814	1824	1814
q21	5228	4874	4975	4874
q22	504	444	448	444
Total cold run time: 71067 ms
Total hot run time: 55147 ms

doris-robot · 2024-04-04T15:03:00Z

TPC-DS: Total hot run time: 181579 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca8109be3103ab4cf3c7abb54833e7d98d238758, data reload: false
query1	1228	372	1107	372
query2	6361	1985	1910	1910
query3	6662	214	218	214
query4	24847	21611	21621	21611
query5	4196	390	394	390
query6	260	176	179	176
query7	4612	303	304	303
query8	225	178	176	176
query9	8464	2206	2200	2200
query10	452	261	251	251
query11	14965	14549	14494	14494
query12	135	88	89	88
query13	1645	385	382	382
query14	8469	7048	6871	6871
query15	220	185	184	184
query16	6870	269	267	267
query17	999	598	582	582
query18	1870	285	281	281
query19	196	162	157	157
query20	92	87	93	87
query21	200	129	131	129
query22	4989	4806	4748	4748
query23	33583	32605	32627	32605
query24	10930	3200	3150	3150
query25	724	419	407	407
query26	1982	165	168	165
query27	3220	377	373	373
query28	7268	1844	1840	1840
query29	1336	609	590	590
query30	309	181	188	181
query31	1002	720	728	720
query32	97	60	56	56
query33	656	239	245	239
query34	1270	516	524	516
query35	866	720	714	714
query36	994	860	889	860
query37	280	82	81	81
query38	3594	3617	3642	3617
query39	1630	1591	1570	1570
query40	242	126	133	126
query41	46	43	46	43
query42	109	111	105	105
query43	470	412	430	412
query44	1169	731	747	731
query45	296	260	277	260
query46	1139	847	798	798
query47	1945	1870	1860	1860
query48	380	317	307	307
query49	943	366	367	366
query50	819	411	408	408
query51	6811	6773	6760	6760
query52	105	94	102	94
query53	368	293	289	289
query54	284	258	238	238
query55	83	81	78	78
query56	260	237	231	231
query57	1292	1188	1155	1155
query58	255	228	227	227
query59	2721	2636	2494	2494
query60	259	246	248	246
query61	111	109	106	106
query62	656	456	442	442
query63	311	280	285	280
query64	5883	3228	3236	3228
query65	3048	2997	3016	2997
query66	1311	336	307	307
query67	15525	15281	14886	14886
query68	8935	573	579	573
query69	548	317	315	315
query70	1362	1110	1098	1098
query71	526	274	275	274
query72	6251	2557	2412	2412
query73	797	326	328	326
query74	6800	6318	6327	6318
query75	3647	2313	2292	2292
query76	5888	1215	1231	1215
query77	638	251	246	246
query78	10862	10076	10077	10076
query79	11173	539	531	531
query80	1862	422	418	418
query81	496	243	242	242
query82	580	92	97	92
query83	217	170	178	170
query84	275	85	91	85
query85	1235	286	278	278
query86	374	297	280	280
query87	3654	3492	3483	3483
query88	4227	2341	2354	2341
query89	559	373	369	369
query90	2013	178	185	178
query91	136	104	101	101
query92	58	49	51	49
query93	6785	522	532	522
query94	1139	181	179	179
query95	438	312	313	312
query96	610	272	275	272
query97	2678	2497	2449	2449
query98	230	217	216	216
query99	1299	836	836	836
Total cold run time: 300831 ms
Total hot run time: 181579 ms

doris-robot · 2024-04-04T15:08:28Z

ClickBench: Total hot run time: 29.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca8109be3103ab4cf3c7abb54833e7d98d238758, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.05	0.04
query4	1.68	0.07	0.07
query5	0.48	0.48	0.49
query6	1.14	0.64	0.66
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.52	0.50
query10	0.57	0.57	0.55
query11	0.15	0.11	0.12
query12	0.15	0.11	0.12
query13	0.62	0.59	0.59
query14	0.79	0.78	0.80
query15	0.87	0.85	0.85
query16	0.37	0.36	0.35
query17	1.01	0.98	1.02
query18	0.25	0.26	0.26
query19	1.82	1.68	1.75
query20	0.01	0.01	0.01
query21	15.41	0.64	0.65
query22	4.11	7.73	1.43
query23	18.08	1.33	1.28
query24	1.56	0.18	0.18
query25	0.16	0.08	0.07
query26	0.28	0.17	0.18
query27	0.08	0.08	0.08
query28	13.81	0.96	0.95
query29	12.55	3.28	3.29
query30	0.25	0.06	0.07
query31	2.84	0.39	0.40
query32	3.27	0.48	0.48
query33	2.84	2.88	2.80
query34	15.54	4.35	4.36
query35	4.38	4.37	4.37
query36	0.68	0.47	0.47
query37	0.19	0.15	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.04	0.05
query42	0.06	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 107.48 s
Total hot run time: 29.69 s

doris-robot · 2024-04-04T15:11:36Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ca8109be3103ab4cf3c7abb54833e7d98d238758 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 17.4 seconds inserted 10000000 Rows, about 574K ops/s

Yukang-Lian · 2024-04-04T15:47:54Z

run p0

Yukang-Lian · 2024-04-05T02:48:18Z

run buildall

doris-robot · 2024-04-05T03:10:22Z

TeamCity be ut coverage result:
Function Coverage: 35.62% (8881/24932)
Line Coverage: 27.35% (72904/266540)
Region Coverage: 26.54% (37714/142097)
Branch Coverage: 23.34% (19216/82340)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ca8109be3103ab4cf3c7abb54833e7d98d238758_ca8109be3103ab4cf3c7abb54833e7d98d238758/report/index.html

Yukang-Lian · 2024-08-13T07:26:07Z

run buildall

github-actions · 2024-08-13T07:32:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-13T08:02:58Z

TPC-H: Total hot run time: 39722 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9946a8732a8b3a4ff24ab5a93a023182d7885c7, data reload: false
------ Round 1 ----------------------------------
q1	17610	4423	4267	4267
q2	2017	181	173	173
q3	11751	1016	1087	1016
q4	10536	753	712	712
q5	7736	2779	2804	2779
q6	226	139	134	134
q7	970	588	593	588
q8	9531	2033	2060	2033
q9	10159	6543	6573	6543
q10	7014	2163	2146	2146
q11	450	243	252	243
q12	390	217	212	212
q13	19002	2957	2971	2957
q14	278	236	250	236
q15	544	483	490	483
q16	509	408	380	380
q17	985	702	703	702
q18	7907	7353	7437	7353
q19	7177	1068	958	958
q20	674	343	331	331
q21	5833	4476	4550	4476
q22	1123	1033	1000	1000
Total cold run time: 122422 ms
Total hot run time: 39722 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4494	4328	4266	4266
q2	366	290	284	284
q3	2979	2828	2732	2732
q4	2019	1735	1661	1661
q5	5609	5548	5625	5548
q6	226	136	143	136
q7	2133	1759	1764	1759
q8	3296	3447	3429	3429
q9	8730	8871	8794	8794
q10	3416	3276	3270	3270
q11	596	503	506	503
q12	830	598	618	598
q13	16087	3160	3190	3160
q14	320	285	296	285
q15	548	474	496	474
q16	496	433	424	424
q17	1803	1529	1493	1493
q18	8184	8123	7740	7740
q19	2809	1533	1652	1533
q20	2765	1908	1880	1880
q21	5520	5118	5508	5118
q22	1132	1058	1048	1048
Total cold run time: 74358 ms
Total hot run time: 56135 ms

doris-robot · 2024-08-13T08:15:40Z

TPC-DS: Total hot run time: 190245 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9946a8732a8b3a4ff24ab5a93a023182d7885c7, data reload: false
query1	1245	876	870	870
query2	6384	2030	1930	1930
query3	10648	3814	3844	3814
query4	59660	24048	23103	23103
query5	5921	492	464	464
query6	492	159	158	158
query7	6538	293	287	287
query8	304	204	203	203
query9	9182	2417	2369	2369
query10	534	270	267	267
query11	15743	15064	15083	15064
query12	171	105	112	105
query13	1630	406	391	391
query14	12943	7706	7385	7385
query15	245	173	181	173
query16	7626	472	483	472
query17	1177	583	575	575
query18	1964	317	325	317
query19	286	144	149	144
query20	119	110	110	110
query21	214	102	107	102
query22	4744	4516	4490	4490
query23	34283	33099	33100	33099
query24	5635	2882	2934	2882
query25	554	398	414	398
query26	699	156	163	156
query27	1799	276	276	276
query28	3846	2008	2011	2008
query29	712	420	426	420
query30	231	151	156	151
query31	935	746	702	702
query32	105	57	59	57
query33	472	293	289	289
query34	850	462	469	462
query35	805	720	725	720
query36	1055	924	931	924
query37	139	84	81	81
query38	3970	3916	3860	3860
query39	1428	1410	1386	1386
query40	203	123	118	118
query41	48	44	45	44
query42	115	100	94	94
query43	524	475	467	467
query44	1063	753	758	753
query45	196	166	166	166
query46	1081	750	760	750
query47	1844	1745	1761	1745
query48	367	303	296	296
query49	767	434	431	431
query50	806	399	396	396
query51	6795	6666	6677	6666
query52	101	90	91	90
query53	260	187	185	185
query54	663	441	441	441
query55	76	77	74	74
query56	283	238	243	238
query57	1123	1034	1059	1034
query58	220	254	218	218
query59	2973	2805	2747	2747
query60	280	265	260	260
query61	93	93	92	92
query62	744	650	659	650
query63	202	180	179	179
query64	9162	2297	1697	1697
query65	3172	3155	3155	3155
query66	708	326	323	323
query67	15355	14813	14852	14813
query68	7902	565	559	559
query69	657	424	309	309
query70	1192	1127	1069	1069
query71	506	280	285	280
query72	6924	2247	2048	2048
query73	808	331	325	325
query74	9261	8977	8678	8678
query75	3747	2675	2706	2675
query76	4343	1036	960	960
query77	706	306	311	306
query78	9916	9072	9003	9003
query79	10508	534	549	534
query80	1716	504	501	501
query81	560	228	218	218
query82	890	132	130	130
query83	310	143	145	143
query84	275	78	75	75
query85	1308	318	266	266
query86	366	299	267	267
query87	4358	4257	4237	4237
query88	4758	2426	2398	2398
query89	512	292	279	279
query90	1979	194	192	192
query91	122	97	97	97
query92	66	49	50	49
query93	6258	555	551	551
query94	739	285	291	285
query95	344	259	265	259
query96	622	269	274	269
query97	3235	3052	3044	3044
query98	239	205	212	205
query99	1658	1248	1297	1248
Total cold run time: 341297 ms
Total hot run time: 190245 ms

doris-robot · 2024-08-13T08:21:12Z

ClickBench: Total hot run time: 31.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9946a8732a8b3a4ff24ab5a93a023182d7885c7, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.09
query5	0.49	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.56	0.49	0.49
query10	0.55	0.53	0.55
query11	0.15	0.11	0.12
query12	0.16	0.12	0.12
query13	0.60	0.60	0.58
query14	0.76	0.78	0.76
query15	0.85	0.81	0.81
query16	0.36	0.34	0.37
query17	1.01	1.05	0.96
query18	0.22	0.22	0.21
query19	1.78	1.73	1.74
query20	0.01	0.01	0.02
query21	15.39	0.73	0.66
query22	4.21	6.50	2.49
query23	18.35	1.35	1.24
query24	2.11	0.24	0.22
query25	0.15	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.22	0.21
query28	13.19	1.02	0.99
query29	12.63	3.35	3.38
query30	0.24	0.05	0.05
query31	2.88	0.39	0.39
query32	3.28	0.48	0.47
query33	2.93	2.96	2.99
query34	17.09	4.34	4.33
query35	4.39	4.41	4.38
query36	0.66	0.46	0.48
query37	0.19	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.68 s
Total hot run time: 31.18 s

Yukang-Lian · 2024-08-13T16:31:26Z

run buildall

doris-robot · 2024-08-13T16:55:33Z

TPC-H: Total hot run time: 39610 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9946a8732a8b3a4ff24ab5a93a023182d7885c7, data reload: false
------ Round 1 ----------------------------------
q1	18014	4520	4479	4479
q2	2555	185	183	183
q3	11295	1258	1072	1072
q4	10501	730	739	730
q5	8258	2792	2825	2792
q6	231	138	138	138
q7	975	608	599	599
q8	9379	2035	1999	1999
q9	8629	6554	6530	6530
q10	7046	2197	2179	2179
q11	473	249	250	249
q12	392	221	217	217
q13	18057	2977	2964	2964
q14	272	243	233	233
q15	539	485	498	485
q16	504	382	396	382
q17	970	654	715	654
q18	8047	7438	7334	7334
q19	2739	986	1044	986
q20	680	333	325	325
q21	5310	4090	4247	4090
q22	1104	999	990	990
Total cold run time: 115970 ms
Total hot run time: 39610 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4463	4253	4241	4241
q2	363	272	271	271
q3	2810	2547	2547	2547
q4	1884	1620	1606	1606
q5	5325	5352	5321	5321
q6	219	129	129	129
q7	2029	1656	1634	1634
q8	3162	3286	3330	3286
q9	8396	8357	8300	8300
q10	3398	3138	3119	3119
q11	595	486	488	486
q12	743	582	598	582
q13	17307	2965	3013	2965
q14	295	275	273	273
q15	510	484	493	484
q16	466	419	407	407
q17	1763	1499	1458	1458
q18	7740	7502	7321	7321
q19	1658	1548	1575	1548
q20	1998	1816	1772	1772
q21	5233	5083	5018	5018
q22	1090	985	993	985
Total cold run time: 71447 ms
Total hot run time: 53753 ms

doris-robot · 2024-08-13T17:07:48Z

TPC-DS: Total hot run time: 186658 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9946a8732a8b3a4ff24ab5a93a023182d7885c7, data reload: false
query1	923	380	372	372
query2	6446	2028	1939	1939
query3	6663	212	214	212
query4	29184	23120	23108	23108
query5	4180	478	482	478
query6	279	160	161	160
query7	4585	294	294	294
query8	260	200	204	200
query9	8545	2446	2421	2421
query10	437	276	262	262
query11	17810	14836	15062	14836
query12	147	101	104	101
query13	1632	380	365	365
query14	9461	7651	7633	7633
query15	265	169	165	165
query16	7993	503	468	468
query17	1590	567	549	549
query18	2086	293	286	286
query19	190	147	147	147
query20	115	104	106	104
query21	213	102	101	101
query22	4427	4169	4263	4169
query23	33989	33729	33351	33351
query24	12066	2862	2806	2806
query25	686	401	389	389
query26	1723	160	161	160
query27	2888	272	272	272
query28	7571	2037	2020	2020
query29	1015	422	415	415
query30	304	151	156	151
query31	983	740	769	740
query32	99	54	57	54
query33	761	296	288	288
query34	951	460	462	460
query35	863	724	705	705
query36	1104	914	946	914
query37	277	83	81	81
query38	3949	3816	3796	3796
query39	1450	1378	1415	1378
query40	274	117	116	116
query41	50	47	46	46
query42	107	96	98	96
query43	507	485	478	478
query44	1238	728	731	728
query45	195	166	168	166
query46	1115	718	740	718
query47	1878	1769	1774	1769
query48	369	294	303	294
query49	1198	423	417	417
query50	797	406	397	397
query51	6878	6660	6633	6633
query52	98	90	86	86
query53	255	184	178	178
query54	948	456	451	451
query55	75	79	75	75
query56	275	251	261	251
query57	1197	1054	1060	1054
query58	256	266	246	246
query59	3009	2950	2750	2750
query60	302	269	275	269
query61	120	117	221	117
query62	825	619	663	619
query63	223	180	197	180
query64	10586	2300	1752	1752
query65	3209	3165	3151	3151
query66	1370	337	325	325
query67	15625	15068	14851	14851
query68	8719	553	561	553
query69	673	400	279	279
query70	1260	1107	1108	1107
query71	524	273	266	266
query72	7747	2250	2031	2031
query73	1968	325	325	325
query74	9288	8745	8711	8711
query75	4927	2609	2690	2609
query76	5135	1030	1064	1030
query77	723	305	343	305
query78	12686	9672	10536	9672
query79	10562	522	529	522
query80	1098	493	487	487
query81	584	224	223	223
query82	289	134	132	132
query83	313	146	142	142
query84	275	79	77	77
query85	715	300	270	270
query86	345	300	263	263
query87	4418	4190	4209	4190
query88	4764	2402	2398	2398
query89	408	295	285	285
query90	2421	192	193	192
query91	121	101	99	99
query92	63	51	49	49
query93	3003	531	535	531
query94	1155	299	284	284
query95	369	261	264	261
query96	606	271	279	271
query97	3174	3083	3061	3061
query98	223	201	196	196
query99	1519	1297	1274	1274
Total cold run time: 321164 ms
Total hot run time: 186658 ms

doris-robot · 2024-08-13T17:13:21Z

ClickBench: Total hot run time: 32.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9946a8732a8b3a4ff24ab5a93a023182d7885c7, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.08	0.08
query5	0.49	0.50	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.55	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.58
query14	0.77	0.77	0.76
query15	0.88	0.82	0.82
query16	0.37	0.36	0.37
query17	1.05	1.03	1.06
query18	0.23	0.22	0.22
query19	1.94	1.73	1.77
query20	0.01	0.01	0.02
query21	15.39	0.74	0.65
query22	3.52	7.22	3.24
query23	18.28	1.42	1.32
query24	2.11	0.22	0.22
query25	0.15	0.08	0.08
query26	0.31	0.22	0.21
query27	0.45	0.22	0.23
query28	13.27	1.01	0.99
query29	12.65	3.33	3.31
query30	0.24	0.05	0.05
query31	2.89	0.42	0.41
query32	3.23	0.50	0.48
query33	2.99	2.98	2.96
query34	17.08	4.30	4.34
query35	4.44	4.48	4.39
query36	0.66	0.51	0.46
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.14
query41	0.09	0.05	0.06
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.35 s
Total hot run time: 32.13 s

Yukang-Lian · 2024-08-14T12:12:01Z

run cloud_p0

dataroaring

LGTM

github-actions · 2024-08-16T08:42:07Z

PR approved by at least one committer and no changes requested.

Problem: When the process stops, there is a heap use after free error with the WAL manager. Reason: During the startup process, if the storage engine does not initialize successfully and the main program directly returns 0, the WAL manager, which was created during initialization and started a thread to periodically check disk space, will encounter an issue. When the program exits and returns 0, local variables are destroyed first before the thread is properly terminated. If the thread attempts to access those local variables at this point, it leads to a heap use after free error. Solution: Ensure that the thread for periodically checking disk space is only started after the storage engine has been successfully initialized. ## Proposed changes Issue Number: close #xxx  ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...

…3131) Problem: When the process stops, there is a heap use after free error with the WAL manager. Reason: During the startup process, if the storage engine does not initialize successfully and the main program directly returns 0, the WAL manager, which was created during initialization and started a thread to periodically check disk space, will encounter an issue. When the program exits and returns 0, local variables are destroyed first before the thread is properly terminated. If the thread attempts to access those local variables at this point, it leads to a heap use after free error. Solution: Ensure that the thread for periodically checking disk space is only started after the storage engine has been successfully initialized. ## Proposed changes Issue Number: close #xxx  ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...

…r free when stop doris (#33131)" (#39545) Pick #33131

…3131) Problem: When the process stops, there is a heap use after free error with the WAL manager. Reason: During the startup process, if the storage engine does not initialize successfully and the main program directly returns 0, the WAL manager, which was created during initialization and started a thread to periodically check disk space, will encounter an issue. When the program exits and returns 0, local variables are destroyed first before the thread is properly terminated. If the thread attempts to access those local variables at this point, it leads to a heap use after free error. Solution: Ensure that the thread for periodically checking disk space is only started after the storage engine has been successfully initialized. ## Proposed changes Issue Number: close #xxx  ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...

dataroaring previously approved these changes Apr 1, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 1, 2024

github-actionsBot added the reviewed label Apr 1, 2024

mymeiyi approved these changes Apr 2, 2024
View reviewed changes

gavinchou reviewed Apr 2, 2024
View reviewed changes

Comment threadbe/src/olap/wal/wal_manager.cpp Outdated

Yukang-Lian dismissed dataroaring’s stale review via da2877eApril 2, 2024 07:15

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Apr 2, 2024

Yukang-Lian force-pushed the Fix_Wal_Manager_Init_Use_After_Free branch from da2877e to ca8109bCompare April 4, 2024 14:13

Yukang-Lian force-pushed the Fix_Wal_Manager_Init_Use_After_Free branch from 7c0e15f to d9946a8Compare August 13, 2024 07:25

github-actionsBot added the doing label Aug 13, 2024

Yukang-Lian marked this pull request as ready for review August 13, 2024 07:25

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 13, 2024

dataroaring approved these changes Aug 16, 2024
View reviewed changes

dataroaring added dev/2.1.x dev/3.0.x labels Aug 16, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 16, 2024

dataroaring merged commit ec249e7 into apache:masterAug 16, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 17, 2024

Yukang-Lian mentioned this pull request Aug 19, 2024
[cherry-pick](branch-2.1) Pick "[Fix](core) Fix wal mgr heap use after free when stop doris (#33131)" #39545
Merged

dataroaring pushed a commit that referenced this pull request Aug 19, 2024
[cherry-pick](branch-2.1) Pick "[Fix](core) Fix wal mgr heap use afte…
3922fdd
…r free when stop doris (#33131)" (#39545) Pick #33131

dataroaring added dev/2.1.6-merged and removed dev/2.1.x labels Aug 19, 2024

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

Uh oh!

Conversation

Yukang-Lian commented Apr 1, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

Yukang-Lian commented Apr 1, 2024

Uh oh!

github-actionsBot commented Apr 1, 2024

Uh oh!

doris-robot commented Apr 1, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 1, 2024

Uh oh!

github-actionsBot commented Apr 1, 2024

Uh oh!

Uh oh!

Yukang-Lian commented Apr 2, 2024

Uh oh!

github-actionsBot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

doris-robot commented Apr 2, 2024

Uh oh!

Yukang-Lian commented Apr 4, 2024

Uh oh!

github-actionsBot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

Yukang-Lian commented Apr 4, 2024

Uh oh!

Yukang-Lian commented Apr 5, 2024

Uh oh!

doris-robot commented Apr 5, 2024

Uh oh!

Yukang-Lian commented Aug 13, 2024

Uh oh!

github-actionsBot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

Yukang-Lian commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

doris-robot commented Aug 13, 2024

Uh oh!

Yukang-Lian commented Aug 14, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 16, 2024

Uh oh!

Reviewers