[enhancement](memory) Allocator support address sanitizers by xinyiZzz · Pull Request #33396 · apache/doris

xinyiZzz · 2024-04-08T17:16:24Z

Proposed changes

If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace.
add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query.
If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-08T17:16:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-04-08T17:16:32Z

run buildall

github-actions

clang-tidy made some suggestions

xinyiZzz · 2024-04-09T11:20:56Z

run buildall

xinyiZzz · 2024-04-09T16:26:40Z

run buildall

xinyiZzz · 2024-04-10T02:40:56Z

run buildall

github-actions · 2024-04-10T02:46:03Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-10T05:27:04Z

run buildall

github-actions · 2024-04-10T05:32:47Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-10T08:27:16Z

run buildall

github-actions · 2024-04-10T08:32:33Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-11T04:16:30Z

run buildall

github-actions · 2024-04-11T04:22:03Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-11T16:33:07Z

run buildall

github-actions · 2024-04-11T16:38:40Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-12T02:32:16Z

run buildall

github-actions · 2024-04-12T02:37:49Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-12T09:21:56Z

run p0

github-actions · 2024-04-12T09:27:53Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-12T09:38:18Z

run p0

github-actions · 2024-04-12T09:44:14Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-13T07:26:19Z

run p0

github-actions · 2024-04-13T07:32:11Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-04-13T13:24:28Z

run buildall

xinyiZzz · 2024-04-13T13:27:11Z

run buildall

github-actions

clang-tidy made some suggestions

xinyiZzz · 2024-04-13T18:28:36Z

run buildall

github-actions · 2024-04-13T18:31:44Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-17T16:14:41Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f991e14ff43dff01992c239365e11a2b0babf50b with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s

xinyiZzz · 2024-04-17T16:49:38Z

run buildall

github-actions · 2024-04-17T16:55:21Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-17T18:02:40Z

TPC-H: Total hot run time: 38752 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false
------ Round 1 ----------------------------------
q1	17607	4480	4422	4422
q2	2003	181	178	178
q3	10475	1111	1167	1111
q4	10193	771	831	771
q5	7535	2669	2647	2647
q6	214	133	134	133
q7	1054	635	588	588
q8	9243	2073	2053	2053
q9	7348	6606	6543	6543
q10	8562	3538	3515	3515
q11	474	223	235	223
q12	398	221	212	212
q13	17758	2922	2952	2922
q14	276	237	220	220
q15	503	478	471	471
q16	532	396	386	386
q17	962	756	668	668
q18	7563	6970	6828	6828
q19	7367	1512	1463	1463
q20	665	301	305	301
q21	3463	2823	2791	2791
q22	358	318	306	306
Total cold run time: 114553 ms
Total hot run time: 38752 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4391	4238	4207	4207
q2	371	275	264	264
q3	3003	2774	2749	2749
q4	1901	1626	1561	1561
q5	5403	5390	5308	5308
q6	209	123	124	123
q7	2201	1831	1877	1831
q8	3248	3388	3367	3367
q9	8592	8600	8577	8577
q10	3883	3688	3692	3688
q11	594	479	490	479
q12	758	586	582	582
q13	16870	2967	2926	2926
q14	289	283	287	283
q15	512	470	476	470
q16	477	419	433	419
q17	1793	1483	1463	1463
q18	8144	7953	7911	7911
q19	1684	1621	1674	1621
q20	2069	1869	1828	1828
q21	10526	4943	4989	4943
q22	529	470	458	458
Total cold run time: 77447 ms
Total hot run time: 55058 ms

doris-robot · 2024-04-17T18:12:55Z

TeamCity be ut coverage result:
Function Coverage: 35.45% (8909/25133)
Line Coverage: 27.14% (73180/269658)
Region Coverage: 26.28% (37828/143964)
Branch Coverage: 23.07% (19270/83540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c4f25badbbbad57a4f0330291beaff7d61f2c72d_c4f25badbbbad57a4f0330291beaff7d61f2c72d/report/index.html

doris-robot · 2024-04-17T19:02:37Z

TPC-H: Total hot run time: 38486 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false
------ Round 1 ----------------------------------
q1	17608	4403	4243	4243
q2	2015	189	179	179
q3	10472	1145	1157	1145
q4	10193	794	837	794
q5	7551	2676	2606	2606
q6	213	129	130	129
q7	982	613	593	593
q8	9226	2063	2051	2051
q9	7272	6546	6555	6546
q10	8647	3523	3536	3523
q11	448	228	225	225
q12	407	215	219	215
q13	17773	2930	2954	2930
q14	273	232	237	232
q15	528	481	473	473
q16	517	380	369	369
q17	962	723	661	661
q18	7309	6793	6775	6775
q19	6731	1492	1494	1492
q20	635	309	294	294
q21	3422	2708	2795	2708
q22	375	303	303	303
Total cold run time: 113559 ms
Total hot run time: 38486 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4382	4254	4233	4233
q2	359	273	272	272
q3	2958	2740	2800	2740
q4	1866	1559	1548	1548
q5	5373	5388	5324	5324
q6	209	123	125	123
q7	2246	1874	1879	1874
q8	3214	3330	3346	3330
q9	8593	8624	8607	8607
q10	4072	3925	3971	3925
q11	594	490	502	490
q12	810	633	621	621
q13	17133	3224	3135	3135
q14	329	296	313	296
q15	533	491	488	488
q16	486	448	436	436
q17	1826	1529	1494	1494
q18	8269	7982	7944	7944
q19	1675	1541	1539	1539
q20	2083	1831	1847	1831
q21	8983	4916	4914	4914
q22	556	487	484	484
Total cold run time: 76549 ms
Total hot run time: 55648 ms

doris-robot · 2024-04-17T19:14:34Z

TPC-DS: Total hot run time: 185549 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false
query1	884	382	363	363
query2	6198	2592	2434	2434
query3	6660	209	207	207
query4	22778	21239	21226	21226
query5	4171	403	408	403
query6	262	177	173	173
query7	4581	301	293	293
query8	229	173	172	172
query9	8545	2320	2301	2301
query10	417	250	270	250
query11	14769	14131	14259	14131
query12	132	86	82	82
query13	1624	375	368	368
query14	8641	8254	7406	7406
query15	260	185	185	185
query16	8186	256	257	256
query17	1946	566	539	539
query18	2104	277	263	263
query19	328	150	152	150
query20	89	81	86	81
query21	197	137	122	122
query22	4985	4799	4854	4799
query23	33866	33049	33164	33049
query24	10824	3115	3038	3038
query25	654	417	413	413
query26	1175	164	160	160
query27	2381	376	403	376
query28	6975	2075	2068	2068
query29	908	678	641	641
query30	285	183	180	180
query31	1003	776	778	776
query32	97	54	60	54
query33	794	265	257	257
query34	1332	492	504	492
query35	847	722	727	722
query36	1076	913	929	913
query37	127	71	70	70
query38	3516	3411	3339	3339
query39	1646	1601	1581	1581
query40	178	133	131	131
query41	47	49	44	44
query42	108	100	101	100
query43	594	537	542	537
query44	1222	760	758	758
query45	287	288	279	279
query46	1109	749	778	749
query47	2039	1947	1978	1947
query48	389	302	310	302
query49	870	406	388	388
query50	798	408	406	406
query51	6794	6717	6804	6717
query52	109	89	95	89
query53	357	282	281	281
query54	308	240	240	240
query55	76	73	73	73
query56	246	229	232	229
query57	1220	1162	1106	1106
query58	226	205	214	205
query59	3423	3167	3079	3079
query60	258	262	235	235
query61	127	104	103	103
query62	606	453	445	445
query63	315	285	279	279
query64	5167	3928	3947	3928
query65	3060	3002	3040	3002
query66	752	325	319	319
query67	15498	15041	15155	15041
query68	9479	565	571	565
query69	594	301	316	301
query70	1444	1140	1115	1115
query71	1489	1267	1271	1267
query72	6983	2598	2434	2434
query73	1642	328	328	328
query74	6788	6465	6403	6403
query75	4486	2662	2622	2622
query76	5921	1064	999	999
query77	647	270	274	270
query78	11106	10137	10213	10137
query79	12362	527	527	527
query80	2834	434	442	434
query81	538	240	241	240
query82	282	103	95	95
query83	213	164	165	164
query84	263	83	82	82
query85	1059	256	256	256
query86	343	317	303	303
query87	3543	3301	3283	3283
query88	5343	2430	2490	2430
query89	506	384	375	375
query90	2380	178	179	178
query91	121	93	95	93
query92	60	45	47	45
query93	8116	509	504	504
query94	1525	173	174	173
query95	379	284	286	284
query96	602	270	268	268
query97	3140	2961	2973	2961
query98	245	215	216	215
query99	1217	865	853	853
Total cold run time: 307325 ms
Total hot run time: 185549 ms

doris-robot · 2024-04-17T19:20:02Z

ClickBench: Total hot run time: 30.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.47
query6	1.47	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.54	0.51	0.49
query10	0.56	0.55	0.54
query11	0.16	0.12	0.12
query12	0.14	0.12	0.13
query13	0.61	0.58	0.59
query14	0.75	0.77	0.78
query15	0.82	0.81	0.81
query16	0.38	0.37	0.36
query17	0.95	0.93	0.97
query18	0.19	0.29	0.24
query19	1.85	1.71	1.66
query20	0.02	0.01	0.01
query21	15.40	0.65	0.64
query22	4.56	6.71	2.20
query23	18.33	1.35	1.20
query24	1.81	0.22	0.22
query25	0.14	0.08	0.08
query26	0.25	0.16	0.16
query27	0.09	0.07	0.07
query28	13.37	1.00	0.97
query29	12.62	3.24	3.22
query30	0.26	0.06	0.06
query31	2.89	0.38	0.37
query32	3.30	0.45	0.46
query33	2.83	2.77	2.83
query34	17.06	4.42	4.45
query35	4.45	4.46	4.48
query36	0.64	0.45	0.46
query37	0.18	0.15	0.15
query38	0.15	0.13	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.03	0.03
Total cold run time: 109.79 s
Total hot run time: 30.4 s

doris-robot · 2024-04-17T19:23:08Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c4f25badbbbad57a4f0330291beaff7d61f2c72d with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select: 14.5 seconds inserted 10000000 Rows, about 689K ops/s

yiguolei

LGTM

github-actions · 2024-04-18T04:30:04Z

PR approved by at least one committer and no changes requested.

wangbo

LGTM

) If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace. add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query. If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.

pick #33396 #33862 #33853 #33732 #33841 #33933 #34901 #35014 --------- Co-authored-by: yiguolei <676222867@qq.com> Co-authored-by: yiguolei <yiguolei@gmail.com>

) If DEBUG build type, record size of each memory alloc and free, print no free size or no alloc size when query MemTracker is destructed, if necessary, record stack trace. add global PointQueryExecutor memory tracker in ExecEnv, because memory may be shared between PointQueryExecutors of different pointer queries, but memory will not be shared between PointQueryExecutor and Fragment of the same pointer query. If DEBUG build type, if query memory tracker not equal to 0 when query ends, BE will crash.

github-actionsBot reviewed Apr 8, 2024
View reviewed changes

Comment threadbe/src/runtime/memory/mem_tracker_limiter.cpp Outdated
Comment threadbe/src/runtime/memory/mem_tracker_limiter.cpp Outdated

xinyiZzz force-pushed the 20240407_fix_tracker0 branch from 4c6f389 to 07b018fCompare April 11, 2024 04:16

xinyiZzz force-pushed the 20240407_fix_tracker0 branch from f4f5d96 to e2057daCompare April 13, 2024 13:26

github-actionsBot reviewed Apr 13, 2024
View reviewed changes

Comment threadbe/src/vec/common/pod_array.h Outdated
Comment threadbe/src/vec/common/pod_array.h Outdated

8
c4f25ba

yiguolei approved these changes Apr 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 18, 2024

wangbo approved these changes Apr 18, 2024
View reviewed changes

xinyiZzz merged commit 34c57fa into apache:masterApr 18, 2024

yiguolei added the 2.1.0-conflict label Apr 18, 2024

This was referenced Apr 18, 2024
[fix](memory) Allocator address sanitizers enable print stack trace #33853
Merged
[fix](memory) temporarily close Allocator address sanitizers #33862
Merged

xinyiZzz mentioned this pull request Sep 9, 2024
[branch-2.1](memory) Pick query memory tracker value is equal to 0 and exception handle #40564
Closed

xinyiZzz mentioned this pull request Sep 13, 2024
[branch-2.1](memory) Allocator support address sanitizers #40836
Merged

xinyiZzz added dev/2.1.7-merged and removed 2.1.0-conflict labels Sep 20, 2024

Uh oh!

Conversation

xinyiZzz commented Apr 8, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 8, 2024

Uh oh!

xinyiZzz commented Apr 8, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xinyiZzz commented Apr 9, 2024

Uh oh!

xinyiZzz commented Apr 9, 2024

Uh oh!

xinyiZzz commented Apr 10, 2024

Uh oh!

github-actionsBot commented Apr 10, 2024

Uh oh!

xinyiZzz commented Apr 10, 2024

Uh oh!

github-actionsBot commented Apr 10, 2024

Uh oh!

xinyiZzz commented Apr 10, 2024

Uh oh!

github-actionsBot commented Apr 10, 2024

Uh oh!

xinyiZzz commented Apr 11, 2024

Uh oh!

github-actionsBot commented Apr 11, 2024

Uh oh!

xinyiZzz commented Apr 11, 2024

Uh oh!

github-actionsBot commented Apr 11, 2024

Uh oh!

xinyiZzz commented Apr 12, 2024

Uh oh!

github-actionsBot commented Apr 12, 2024

Uh oh!

xinyiZzz commented Apr 12, 2024

Uh oh!

github-actionsBot commented Apr 12, 2024

Uh oh!

xinyiZzz commented Apr 12, 2024

Uh oh!

github-actionsBot commented Apr 12, 2024

Uh oh!

xinyiZzz commented Apr 13, 2024

Uh oh!

github-actionsBot commented Apr 13, 2024

Uh oh!

xinyiZzz commented Apr 13, 2024

Uh oh!

xinyiZzz commented Apr 13, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xinyiZzz commented Apr 13, 2024

Uh oh!

github-actionsBot commented Apr 13, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

xinyiZzz commented Apr 17, 2024

Uh oh!

github-actionsBot commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

xinyiZzz commented Apr 8, 2024•
edited
Loading