[bugfix](memtracker) memory not consumed by memtracker by yiguolei · Pull Request #55796 · apache/doris

yiguolei · 2025-09-09T03:58:31Z

What problem does this PR solve?

PipelineTask is also hold by task queue( #49753), so that it maybe the last one to be destructed. But pipeline task hold some objects, like operators, shared state, etc. So that should release memory manually.

F20250908 20:07:41.329619 39575 mem_tracker_limiter.cpp:112] mem tracker label: Query#Id=ec8535b35ed34f54-afd752d5d1dd97c1, consumption: 16640, peak consumption: 16640, mem tracker not equal to 0 when mem
tracker destruct, this usually means that memory tracking is inaccurate and SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. If the log is truncated, search for Address Sanitizer in the be.INFO log to see more information.1. For query and load, memory leaks may have occurred, it is expected that the query mem tracker will be bound to the thread context using SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER before all memory alloc and free. 2. If a memory alloc is recorded by this tracker, it is expected that be recorded in this tracker when memory is freed. 3. Merge the remaining memory tracking value by this tracker into Orphan, if you observe that Orphan is not equal to 0 in the mem tracker web or log, this indicates that there may be a memory leak. 4. If you need to transfer memory tracking value between two trackers, can use transfer_to..[Address Sanitizer]:
memory not be freed:
[Address Sanitizer] buf not be freed, mem tracker label: Query#Id=ec8535b35ed34f54-afd752d5d1dd97c1, consumption: 16640, peak consumption: 16640, buf: 0x7d87c8761d00, size 4096, strack trace:
0# doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>::alloc(unsigned long, unsigned long)
1# void doris::vectorized::PODArrayBase<1ul, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>::alloc<>(unsigned long)
2# void doris::vectorized::PODArray<signed char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>::push_back(long&&)
3# doris::vectorized::ColumnVector<(doris::PrimitiveType)3>::insert(doris::vectorized::Field const&)
4# doris::vectorized::IDataType::create_column_const(unsigned long, doris::vectorized::Field const&) const
5# doris::vectorized::VLiteral::init(doris::TExprNode const&)
6# doris::vectorized::VLiteral::VLiteral(doris::TExprNode const&, bool)
7# std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::vectorized::VLiteral, std::allocator, doris::TExprNode const&>(doris::vectorized::VLiteral*&, std::_Sp_alloc_shared_tag<std::allocator >, doris::TExprNode const&)
8# doris::vectorized::VExpr::create_expr(doris::TExprNode const&, std::shared_ptrdoris::vectorized::VExpr&)
9# doris::vectorized::VExpr::create_tree_from_thrift(std::vector<doris::TExprNode, std::allocatordoris::TExprNode > const&, int*, std::shared_ptrdoris::vectorized::VExpr&, std::shared_ptrdoris::vectorized::VExprContext&)
10# doris::vectorized::VExpr::create_expr_tree(doris::TExpr const&, std::shared_ptrdoris::vectorized::VExprContext&)
11# doris::pipeline::OperatorXBase::init(doris::TPlanNode const&, doris::RuntimeState*)
12# doris::pipeline::ScanOperatorXdoris::pipeline::OlapScanLocalState::init(doris::TPlanNode const&, doris::RuntimeState*)
13# doris::pipeline::PipelineFragmentContext::_create_tree_helper(doris::ObjectPool*, std::vector<doris::TPlanNode, std::allocatordoris::TPlanNode > const&, doris::TPipelineFragmentParams const&, doris::DescriptorTbl const&, std::shared_ptrdoris::pipeline::OperatorXBase, int*, std::shared_ptrdoris::pipeline::OperatorXBase, std::shared_ptrdoris::pipeline::Pipeline&, int, bool)
14# doris::pipeline::PipelineFragmentContext::_build_pipelines(doris::ObjectPool, doris::TPipelineFragmentParams const&, doris::DescriptorTbl const&, std::shared_ptrdoris::pipeline::OperatorXBase, std::shared_ptrdoris::pipeline::Pipeline)
15# doris::pipeline::PipelineFragmentContext::prepare(doris::TPipelineFragmentParams const&, doris::ThreadPool)
16# doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, std::function<void (doris::RuntimeState*, doris::Status*)> const&, doris::TPipelineFragmentParamsList const&)
17# doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, doris::TPipelineFragmentParamsList const&)
18# doris::PInternalService::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, doris::PFragmentRequestVersion, bool, std::function<void (doris::RuntimeState*, doris::Status*)> const&)
19# doris::PInternalService::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)
20# doris::WorkThreadPool::work_thread(int)
21# execute_native_thread_routine
22# asan_thread_start(void*)
23# ?

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-09-09T03:58:36Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

yiguolei · 2025-09-09T03:59:40Z

run buildall

doris-robot · 2025-09-09T04:38:38Z

TPC-H: Total hot run time: 34829 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9301d43aa1e940d2dd8432cfb81ec3975f2f6ed1, data reload: false
------ Round 1 ----------------------------------
q1	17629	5195	5089	5089
q2	1986	357	216	216
q3	10237	1301	731	731
q4	10232	1041	521	521
q5	7514	2469	2350	2350
q6	194	169	138	138
q7	928	778	656	656
q8	9343	1350	1080	1080
q9	6938	5096	5164	5096
q10	6949	2386	1995	1995
q11	495	306	287	287
q12	369	376	236	236
q13	17766	3669	3034	3034
q14	247	240	226	226
q15	583	507	490	490
q16	1002	1004	956	956
q17	606	869	398	398
q18	7396	7142	7199	7142
q19	1481	961	571	571
q20	357	356	228	228
q21	3979	3223	2408	2408
q22	1069	1024	981	981
Total cold run time: 107300 ms
Total hot run time: 34829 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5281	5092	5084	5084
q2	252	352	231	231
q3	2154	2715	2285	2285
q4	1304	1757	1329	1329
q5	4230	4482	4573	4482
q6	229	183	138	138
q7	2033	1994	1871	1871
q8	2714	2639	2655	2639
q9	7315	7368	7374	7368
q10	3128	3283	2896	2896
q11	583	576	511	511
q12	945	778	609	609
q13	3547	3892	3264	3264
q14	293	306	271	271
q15	546	535	551	535
q16	1070	1128	1062	1062
q17	1215	1576	1422	1422
q18	7802	7691	7771	7691
q19	882	894	971	894
q20	2013	2080	1905	1905
q21	4731	4397	4245	4245
q22	1052	1062	1013	1013
Total cold run time: 53319 ms
Total hot run time: 51745 ms

doris-robot · 2025-09-09T04:50:24Z

TPC-DS: Total hot run time: 189474 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9301d43aa1e940d2dd8432cfb81ec3975f2f6ed1, data reload: false
query1	1031	452	401	401
query2	6574	1694	1725	1694
query3	6752	231	220	220
query4	26399	23586	23294	23294
query5	4456	682	558	558
query6	368	264	235	235
query7	4800	529	307	307
query8	314	258	260	258
query9	9023	2960	2963	2960
query10	492	382	323	323
query11	16068	14969	15059	14969
query12	189	124	120	120
query13	1689	563	444	444
query14	11237	9159	9227	9159
query15	219	197	180	180
query16	7686	675	527	527
query17	1479	781	643	643
query18	2066	437	352	352
query19	260	197	177	177
query20	132	132	131	131
query21	214	133	114	114
query22	4123	4367	4032	4032
query23	33846	32949	32821	32821
query24	8206	2388	2408	2388
query25	553	517	449	449
query26	832	290	168	168
query27	2687	525	366	366
query28	4343	2270	2259	2259
query29	727	609	498	498
query30	295	222	199	199
query31	940	841	730	730
query32	89	81	82	81
query33	557	410	356	356
query34	805	853	513	513
query35	822	816	763	763
query36	1014	1037	950	950
query37	138	121	101	101
query38	3558	3502	3482	3482
query39	1500	1436	1414	1414
query40	258	135	130	130
query41	63	68	61	61
query42	131	117	124	117
query43	525	487	464	464
query44	1379	890	870	870
query45	199	181	166	166
query46	852	1005	645	645
query47	1748	1791	1731	1731
query48	401	435	326	326
query49	713	504	418	418
query50	682	689	401	401
query51	3927	3909	3836	3836
query52	122	113	109	109
query53	250	277	191	191
query54	634	597	561	561
query55	95	93	94	93
query56	346	354	321	321
query57	1204	1204	1119	1119
query58	287	284	283	283
query59	2544	2603	2536	2536
query60	370	359	351	351
query61	167	159	160	159
query62	810	721	648	648
query63	231	194	209	194
query64	3708	1149	834	834
query65	4048	3939	3958	3939
query66	913	435	353	353
query67	15546	15194	15282	15194
query68	8872	945	590	590
query69	515	335	292	292
query70	1408	1317	1284	1284
query71	570	345	322	322
query72	5708	5021	5224	5021
query73	762	672	365	365
query74	9190	8819	8922	8819
query75	4273	3229	2832	2832
query76	3567	1172	762	762
query77	808	406	336	336
query78	9621	9697	8913	8913
query79	2810	835	606	606
query80	727	584	515	515
query81	494	265	234	234
query82	552	161	137	137
query83	292	265	251	251
query84	297	117	94	94
query85	874	470	426	426
query86	346	286	334	286
query87	3747	3716	3707	3707
query88	3051	2225	2242	2225
query89	413	337	303	303
query90	1958	234	228	228
query91	160	162	133	133
query92	88	79	74	74
query93	1359	1002	667	667
query94	701	429	324	324
query95	413	338	327	327
query96	492	586	284	284
query97	2978	2985	2867	2867
query98	243	219	216	216
query99	1432	1392	1348	1348
Total cold run time: 277357 ms
Total hot run time: 189474 ms

doris-robot · 2025-09-09T04:55:52Z

ClickBench: Total hot run time: 29.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9301d43aa1e940d2dd8432cfb81ec3975f2f6ed1, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.11	0.12
query5	0.27	0.26	0.26
query6	1.17	0.67	0.63
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.62	0.54	0.53
query10	0.59	0.57	0.57
query11	0.16	0.11	0.12
query12	0.17	0.12	0.12
query13	0.63	0.63	0.62
query14	1.02	1.02	1.03
query15	0.87	0.85	0.86
query16	0.41	0.41	0.40
query17	1.06	1.07	1.09
query18	0.22	0.23	0.20
query19	1.95	1.81	1.90
query20	0.01	0.02	0.01
query21	15.39	0.94	0.58
query22	0.75	1.10	0.73
query23	14.96	1.38	0.62
query24	6.68	1.34	0.35
query25	0.47	0.14	0.12
query26	0.60	0.16	0.14
query27	0.07	0.06	0.06
query28	10.03	1.01	0.43
query29	12.56	3.91	3.22
query30	0.29	0.13	0.12
query31	2.83	0.61	0.39
query32	3.25	0.54	0.48
query33	3.00	3.16	3.01
query34	16.19	5.50	4.84
query35	4.92	4.92	4.93
query36	0.69	0.52	0.50
query37	0.11	0.08	0.07
query38	0.07	0.05	0.05
query39	0.04	0.03	0.03
query40	0.19	0.17	0.14
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.51 s
Total hot run time: 29.44 s

yiguolei · 2025-09-09T04:56:50Z

run beut

yiguolei · 2025-09-09T05:31:11Z

run buildall

doris-robot · 2025-09-09T05:56:51Z

TPC-H: Total hot run time: 34593 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f28d116aaa782852d2f516ed6e953447790edfdb, data reload: false
------ Round 1 ----------------------------------
q1	17619	5129	5042	5042
q2	1987	342	209	209
q3	10244	1325	708	708
q4	10233	1037	515	515
q5	7539	2395	2333	2333
q6	184	165	137	137
q7	952	754	616	616
q8	9324	1364	1109	1109
q9	6915	5085	5146	5085
q10	6945	2397	1984	1984
q11	482	298	302	298
q12	373	362	234	234
q13	17783	3668	3067	3067
q14	242	238	226	226
q15	583	496	490	490
q16	994	1014	967	967
q17	597	851	369	369
q18	7496	7276	7122	7122
q19	1281	966	539	539
q20	340	329	222	222
q21	3744	3169	2353	2353
q22	1102	998	968	968
Total cold run time: 106959 ms
Total hot run time: 34593 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5188	5081	5065	5065
q2	250	328	230	230
q3	2202	2689	2289	2289
q4	1314	1786	1382	1382
q5	4229	4586	4639	4586
q6	228	188	135	135
q7	2046	2032	1875	1875
q8	2737	2650	2624	2624
q9	7490	7505	7544	7505
q10	3178	3437	2935	2935
q11	601	548	513	513
q12	710	816	668	668
q13	3471	3911	3395	3395
q14	296	342	274	274
q15	509	485	487	485
q16	1096	1125	1059	1059
q17	1192	1623	1469	1469
q18	7948	7679	7671	7671
q19	928	835	887	835
q20	1955	2120	1836	1836
q21	4919	4376	4222	4222
q22	1074	1053	1009	1009
Total cold run time: 53561 ms
Total hot run time: 52062 ms

doris-robot · 2025-09-09T06:31:53Z

TPC-H: Total hot run time: 35185 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f28d116aaa782852d2f516ed6e953447790edfdb, data reload: false
------ Round 1 ----------------------------------
q1	17598	5267	5123	5123
q2	1990	321	229	229
q3	10273	1365	764	764
q4	10374	1033	534	534
q5	9758	2488	2364	2364
q6	230	171	141	141
q7	942	777	644	644
q8	9364	1395	1142	1142
q9	7158	5204	5203	5203
q10	7002	2420	2030	2030
q11	491	302	298	298
q12	378	372	231	231
q13	17785	3708	3062	3062
q14	253	246	221	221
q15	586	509	482	482
q16	1029	996	953	953
q17	626	870	393	393
q18	7475	7290	7140	7140
q19	1101	955	574	574
q20	355	352	233	233
q21	3826	3363	2445	2445
q22	1074	1036	979	979
Total cold run time: 109668 ms
Total hot run time: 35185 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5202	5110	5152	5110
q2	251	325	225	225
q3	2210	2702	2318	2318
q4	1381	1812	1323	1323
q5	4275	4115	4182	4115
q6	216	173	127	127
q7	1945	1815	1702	1702
q8	2535	2514	2422	2422
q9	6835	6955	6836	6836
q10	2943	3166	2743	2743
q11	562	530	513	513
q12	663	765	601	601
q13	3310	3707	3111	3111
q14	288	286	265	265
q15	520	467	478	467
q16	1062	1048	1028	1028
q17	1131	1529	1333	1333
q18	7280	7192	6995	6995
q19	803	817	809	809
q20	1911	1961	1807	1807
q21	4841	4446	4338	4338
q22	1084	1088	987	987
Total cold run time: 51248 ms
Total hot run time: 49175 ms

doris-robot · 2025-09-09T07:12:58Z

TPC-H: Total hot run time: 34806 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f28d116aaa782852d2f516ed6e953447790edfdb, data reload: false
------ Round 1 ----------------------------------
q1	17605	5213	5075	5075
q2	2048	329	213	213
q3	10239	1296	715	715
q4	10235	1024	527	527
q5	7533	2429	2319	2319
q6	187	170	135	135
q7	932	764	664	664
q8	9348	1345	1198	1198
q9	7034	5068	5205	5068
q10	6902	2411	1970	1970
q11	509	305	282	282
q12	356	366	222	222
q13	17762	3682	3109	3109
q14	247	237	220	220
q15	558	507	491	491
q16	991	1005	937	937
q17	586	876	380	380
q18	7538	7141	7156	7141
q19	1257	953	558	558
q20	347	342	239	239
q21	3841	3211	2363	2363
q22	1065	1039	980	980
Total cold run time: 107120 ms
Total hot run time: 34806 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5133	5098	5108	5098
q2	255	346	227	227
q3	2170	2687	2312	2312
q4	1352	1806	1362	1362
q5	4271	4522	4686	4522
q6	215	170	131	131
q7	2035	1954	1848	1848
q8	2701	2681	2567	2567
q9	7384	7207	7412	7207
q10	3123	3349	2935	2935
q11	573	508	515	508
q12	708	804	684	684
q13	3527	3876	3332	3332
q14	298	319	284	284
q15	536	475	554	475
q16	1152	1136	1069	1069
q17	1200	1536	1436	1436
q18	8170	7904	7628	7628
q19	891	819	841	819
q20	1997	2112	1932	1932
q21	5030	4335	4303	4303
q22	1118	1046	987	987
Total cold run time: 53839 ms
Total hot run time: 51666 ms

doris-robot · 2025-09-09T07:24:49Z

TPC-DS: Total hot run time: 189027 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f28d116aaa782852d2f516ed6e953447790edfdb, data reload: false
query1	1083	457	425	425
query2	6563	1706	1685	1685
query3	6763	224	228	224
query4	26282	23614	22910	22910
query5	4402	642	507	507
query6	350	256	230	230
query7	4665	521	304	304
query8	310	270	264	264
query9	8690	2927	2929	2927
query10	520	359	300	300
query11	15973	15216	14787	14787
query12	189	126	122	122
query13	1696	578	439	439
query14	11467	9288	9157	9157
query15	213	196	187	187
query16	7689	671	492	492
query17	1473	785	666	666
query18	2064	450	356	356
query19	251	201	172	172
query20	137	126	134	126
query21	217	132	114	114
query22	4282	4323	4154	4154
query23	34071	33143	33204	33143
query24	8133	2341	2413	2341
query25	613	510	446	446
query26	1235	281	176	176
query27	2674	513	350	350
query28	4352	2279	2268	2268
query29	792	624	498	498
query30	289	242	201	201
query31	934	801	729	729
query32	88	85	76	76
query33	573	398	363	363
query34	801	880	530	530
query35	829	842	754	754
query36	977	1031	921	921
query37	123	114	94	94
query38	3488	3529	3499	3499
query39	1517	1436	1453	1436
query40	238	134	133	133
query41	66	62	62	62
query42	130	120	116	116
query43	522	494	480	480
query44	1358	861	858	858
query45	186	183	169	169
query46	864	1027	647	647
query47	1808	1795	1698	1698
query48	417	428	331	331
query49	754	520	424	424
query50	670	687	408	408
query51	3954	3972	3880	3880
query52	130	119	115	115
query53	262	279	200	200
query54	613	601	554	554
query55	103	96	91	91
query56	335	357	354	354
query57	1182	1194	1115	1115
query58	302	286	286	286
query59	2614	2702	2600	2600
query60	370	366	359	359
query61	169	160	157	157
query62	845	735	686	686
query63	238	204	206	204
query64	4489	1149	841	841
query65	4063	3964	3966	3964
query66	1115	440	357	357
query67	15592	15291	15161	15161
query68	9308	937	584	584
query69	498	333	292	292
query70	1436	1221	1324	1221
query71	582	350	329	329
query72	6040	5125	4997	4997
query73	740	679	361	361
query74	8984	8859	8804	8804
query75	4437	3316	2710	2710
query76	3698	1202	758	758
query77	813	405	339	339
query78	9584	9682	8887	8887
query79	4588	836	581	581
query80	694	586	524	524
query81	493	258	224	224
query82	580	175	138	138
query83	306	257	263	257
query84	304	119	95	95
query85	891	470	440	440
query86	344	298	304	298
query87	3695	3750	3646	3646
query88	3007	2233	2235	2233
query89	453	332	296	296
query90	1967	223	230	223
query91	162	179	139	139
query92	92	80	74	74
query93	2377	1001	652	652
query94	702	417	319	319
query95	422	333	328	328
query96	482	596	283	283
query97	2961	2980	2882	2882
query98	233	228	217	217
query99	1461	1434	1273	1273
Total cold run time: 282491 ms
Total hot run time: 189027 ms

doris-robot · 2025-09-09T07:30:16Z

ClickBench: Total hot run time: 29.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f28d116aaa782852d2f516ed6e953447790edfdb, data reload: false
query1	0.05	0.05	0.05
query2	0.10	0.05	0.06
query3	0.25	0.09	0.08
query4	1.64	0.12	0.12
query5	0.27	0.27	0.25
query6	1.20	0.66	0.64
query7	0.04	0.03	0.03
query8	0.06	0.04	0.04
query9	0.62	0.53	0.52
query10	0.58	0.61	0.59
query11	0.16	0.11	0.12
query12	0.16	0.12	0.12
query13	0.63	0.63	0.63
query14	1.03	1.04	1.04
query15	0.87	0.84	0.85
query16	0.42	0.39	0.40
query17	1.03	1.06	1.05
query18	0.22	0.20	0.20
query19	1.96	1.83	1.92
query20	0.02	0.02	0.01
query21	15.41	0.94	0.60
query22	0.80	1.07	0.72
query23	14.98	1.41	0.64
query24	6.74	1.16	0.41
query25	0.51	0.16	0.13
query26	0.71	0.17	0.13
query27	0.06	0.05	0.05
query28	9.20	0.95	0.42
query29	12.66	3.91	3.31
query30	0.30	0.13	0.11
query31	2.84	0.59	0.39
query32	3.25	0.57	0.49
query33	3.15	3.10	3.18
query34	16.04	5.61	4.83
query35	4.91	4.92	4.89
query36	0.70	0.52	0.53
query37	0.11	0.07	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 104.13 s
Total hot run time: 29.71 s

github-actions · 2025-09-09T08:42:31Z

PR approved by anyone and no changes requested.

github-actions · 2025-09-09T08:45:34Z

PR approved by at least one committer and no changes requested.

yiguolei · 2025-09-09T08:46:22Z

skip check_coverage

PipelineTask is also hold by task queue( apache#49753), so that it maybe the last one to be destructed. But pipeline task hold some objects, like operators, shared state, etc. So that should release memory manually. F20250908 20:07:41.329619 39575 mem_tracker_limiter.cpp:112] mem tracker label: Query#Id=ec8535b35ed34f54-afd752d5d1dd97c1, consumption: 16640, peak consumption: 16640, mem tracker not equal to 0 when mem tracker destruct, this usually means that memory tracking is inaccurate and SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. If the log is truncated, search for `Address Sanitizer` in the be.INFO log to see more information.1. For query and load, memory leaks may have occurred, it is expected that the query mem tracker will be bound to the thread context using SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER before all memory alloc and free. 2. If a memory alloc is recorded by this tracker, it is expected that be recorded in this tracker when memory is freed. 3. Merge the remaining memory tracking value by this tracker into Orphan, if you observe that Orphan is not equal to 0 in the mem tracker web or log, this indicates that there may be a memory leak. 4. If you need to transfer memory tracking value between two trackers, can use transfer_to..[Address Sanitizer]: memory not be freed: [Address Sanitizer] buf not be freed, mem tracker label: Query#Id=ec8535b35ed34f54-afd752d5d1dd97c1, consumption: 16640, peak consumption: 16640, buf: 0x7d87c8761d00, size 4096, strack trace: 0# doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>::alloc(unsigned long, unsigned long) 1# void doris::vectorized::PODArrayBase<1ul, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>::alloc<>(unsigned long) 2# void doris::vectorized::PODArray<signed char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>::push_back<long>(long&&) 3# doris::vectorized::ColumnVector<(doris::PrimitiveType)3>::insert(doris::vectorized::Field const&) 4# doris::vectorized::IDataType::create_column_const(unsigned long, doris::vectorized::Field const&) const 5# doris::vectorized::VLiteral::init(doris::TExprNode const&) 6# doris::vectorized::VLiteral::VLiteral(doris::TExprNode const&, bool) 7# std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::vectorized::VLiteral, std::allocator<void>, doris::TExprNode const&>(doris::vectorized::VLiteral*&, std::_Sp_alloc_shared_tag<std::allocator<void> >, doris::TExprNode const&) 8# doris::vectorized::VExpr::create_expr(doris::TExprNode const&, std::shared_ptr<doris::vectorized::VExpr>&) 9# doris::vectorized::VExpr::create_tree_from_thrift(std::vector<doris::TExprNode, std::allocator<doris::TExprNode> > const&, int*, std::shared_ptr<doris::vectorized::VExpr>&, std::shared_ptr<doris::vectorized::VExprContext>&) 10# doris::vectorized::VExpr::create_expr_tree(doris::TExpr const&, std::shared_ptr<doris::vectorized::VExprContext>&) 11# doris::pipeline::OperatorXBase::init(doris::TPlanNode const&, doris::RuntimeState*) 12# doris::pipeline::ScanOperatorX<doris::pipeline::OlapScanLocalState>::init(doris::TPlanNode const&, doris::RuntimeState*) 13# doris::pipeline::PipelineFragmentContext::_create_tree_helper(doris::ObjectPool*, std::vector<doris::TPlanNode, std::allocator<doris::TPlanNode> > const&, doris::TPipelineFragmentParams const&, doris::DescriptorTbl const&, std::shared_ptr<doris::pipeline::OperatorXBase>, int*, std::shared_ptr<doris::pipeline::OperatorXBase>*, std::shared_ptr<doris::pipeline::Pipeline>&, int, bool) 14# doris::pipeline::PipelineFragmentContext::_build_pipelines(doris::ObjectPool*, doris::TPipelineFragmentParams const&, doris::DescriptorTbl const&, std::shared_ptr<doris::pipeline::OperatorXBase>*, std::shared_ptr<doris::pipeline::Pipeline>) 15# doris::pipeline::PipelineFragmentContext::prepare(doris::TPipelineFragmentParams const&, doris::ThreadPool*) 16# doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, std::function<void (doris::RuntimeState*, doris::Status*)> const&, doris::TPipelineFragmentParamsList const&) 17# doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, doris::TPipelineFragmentParamsList const&) 18# doris::PInternalService::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::PFragmentRequestVersion, bool, std::function<void (doris::RuntimeState*, doris::Status*)> const&) 19# doris::PInternalService::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) 20# doris::WorkThreadPool<false>::work_thread(int) 21# execute_native_thread_routine 22# asan_thread_start(void*) 23# ?

…5823) picked from #55796

… (apache#55823) picked from apache#55796

yiguolei added 3 commits September 9, 2025 11:33

[bugfix](memtracker) memory not consumed by memtracker
8b1fdad

f
33a3acc

f
9301d43

yiguolei added dev/3.0.x dev/3.1.x labels Sep 9, 2025

f
f28d116

dwdwqfwe approved these changes Sep 9, 2025
View reviewed changes

github-actionsBot added the reviewed label Sep 9, 2025

jacktengg approved these changes Sep 9, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 9, 2025

yiguolei merged commit 47e1b90 into apache:masterSep 9, 2025
26 of 28 checks passed

github-actionsBot added dev/3.0.x-conflict dev/3.1.x-conflict labels Sep 9, 2025

morrySnow mentioned this pull request Sep 11, 2025
branch-3.1: (memtracker) memory not consumed by memtracker #55796 #55823
Merged

morrySnow pushed a commit that referenced this pull request Sep 11, 2025
branch-3.1: (memtracker) memory not consumed by memtracker#55796(#5…
c58b3e9
…5823) picked from #55796

morrySnow added dev/3.1.1-merged and removed dev/3.1.x dev/3.1.x-conflict labels Sep 11, 2025

morrySnow mentioned this pull request Sep 22, 2025
Release Note 3.1.1 #56289
Closed

yiguolei added a commit to yiguolei/incubator-doris that referenced this pull request Sep 28, 2025
branch-3.1: (memtracker) memory not consumed by memtrackerapache#55796…
670499b
… (apache#55823) picked from apache#55796

Uh oh!

Conversation

yiguolei commented Sep 9, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 9, 2025

Uh oh!

yiguolei commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

yiguolei commented Sep 9, 2025

Uh oh!

yiguolei commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

doris-robot commented Sep 9, 2025

Uh oh!

github-actionsBot commented Sep 9, 2025

Uh oh!

github-actionsBot commented Sep 9, 2025

Uh oh!

yiguolei commented Sep 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yiguolei commented Sep 9, 2025•
edited
Loading