[fix](memory) Fix BE memory info compatible with Cgroup by xinyiZzz · Pull Request #35412 · apache/doris

xinyiZzz · 2024-05-26T15:23:51Z

Proposed changes

memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff), free cache can be reused,
so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"].
If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-26T15:23:56Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-05-26T15:23:57Z

run buildall

github-actions · 2024-05-26T15:29:57Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-26T16:53:41Z

TPC-H: Total hot run time: 41403 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c567fa8b79303374fe272a7cdf7f7d1a662485b1, data reload: false
------ Round 1 ----------------------------------
q1	17597	4402	4312	4312
q2	2012	197	194	194
q3	10457	1241	1243	1241
q4	10216	790	863	790
q5	7501	2747	2726	2726
q6	230	135	135	135
q7	968	615	600	600
q8	9235	2148	2115	2115
q9	9611	6778	6826	6778
q10	9413	3842	3975	3842
q11	485	247	253	247
q12	468	227	225	225
q13	17442	3188	3204	3188
q14	262	223	213	213
q15	509	461	475	461
q16	500	431	399	399
q17	989	702	717	702
q18	8423	7931	7777	7777
q19	4458	1579	1522	1522
q20	663	326	313	313
q21	5159	3339	3949	3339
q22	348	284	285	284
Total cold run time: 116946 ms
Total hot run time: 41403 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4544	4418	4432	4418
q2	378	263	257	257
q3	3146	2906	2922	2906
q4	1875	1630	1619	1619
q5	5438	5511	5503	5503
q6	211	124	125	124
q7	2190	1824	1832	1824
q8	3255	3382	3415	3382
q9	8650	8701	8733	8701
q10	3989	3687	3816	3687
q11	585	507	494	494
q12	812	628	625	625
q13	16150	3143	3187	3143
q14	299	284	293	284
q15	538	472	483	472
q16	483	429	424	424
q17	1790	1495	1449	1449
q18	7682	7816	7505	7505
q19	1696	1520	1581	1520
q20	2027	1810	1767	1767
q21	4894	4757	4771	4757
q22	598	481	501	481
Total cold run time: 71230 ms
Total hot run time: 55342 ms

doris-robot · 2024-05-26T17:04:21Z

TPC-DS: Total hot run time: 169391 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c567fa8b79303374fe272a7cdf7f7d1a662485b1, data reload: false
query1	911	386	373	373
query2	6438	2350	2360	2350
query3	6648	209	209	209
query4	19209	17382	17318	17318
query5	4143	414	417	414
query6	240	161	153	153
query7	4594	295	301	295
query8	252	197	181	181
query9	8468	2409	2392	2392
query10	457	274	290	274
query11	10682	10200	10061	10061
query12	144	88	85	85
query13	1629	364	358	358
query14	9434	6799	6897	6799
query15	206	168	169	168
query16	7412	255	253	253
query17	1330	516	524	516
query18	1851	282	268	268
query19	204	153	148	148
query20	90	86	88	86
query21	196	135	136	135
query22	4312	4206	4104	4104
query23	33776	33021	32978	32978
query24	7736	2834	2851	2834
query25	589	352	365	352
query26	702	154	151	151
query27	2192	317	320	317
query28	5297	2083	2097	2083
query29	844	595	606	595
query30	260	172	176	172
query31	971	772	746	746
query32	88	55	54	54
query33	534	280	281	280
query34	872	477	494	477
query35	713	642	605	605
query36	1047	941	898	898
query37	108	74	70	70
query38	2907	2788	2763	2763
query39	850	797	822	797
query40	192	127	121	121
query41	45	42	45	42
query42	102	97	99	97
query43	578	557	541	541
query44	1073	734	760	734
query45	176	164	162	162
query46	1069	740	723	723
query47	1861	1790	1831	1790
query48	364	289	298	289
query49	844	371	392	371
query50	778	381	385	381
query51	6930	6786	6677	6677
query52	109	87	97	87
query53	345	283	282	282
query54	559	422	418	418
query55	77	74	72	72
query56	286	254	233	233
query57	1125	1057	1074	1057
query58	225	210	209	209
query59	3459	2948	2967	2948
query60	271	257	252	252
query61	90	85	85	85
query62	590	468	441	441
query63	313	280	283	280
query64	8461	2190	1739	1739
query65	3145	3056	3105	3056
query66	806	323	322	322
query67	15260	15233	14898	14898
query68	4492	533	536	533
query69	435	261	259	259
query70	1132	1030	1139	1030
query71	370	264	267	264
query72	7461	5648	2745	2745
query73	728	326	319	319
query74	6028	5630	5518	5518
query75	3318	2640	2651	2640
query76	2345	1020	1012	1012
query77	397	263	267	263
query78	10237	9850	10100	9850
query79	1989	522	517	517
query80	1116	436	426	426
query81	539	242	244	242
query82	614	95	91	91
query83	245	176	170	170
query84	247	88	82	82
query85	1692	277	268	268
query86	510	286	322	286
query87	3308	3132	3122	3122
query88	4170	2348	2341	2341
query89	469	377	390	377
query90	2041	185	184	184
query91	121	99	94	94
query92	62	47	48	47
query93	2539	527	505	505
query94	1144	190	178	178
query95	404	306	303	303
query96	585	272	261	261
query97	3197	3005	3078	3005
query98	251	226	209	209
query99	1140	858	831	831
Total cold run time: 260795 ms
Total hot run time: 169391 ms

doris-robot · 2024-05-26T17:09:51Z

ClickBench: Total hot run time: 31.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c567fa8b79303374fe272a7cdf7f7d1a662485b1, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.69	0.09	0.08
query5	0.51	0.49	0.50
query6	1.11	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.48	0.47
query10	0.54	0.54	0.53
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.58	0.59
query14	0.78	0.78	0.78
query15	0.83	0.84	0.80
query16	0.36	0.36	0.37
query17	0.99	0.95	0.93
query18	0.20	0.22	0.23
query19	1.75	1.76	1.74
query20	0.01	0.01	0.02
query21	15.52	0.69	0.66
query22	4.37	6.41	2.91
query23	18.31	1.32	1.32
query24	1.33	0.33	0.28
query25	0.14	0.08	0.08
query26	0.27	0.16	0.17
query27	0.07	0.08	0.07
query28	13.38	1.00	1.00
query29	13.54	3.32	3.23
query30	0.24	0.06	0.05
query31	2.88	0.37	0.38
query32	3.30	0.47	0.46
query33	2.90	2.98	2.80
query34	17.17	4.46	4.48
query35	4.55	4.50	4.59
query36	0.67	0.45	0.47
query37	0.17	0.15	0.14
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.97 s
Total hot run time: 31.48 s

doris-robot · 2024-05-26T17:10:10Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (9015/25291)
Line Coverage: 27.33% (74571/272854)
Region Coverage: 26.55% (38579/145313)
Branch Coverage: 23.42% (19666/83984)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c567fa8b79303374fe272a7cdf7f7d1a662485b1_c567fa8b79303374fe272a7cdf7f7d1a662485b1/report/index.html

yiguolei

LGTM

github-actions · 2024-05-27T03:10:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-27T03:10:39Z

PR approved by anyone and no changes requested.

morningman

LGTM

1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused, so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"]. 2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.

) 1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused, so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"]. 2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.

1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused, so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"]. 2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.

…he#35083 apache#35557 (apache#3405) [fix](memory) Fix BE memory info compatible with Cgroup (apache#35412) [fix](memory) Avoid frequently refresh cgroup memory info (apache#35083) [fix](libjdk) Revert support loading libjvm at runtime (apache#35557)

Introduced in this PR: apache/doris#35412

1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused, so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"]. 2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.

1
c567fa8

yiguolei approved these changes May 27, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 27, 2024

github-actionsBot added the reviewed label May 27, 2024

morningman approved these changes May 27, 2024
View reviewed changes

morningman added the dev/2.1.x label May 27, 2024

morningman merged commit 31c3806 into apache:masterMay 27, 2024

xinyiZzz mentioned this pull request May 27, 2024
[branch-2.1](memory) Fix BE memory info compatible with Cgroup #35425
Merged

yiguolei added dev/2.1.4-merged and removed dev/2.1.x labels May 27, 2024

gavinchou added the dev/3.0.x label May 29, 2024

gavinchou added dev/3.0.0-merged and removed dev/3.0.x labels Jun 11, 2024

xinyiZzz mentioned this pull request Jun 12, 2024
[Bug] Could not find subsystem memory in /proc/self/cgroup error stacktrace #32842
Closed
3 tasks

johnpyp added a commit to johnpyp/doris-website that referenced this pull request Aug 7, 2024
[doc] document enable_use_cgroup_memory_info
4b69fa6
Introduced in this PR: apache/doris#35412

johnpyp mentioned this pull request Aug 7, 2024
[doc] document enable_use_cgroup_memory_infoapache/doris-website#964
Closed

Navigation Menu

Uh oh!

[fix](memory) Fix BE memory info compatible with Cgroup - #35412

[fix](memory) Fix BE memory info compatible with Cgroup#35412
morningman merged 1 commit into
apache:masterfrom
xinyiZzz:20240523_fix_cgroup

xinyiZzz commented May 26, 2024•
edited
Loading

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

xinyiZzz commented May 26, 2024

Uh oh!

github-actionsBot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

yiguolei left a comment

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

morningman left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

xinyiZzz commented May 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

xinyiZzz commented May 26, 2024

Uh oh!

github-actionsBot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

doris-robot commented May 26, 2024

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xinyiZzz commented May 26, 2024•
edited
Loading