Skip to content

[opt](cloud) Fix frequent rlock for SystemInfoService.getClusterXxx() - #47203

Merged
gavinchou merged 1 commit into
apache:masterfrom
gavinchou:gavin-fix-cloud-cluster-lock
Jan 19, 2025
Merged

[opt](cloud) Fix frequent rlock for SystemInfoService.getClusterXxx()#47203
gavinchou merged 1 commit into
apache:masterfrom
gavinchou:gavin-fix-cloud-cluster-lock

Conversation

@gavinchou

@gavinchougavinchou commented Jan 19, 2025

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

ReentrantLock may consume lots of CPU in some cases.
Remove some redundant rlock to prevent the potential CPU issue.

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Jan 19, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou

gavinchou commented Jan 19, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

deardeng
deardeng approved these changes Jan 19, 2025
View reviewed changes

@deardengdeardeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jan 19, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Jan 19, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32453 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 410059fb2371b25533ba12b11cac792f905a80f9, data reload: false
------ Round 1 ----------------------------------
q1	17597	5560	5494	5494
q2	2049	313	178	178
q3	10514	1317	741	741
q4	10253	970	518	518
q5	8533	2412	2160	2160
q6	197	170	132	132
q7	886	766	627	627
q8	9249	1379	1183	1183
q9	5311	4919	4969	4919
q10	6799	2329	1887	1887
q11	462	265	258	258
q12	334	354	219	219
q13	17784	3673	3086	3086
q14	226	230	206	206
q15	511	462	468	462
q16	633	627	590	590
q17	587	874	319	319
q18	7363	6526	6405	6405
q19	1806	975	541	541
q20	308	315	191	191
q21	3076	2207	2020	2020
q22	368	329	317	317
Total cold run time: 104846 ms
Total hot run time: 32453 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5588	5528	5517	5517
q2	260	339	244	244
q3	2310	2661	2331	2331
q4	1404	1844	1358	1358
q5	4338	4743	4742	4742
q6	169	161	126	126
q7	2100	1973	1845	1845
q8	2645	2792	2659	2659
q9	7240	7302	7181	7181
q10	2986	3295	2810	2810
q11	582	511	506	506
q12	691	761	619	619
q13	3487	3956	3245	3245
q14	286	296	285	285
q15	518	468	466	466
q16	672	712	653	653
q17	1252	1739	1262	1262
q18	7697	7350	7383	7350
q19	793	906	1163	906
q20	1985	2062	1906	1906
q21	5655	5247	5031	5031
q22	633	617	553	553
Total cold run time: 53291 ms
Total hot run time: 51595 ms

@doris-robot

doris-robot commented Jan 19, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 187017 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 410059fb2371b25533ba12b11cac792f905a80f9, data reload: false
query1	973	407	383	383
query2	6524	2077	2083	2077
query3	6791	212	211	211
query4	33536	23327	22949	22949
query5	4312	608	466	466
query6	297	190	183	183
query7	4589	488	306	306
query8	279	227	222	222
query9	9223	2589	2574	2574
query10	462	322	278	278
query11	18072	15227	15006	15006
query12	169	104	110	104
query13	1662	533	420	420
query14	10416	6932	6885	6885
query15	254	203	186	186
query16	8117	626	467	467
query17	1626	748	595	595
query18	2130	413	307	307
query19	219	180	165	165
query20	115	110	117	110
query21	213	121	103	103
query22	4330	4228	4097	4097
query23	34064	33830	32920	32920
query24	6484	2305	2249	2249
query25	456	438	419	419
query26	1060	246	156	156
query27	2034	467	329	329
query28	5339	2430	2402	2402
query29	543	531	430	430
query30	227	183	157	157
query31	966	881	758	758
query32	93	60	60	60
query33	508	343	298	298
query34	764	877	519	519
query35	810	803	768	768
query36	994	1023	936	936
query37	135	99	81	81
query38	4240	4215	4121	4121
query39	1445	1558	1394	1394
query40	208	114	100	100
query41	53	51	59	51
query42	121	103	109	103
query43	500	543	482	482
query44	1335	796	805	796
query45	184	174	188	174
query46	870	1044	645	645
query47	1794	1872	1786	1786
query48	374	403	305	305
query49	786	479	376	376
query50	647	654	414	414
query51	6910	6975	6740	6740
query52	106	103	94	94
query53	225	246	192	192
query54	476	522	414	414
query55	83	82	82	82
query56	259	275	244	244
query57	1156	1155	1108	1108
query58	243	226	241	226
query59	2954	3044	2983	2983
query60	273	266	246	246
query61	120	117	114	114
query62	803	721	677	677
query63	224	196	193	193
query64	4589	1017	652	652
query65	3299	3146	3181	3146
query66	1043	417	309	309
query67	15982	15471	15300	15300
query68	5722	819	520	520
query69	489	293	258	258
query70	1243	1110	1155	1110
query71	381	290	254	254
query72	5845	3808	3741	3741
query73	651	760	355	355
query74	10489	9219	8558	8558
query75	3170	3176	2651	2651
query76	3156	1202	766	766
query77	469	360	290	290
query78	10050	10018	9356	9356
query79	2828	812	624	624
query80	629	538	501	501
query81	535	279	246	246
query82	226	152	124	124
query83	187	176	148	148
query84	246	98	77	77
query85	768	340	295	295
query86	400	329	289	289
query87	4531	4521	4414	4414
query88	4850	2173	2137	2137
query89	389	340	303	303
query90	1915	198	195	195
query91	196	135	108	108
query92	67	58	60	58
query93	1984	880	533	533
query94	667	390	299	299
query95	343	271	254	254
query96	492	614	293	293
query97	2776	2842	2757	2757
query98	234	197	197	197
query99	1264	1385	1263	1263
Total cold run time: 287280 ms
Total hot run time: 187017 ms

dataroaring
dataroaring approved these changes Jan 19, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot

doris-robot commented Jan 19, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 410059fb2371b25533ba12b11cac792f905a80f9, data reload: false
query1	0.06	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.40	0.42	0.39
query6	1.17	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.61	0.51	0.51
query10	0.53	0.55	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.61	0.60
query14	2.72	2.73	2.80
query15	0.91	0.84	0.83
query16	0.38	0.37	0.38
query17	1.06	1.02	1.03
query18	0.23	0.21	0.22
query19	1.87	1.83	1.97
query20	0.01	0.01	0.01
query21	15.35	0.88	0.59
query22	0.74	0.73	0.72
query23	15.35	1.48	0.54
query24	3.01	0.58	1.66
query25	0.19	0.15	0.06
query26	0.20	0.14	0.13
query27	0.05	0.05	0.05
query28	14.20	1.03	0.44
query29	12.55	3.94	3.23
query30	0.25	0.09	0.07
query31	2.83	0.59	0.38
query32	3.22	0.55	0.46
query33	3.01	3.00	3.04
query34	16.65	5.16	4.55
query35	4.57	4.55	4.59
query36	0.65	0.48	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.15 s
Total hot run time: 30.33 s

@github-actions

github-actionsBot commented Jan 19, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 19, 2025
@gavinchou
gavinchou merged commit c215bc9 into apache:masterJan 19, 2025
github-actionsBot pushed a commit that referenced this pull request Jan 19, 2025
@gavinchouYour Name
[opt](cloud) Fix frequent rlock for SystemInfoService.getClusterXxx() (…
663fb7c
…#47203)
ReentrantLock may consume lots of CPU in some cases.
Remove some redundant rlock to prevent the potential CPU issue.
@github-actionsgithub-actionsBot mentioned this pull request Jan 19, 2025
Merged
gavinchou added a commit that referenced this pull request Jan 20, 2025
@github-actions@gavinchou
branch-3.0: [opt](cloud) Fix frequent rlock for SystemInfoService.get…
d7f3801
…ClusterXxx() #47203 (#47210)
Cherry-picked from #47203
Co-authored-by: Gavin Chou <gavin@selectdb.com>
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Feb 7, 2025
@gavinchou
[opt](cloud) Fix frequent rlock for SystemInfoService.getClusterXxx() (…
ca1d57a
…apache#47203)
ReentrantLock may consume lots of CPU in some cases.
Remove some redundant rlock to prevent the potential CPU issue.
@gavinchougavinchou mentioned this pull request Feb 18, 2025
Closed
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@gavinchou@lzyy2024
[opt](cloud) Fix frequent rlock for SystemInfoService.getClusterXxx() (…
1d45fd3
…apache#47203)
ReentrantLock may consume lots of CPU in some cases.
Remove some redundant rlock to prevent the potential CPU issue.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deardengdeardengdeardeng approved these changes
@dataroaringdataroaringdataroaring approved these changes
@w41terw41terAwaiting requested review from w41ter

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.4-mergedp0_creviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@gavinchou@hello-stephen@doris-robot@deardeng@dataroaring@wm1581066